DDoS Attack Migrated

3/04 21:38 We have successfully traced the attackers and we will work with the police to crack down on this cybercrime.


3/04 21:19 Attack Ended


We detected a large amount of DDoS traffic from China to JP-NRT-C01:

  • Traffic: 12.01 Gbps
  • PPS Before Migrate: 107107
  • Attack Method: Memcached / UDP
  • PPS Total: 1055000
  • Migrated: Yes
  • Blackhole: No

The "Memcrashed" - Not Just 50,000 Times Larger

A few days ago, Cloudflare blogged that they found a new type of DDoS attack that uses Memcached as the source of reflection amplification.

After I read this article, I looked at Memcached protocol specification.

Memcached Protocol

Each UDP datagram contains a simple frame header, followed by data in the
same format as the TCP protocol described above.

The frame header is 8 bytes long, as follows (all values are 16-bit integers
in network byte order, high byte first):

0-1 Request ID
2-3 Sequence number
4-5 Total number of datagrams in this message
6-7 Reserved for future use; must be 0

So we can use a single command to test it:

➜  ~ python3 -c "print('\x00\x00\x00\x00\x00\x01\x00\x00stats\r')" | nc 127.0.0.1 11211 -u
STAT pid 60014
.... About 1500 Byte Content ...
END

Wow, 15 bytes request triggered more than 1500 bytes of response.

Then, I set up a honeypot to collect some interesting things.

The DDoS Attack

Memcached honeypot collected

Most script kiddies are using the stats command to perform the attack, but one caught my attention when he was using the "gets" command.

Retrieval command:
------------------

The retrieval commands "get" and "gets" operate like this:

get <key>*\r\n
gets <key>*\r\n

- <key>* means one or more key strings separated by whitespace.

This command works like this

Memcached set and get

The "gets" command lets the server return the value of multiple keys. So what is the maximum size of a value?

.B \-I, --max-item-size=<size>
Override the default size of each slab page. The default size is 1mb. Default
value for this parameter is 1m, minimum is 1k, max is 128m.
Adjusting this value changes the item size limit.

So let's try to simulate this DDoS attack. I ran a Memcached server on my Macbook and executed this attack on my wireless router.

First, Write 1MB to Memcached server

Memcached write key

Then, the attack payload looks like this:

\x00\x00\x00\x00\x00\x01\x00\x00get b b b b b b b b b b b b b b b b b\r\n

You can fill about 710 "b" to the maximum MTU, and a "b" will get about 1MB of response packets for a total of 710MB.

In theory, you can get (710 x 1024 x 1024) / 1500 = 496325x amplification.

Memcached attack wireshark screenshot

After I sent this payload at 10Mbps, my Macbook was stuck and had to be forced to restart. Wireshark record shows the response is about 400,000 times larger.

There are about 180,000 open Memcached servers on the internet, the worst part is that you only need 100Mbps bandwidth to launch DDoS attacks over 1Tbps.

How to prevent

Red Hat

Memcached in CentOS, Fedora, and RHEL listening 0.0.0.0 by default, What?

Tutorial writers

Stop writing stupid compile & install scripts for common packages.

Many people advise developers not to use UDP, I can't imagine that there is only TCP in the world, improve security awareness of developers and ops is the most important thing.

DDoS Attack Migrated

3/02 03:00 Attack Ended


3/01 22:43 Attack Started Again


3/01 01:36 Attack Ended


We detected a large amount of DDoS traffic from China to RU-KHV-C01:

  • Traffic: 52.10 Gbps
  • PPS Before Migrate: 2019817
  • Migrated: Yes
  • Blackhole: No

New Location: Frankfurt

A new network location is online!

Name: C01, Frankfurt, DE

Interconnect

Name Type
C01, Amsterdam, NL IX
R01, Falkenstein, DE IX
R01, Paris, FR IX
R03, Paris, FR IX
X01, Hong Kong, CN MPLS
C01, St.Petersburg, RU Normal

Network

Unicast

Transit: NTT Telia

Internet Exchange: DE-CIX AMS-IX

Anycast

Transit: NTT

Internet Exchange: DE-CIX

The new website

I have built this new website, but the old content will not be migrated because I no longer want to maintain them.