How we defended our CTF Competition against a DDoS Attack.

Hello, as you may know our team organized a CTF Competition last year (X-MAS CTF 2018). Plenty of preparation time went into it: We had to design the problems and we had to set up the right infrastructure (Choosing a good VPS and a good CTF Platform).

Then, we had to make sure that the servers would resist under the heavy load that would unfold once the CTF starts (~1000 teams were registered before the competition started).

Therefore, extensive testing started. We took our low-duty python flooding scripts, and we started flooding the website with ~5.000 requests per second. The server went down, and this is where the first round of improvements begun:

Once we blocked /, the site was functioning properly, but there was still room for improvement:

After all these tweaks and fixes, our server was supporting our small python floods without breaking a sweat. It had a theoretical limit of ~60.000 Requests Per Second, and thus we knew that it would be able to host all the participants on the D-Day, and so it was.

The server didn’t lag or slow down for a moment, everything was snappy, and ~ 1500 flags (both right and wrong) were submitted in the first 15 minutes of the competition. The challenges were working fine, and it seemed that it’d be smooth sailing from then on.

The DDoS

After a good saturday of hacking and doing admin work, everything seemed alright. The competition was alive and well and people were having fun.

However, this all took an unexpected turn when we received multiple messages from people asking if the site is down. We checked the site, and indeed, it was down. SSH wasn’t responding, and we didn’t have any response from the server whatsoever. This is when we realized we were being DDoSed (or DoSed, in this case).

The good thing is that after about 2 hours, the attacker got bored, and the site came back online. Judging by the amount of new accounts/teams created, we realized that they spammed the site with registration requests. This is a pretty costly operation (updating the database, etc), and thus the site went down.

Here’s when we added another vital improvement, that anyone should take into consideration when designing a site:

We then unpaused the competition and everything seemed to be normal. There shouldn’t be any more interruptions, and the competition would be smooth sailing from then on, right? Well, not really.

You see, on the Internet there will always be someone wanting to take you down. This is why you should prepare your site for the fiercest attacks, if you want to see it up for the foreseeable future. Here is a pretty neat talk from Cloudflare talking about DDoS attack defense.

Taking that into account, the next day we were heavily DDoSed. This time, a real distributed attack, from a 12K device botnet (Luckily the attackers were kind enough to provide us a screenshot of their C&C program. How considerate!).

Each device was sending ~42K requests per second, totalling to about 500K requests per second, a much higher RPS than our theoretical limit. We saw that afterwards another similar botnet joined in, pumping ~ 1 Million requests per second into our platform. This was a pretty serious attack, no longer a simple Registration-spamming python script, we are talking medium-sized botnets here.

How we defended from this? Well, we did all we could:

With all these new configs, the servers remained alive and well for the rest of the competition, albeit a bit fuzzy while we tweaked the configs to better fit each challenge environment. We have seen several various smaller attacks in the following days, but they all barely scratched the paint of the server.

Another small thing we learned from this experience is that mischevious competitors will always find ways to break your challenges. If there is even the smallest permission overlooked, people will exploit that to take away / modify the challenge flag, or they will render the challenge unusable. Hence, as a rule of thumb, you should:

With this last simple security check, you can be sure that your challenge is safe and sound from interfering players.