Cause:

The incident was caused by the new kind of DDoS attack. Attacker used perfect timing to start every attack wave:

1. Large wave of requests (10k RPS) got “cache miss” on our caching frontend and caused 502 errors in our redirecting module
2. 502 errors weren’t cached, so every new request bypassed cache as well, increasing load to our redirecting module
3. After cache was populated, attacker started a new wave with a different URL

Our mitigation:

1. As an immediate fix we limited impact by disabling domain under attack (it helped other clients' domains to be available)
2. We implemented a solution, which prevents large numbers of requests to bypass cache at the same time
3. We implemented “siege mode” for domains under attack, which adds few more restrictions to these domains and increasing number of DDoS requests we can handle without affecting our legitimate customers