Cloudflare was unexpectedly hit by an enormous HTTP attack that peaked at over 201 million requests per second.
Starting on August 25, 2023, this onslaught posed a significant challenge, especially considering that it was initiated by a relatively modest botnet of just 20,000 machines.
To put this in perspective, the entire web typically handles between 1 to 3 billion requests per second. Detecting and mitigating these attacks required substantial efforts.
During the first wave of attacks, a small fraction of customer requests, approximately 1%, were initially affected.
However, Cloudflare’s existing protection mechanisms were eventually refined to prevent the attacks from affecting its customers without causing harm to the company’s systems.
Notably, these attacks were not exclusive to Cloudflare; other major industry players like Google and AWS experienced similar challenges.
HTTP/2 Rapid Reset Zero-day
To address this, Cloudflare collaborated with Google and AWS to coordinate the disclosure of the attack to affected vendors and critical infrastructure providers.
The root of this issue lies in the abuse of certain features of the HTTP/2 protocol and server implementation details, as detailed in CVE-2023-44487.
With the widespread use of HTTP/2, it has become imperative for web server vendors to develop and apply necessary updates and fixes to ensure a seamless and secure web browsing experience for users.
In the meantime, relying on DDoS mitigation services like Cloudflare became the best defense against such attacks.
This article delves into the technical details of the HTTP/2 protocol, the specific features exploited by the attackers, and the mitigation strategies deployed to safeguard Cloudflare’s customers.
By sharing this information, the aim is to empower other web servers and services to implement similar countermeasures and for protocol standards teams to enhance the design of future web standards to prevent such attacks.
One crucial aspect of the attack was the manipulation of the HTTP/2 protocol’s stream concurrency, which allowed the attacker to flood servers with excessive requests.
HTTP/2’s features, like stream multiplexing, concurrency, and request cancellation, make it more efficient than HTTP/1.1, but they also introduce potential vulnerabilities that attackers can exploit.
One of the vulnerabilities highlighted is rapid request resets in HTTP/2. This abuse involves rapidly resetting an unbounded number of streams, which can lead to a denial of service.
The speed at which an HTTP/2 server can process these resets plays a significant role. If there’s any delay or lag in handling them, a backlog of work accumulates, consuming server resources.
In Cloudflare’s case, the architecture of their reverse proxies and load balancers played a role.
While the architecture allowed for efficient handling of client traffic, it made it challenging to tidy up in-process jobs when a client sent an overwhelming number of rapid resets.
To mitigate the attacks, Cloudflare took multiple actions, including extending its IP Jail system to protect its entire infrastructure and changing its maximum stream concurrency settings.
Despite the challenges, Cloudflare’s commitment to providing accessible, unmetered, and unlimited DDoS protection to its customers remains unwavering.
As they continue to face evolving threats, they remain vigilant in identifying and countering new attack vectors to ensure the security of their millions of customers.
Protect yourself from vulnerabilities using Patch Manager Plus to patch over 850 third-party applications quickly. Take advantage of the free trial to ensure 100% security.