Cloudflare blames global outage on botched firewall rule change

clock • 3 min read

Deployment of single misconfigured firewall rule caused CPU server spike across Cloudflare's infrastructure

Cloudflare has revealed the reasons for its global outage on Tuesday, which made as much as 10 per cent of the global internet unaccessible.

In a blog post, the company claimed that an injudicious update to its firewall rules caused a CPU spike across the Cloudflare infrastructure. That tied up the company's servers, preventing Cloudflare from connecting internet users to websites, resulting in a rash of ‘502 Bad Gateway' errors across the world.

The outage started at precisely 2.42pm, the company admitted, causing a global outage across its network. "The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Application Firewall (WAF) during a routine deployment of new Cloudflare WAF Managed rules," the company explained.

It continued: "The intent of these new rules was to improve the blocking of inline JavaScript that is used in attacks. These rules were being deployed in a simulated mode where issues are identified and logged by the new rule but no customer traffic is actually blocked so that we can measure false positive rates and ensure that the new rules do not cause problems when they are deployed into full production.

"Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100 per cent on our machines worldwide. This 100 per cent CPU spike caused the 502 errors that our customers saw. At its worst traffic dropped by 82 per cent."

The blog post claimed that the company had never seen such CPU ‘exhaustion' before.

"We make software deployments constantly across the network and have automated systems to run test suites and a procedure for deploying progressively to prevent incidents. Unfortunately, these WAF rules were deployed globally in one go and caused today's outage."

At 3.02pm, the company worked out the cause of the problem. "We understood what was happening and decided to issue a ‘global kill' on the WAF Managed Rulesets, which instantly dropped CPU back to normal and restored traffic," and normality was restored by 3.09pm.

A fixed firewall ruleset was re-rolled out just before 4pm with no issues.

"We recognize that an incident like this is very painful for our customers. Our testing processes were insufficient in this case and we are reviewing and making changes to our testing and deployment process to avoid incidents like this in the future."


2 July 2019: The Cloudflare content delivery network should be getting back to normal this afternoon following what the company described as a "network performance issue". 

The networking problem effectively took down websites across the world, particularly in the UK, but also much of Europe and both the east and west coast of America, according to DownDetector, which was also affected. 

As much as 10 per cent of the internet was affected, according to reports. 

Initially, the company had warned in  an update on its website: "Cloudflare is observing network performance issues. Customers may be experiencing 502 errors while accessing sites on Cloudflare. We are working to mitigate impact to Internet users in this region."

Within the last few minutes, though, the company has updated its System Status, claiming to have rolled-out a fix for the issue, and is now "monitoring the results". 

It stated: "Cloudflare has implemented a fix for this issue and is currently monitoring the results. We will update the status once the issue is resolved."

Cloudflare was founded in 2009. Today, it claims the highest number of connections to internet exchange points of any network across the world. Cloudflare caches content to its edge locations, enabling organisations to deliver content faster and with less stress on their own networks.

In addition to content delivery, it also provides DDoS mitigation services and internet security services. In 2014, it claimed to have mitigated the world's biggest-ever (up until then) distributed denial of service attack, going on to provide some detail about the attack

It has also, though, faced legal action from a porn baron for providing the same anti-DDoS services to piracy websites. 

You may also like
Microsoft outage disrupts search engines, AI tools


Extended beyond Microsoft's own products

clock 24 May 2024 • 2 min read
McDonald's blames 'configuration change' for global outage

Business Software

Customers not lovin' it

clock 18 March 2024 • 3 min read
Facebook and Instagram down across the world

Social Networking

Facebook and Instagram are both down for many users around the world.

clock 05 March 2024 • 1 min read

Sign up to our newsletter

The best news, stories, features and photos from the day in one perfectly formed email.

More on Cloud and Infrastructure

UK cloud market consolidating around two major players, CMA says

UK cloud market consolidating around two major players, CMA says

AWS remains the market leader but Microsoft steals the show with growth and new customers

clock 29 May 2024 • 2 min read
Cloud adoption in 2024: Navigating AI, edge computing and the road beyond

Cloud adoption in 2024: Navigating AI, edge computing and the road beyond

CIOs are pursuing best-fit cloud solutions that avoid vendor lock-in

Eric Helmer
clock 09 April 2024 • 3 min read
WebAssembly heralds 'third wave of cloud computing'

WebAssembly heralds 'third wave of cloud computing'

Wasm: 'Speed and agility is the name of the game'

John Leonard
clock 26 March 2024 • 3 min read