Wednesday, 03 July 2019 05:08

Botched firewall update causes Cloudflare outage Featured

By
Botched firewall update causes Cloudflare outage Pixabay

A botched firewall ruleset deployment took down the content delivery network Cloudflare on Tuesday, generating 502 errors. It was caused by a huge spike in CPU utilisation on the company's network.

Cloudflare chief technology officer John Graham-Cumming said in a statement that the CPU spike was caused by the error in software deployment which was rolled back after the problem was noticed.

"Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels," he said.

Graham-Cumming said the outage had been noticed at 1342 UTC (11.42pm AEST 2 July) right across the entire network with all sites that used Cloudflare as a proxy showing a 502 (bad gateway) error.

He said this had been caused by a single misconfigured rule in the company's Web Application Firewall during a routine deployment of new rules.

"The intent of these new rules was to improve the blocking of inline JavaScript that is used in attacks," he said.

"These rules were being deployed in a simulated mode where issues are identified and logged by the new rule, but no customer traffic is actually blocked so that we can measure false positive rates and ensure that the new rules do not cause problems when they are deployed into full production."

But one rule had a regular expression that caused CPU usage to spike to 100% on all Cloudflare machines, with traffic dropping by 82% at the peak of the problem.

Graham-Cumming said the techies had understood what was happening by 1402 UTC (0.02am AEST 3 July) and stopped all the rulesets seven minutes later.

"We then went on to review the offending pull request, roll back the specific rules, test the change to ensure that we were 100% certain that we had the correct fix, and re-enabled the WAF Managed Rulesets at 1452 UTC (0.52am 3 July AEST)," he said.

On 24 June, Cloudflare was affected by a large-scale border gateway protocol leak cause by US telco Verizon, with the routes to many big websites instead transiting through DQE Communications, a small company in Pennsylvania.

At the time, Cloudflare's Tom Strickx said in a blog post that the problem had been magnified by the involvement of a so-called BGP optimiser product from a company known as Noction. The problems began at about 10:30 UTC (8.30pm 24 June AEST) and were sorted two hours later.

Verizon has not yet issued any explanation as to how it caused this issue.

Disclosure: iTWire uses Cloudflare's services.

LEARN HOW TO REDUCE YOUR RISK OF A CYBER ATTACK

Australia is a cyber espionage hot spot.

As we automate, script and move to the cloud, more and more businesses are reliant on infrastructure that has high potential to be exposed to risk.

It only takes one awry email to expose an accounts payable process, and for cyber attackers to cost a business thousands of dollars.

In the free white paper ‘6 steps to improve your Business Cyber Security’ you will learn some simple steps you should be taking to prevent devastating malicious cyber attacks from destroying your business.

Cyber security can no longer be ignored, in this white paper you will learn:

· How does business security get breached?
· What can it cost to get it wrong?
· 6 actionable tips

DOWNLOAD NOW!

ADVERTISE ON ITWIRE NEWS SITE & NEWSLETTER

iTWire can help you promote your company, services, and products.

Get more LEADS & MORE SALES

Advertise on the iTWire News Site / Website

Advertise in the iTWire UPDATE / Newsletter

Promote your message via iTWire Sponsored Content/News

Guest Opinion for Home Page exposure

Contact Andrew on 0412 390 000 or email [email protected]

OR CLICK HERE!

Sam Varghese

website statistics

Sam Varghese has been writing for iTWire since 2006, a year after the site came into existence. For nearly a decade thereafter, he wrote mostly about free and open source software, based on his own use of this genre of software. Since May 2016, he has been writing across many areas of technology. He has been a journalist for nearly 40 years in India (Indian Express and Deccan Herald), the UAE (Khaleej Times) and Australia (Daily Commercial News (now defunct) and The Age). His personal blog is titled Irregular Expression.

VENDOR NEWS & EVENTS

REVIEWS

Recent Comments