Security Market Segment LS
Tuesday, 05 October 2021 18:38

Ten hours after outage ended, Facebook has little to say Featured

Ten hours after outage ended, Facebook has little to say Pixabay

Social media behemoth Facebook has released its first post about how the site and its associated properties Instagram, WhatsApp and Oculus Web went down for six hours on Tuesday AEDT – but reading it is of little use.

Apart from stating the bleeding obvious, that the problem was caused by a BGP misconfiguration — which has been known by world+dog for more than a few hours — Facebook's Engineering and Infrastructure vice-president Santosh Janardhan said little, apart from offering an apology.

The outage, the second longest in the company's history, occurred from about 2.30am AEDT to about 8.20am AEDT on Tuesday.

While Facebook, a trillion-dollar company, could not offer a decent explanation, a much smaller firm, Web infrastructure and website security company Cloudflare, published a detailed blog post two hours after the outage was resolved.

Janardhan said: "Our engineering teams have learned that configuration changes on the backbone routers that co-ordinate network traffic between our data centres caused issues that interrupted this communication.

"This disruption to network traffic had a cascading effect on the way our data centres communicate, bringing our services to a halt."

And then, he offered some platitudes: "Our services are now back online and we’re actively working [as opposed to passively working?] to fully return them to regular operations.

"We want to make clear at this time we believe the root cause of this outage was a faulty configuration change. We also have no evidence that user data was compromised as a result of this downtime."

Facebook's last post on Twitter was 14 hours ago. Its media blog and its so-called newsroom haven't changed from what iTWire reported earlier; the latest item on its newsroom, however, does have a very fetching picture of a woman in sunglasses!

iTWire has already posted a story quoting from the Cloudflare blog post, posted by Tom Strickx and Celso Martinho.

But in the absence of any details from Facebook, here is a little more from what one can only describe as an excellent post:

"BGP stands for Border Gateway Protocol. It's a mechanism to exchange routing information between autonomous systems (AS) on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn't know what to do, and the Internet wouldn't work.

"The Internet is literally a network of networks, and it’s bound together by BGP. BGP allows one network (say Facebook) to advertise its presence to other networks that form the Internet. As we write Facebook is not advertising its presence, ISPs and other networks can’t find Facebook’s network and so it is unavailable.

"The individual networks each have an ASN: an Autonomous System Number. An Autonomous System (AS) is an individual network with a unified internal routing policy. An AS can originate prefixes (say that they control a group of IP addresses), as well as transit prefixes (say they know how to reach specific groups of IP addresses).

"Cloudflare's ASN is AS13335. Every ASN needs to announce its prefix routes to the Internet using BGP; otherwise, no one will know how to connect and where to find us."

There's lots more and Cloudflare's learning centre has an overview of what BGP and ASNs are and how they work.

But then one should not be surprised by Cloudflare providing such a lucid explanation when it was sorely needed; the company's chief executive, Matthew Prince, has a history of being open about routing errors that cause pain to ordinary users, even when they are caused by Cloudflare itself.

There have been a number of snafus involving BGP over the last few years: Google suffered in November 2018; Telstra took down a good part of the Internet in Australia the same month; a Verizon route leak took place in June 2019; a routing error by a Russian provider hit many big sites in April last year; and Telstra proved that one good turn deserves another by stuffing up again in October 2020.

Read 1475 times

Please join our community here and become a VIP.

Subscribe to ITWIRE UPDATE Newsletter here
JOIN our iTWireTV our YouTube Community here


Thoughtworks presents XConf Australia, back in-person in three cities, bringing together people who care deeply about software and its impact on the world.

In its fifth year, XConf is our annual technology event created by technologists for technologists.

Participate in a robust agenda of talks as local thought leaders and Thoughtworks technologists share first-hand experiences and exchange new ways to empower teams, deliver quality software and drive innovation for responsible tech.

Explore how at Thoughtworks, we are making tech better, together.

Tickets are now available and all proceeds will be donated to Indigitek, a not-for-profit organisation that aims to create technology employment pathways for First Nations Peoples.

Click the button below to register and get your ticket for the Melbourne, Sydney or Brisbane event



It's all about Webinars.

Marketing budgets are now focused on Webinars combined with Lead Generation.

If you wish to promote a Webinar we recommend at least a 3 to 4 week campaign prior to your event.

The iTWire campaign will include extensive adverts on our News Site and prominent Newsletter promotion and Promotional News & Editorial. Plus a video interview of the key speaker on iTWire TV which will be used in Promotional Posts on the iTWire Home Page.

Now we are coming out of Lockdown iTWire will be focussed to assisting with your webinars and campaigns and assistance via part payments and extended terms, a Webinar Business Booster Pack and other supportive programs. We can also create your adverts and written content plus coordinate your video interview.

We look forward to discussing your campaign goals with you. Please click the button below.


Sam Varghese

Sam Varghese has been writing for iTWire since 2006, a year after the site came into existence. For nearly a decade thereafter, he wrote mostly about free and open source software, based on his own use of this genre of software. Since May 2016, he has been writing across many areas of technology. He has been a journalist for nearly 40 years in India (Indian Express and Deccan Herald), the UAE (Khaleej Times) and Australia (Daily Commercial News (now defunct) and The Age). His personal blog is titled Irregular Expression.

Share News tips for the iTWire Journalists? Your tip will be anonymous




Guest Opinion

Guest Reviews

Guest Research

Guest Research & Case Studies

Channel News