Dodo explains national Telstra outage

David Heath
Thursday, 23 February 2012 22:19

Business IT - Networking

In a statement released yesterday evening, Dodo tells their side of the outage story.

Dodo has investigated the cause of a national network outage earlier today. As best Dodo can determine, this is what occurred:

Dodo experienced a hardware issue with a Cisco border router. This issue caused Dodo to broadcast network routes to Telstra. In normal circumstances, this would not result in a network outage. However, it appears that these routes were accepted by Telstra and propagated to Telstra's downstream customers rather than Telstra simply filtering the routes. This caused major issues for Telstra and its customers which should have been avoided.

Here are the questions that remain to be answered:

1. What exactly was the 'hardware issue' that caused this routing error?  Border routers are very big, very robust, very expensive beasties that don't (as a rule) have hardware issues.

2. Why was the router (and Telstra's neighbouring router) configured such that route updates were transferred in what was essentially the wrong direction; as a downstream customer, Dodo's network should never be in a position to have updates that Telstra's systems don't already know about.

3. How quickly was Dodo able to advise Telstra what happened?  What remediation steps were taken by Dodo and Telstra?  Did this require a restart of Telstra's main routers?  What steps were taken to ensure that the route updates were not propagated from Telstra into the wider Internet?

4. Who picks up the tab for SLA violations?

iTWire will be raising these questions with both Dodo and Telstra.