It was as if they chopped every cable to Sweden

David Heath
Wednesday, 14 October 2009 10:55

Your IT - Home IT

Yesterday, a DNS configuration error left the entire .SE domain adrift on the Internet sea.

According to the Royal Pingdom Blog "the central record for the .se top-level domain, broke at 21:45 local time and was not returned to normal until 22:43 local time."  Note that this was around 7:45am – 8:45am in south eastern Australia on Tuesday 13th.

"There are just over 900,000 .se domain names, and every single one of these were affected."

However, due to widespread DNS propagation, the error for many locations around the world wasn't rectified until much later than the moment it was fixed.

In fact any ISP anywhere in the world could possibly still have broken connections if they have still not flushed their DNS caches.

The .SE registry announced the problem: ". SE (The Internet Infrastructure Foundation) sent out an incorrect zone file Monday October 12 at 21.45, in connection with a planned maintenance work. The cause was an incorrect software update, which, despite our testing procedures were not detected. Thanks to well-functioning surveillance system .SE discovered the error immediately and a new file with the DNS data (zone file) was produced and distributed within one hour.

"To minimize the impact on the availability of .se domains, we chose to produce and distribute a zone file that lacked proper signatures for DNSSEC to quickly come out with the correct DNS information. Subsequently, we generated immediately initiated a zone file according to normal procedures. It was fully distributed 01:00 am.

"The false information that was sent out affected accessibility to all .se domains for a short time. However, there may still be some name servers that have not changed out of misinformation against the real.

"BIND resolver can be forced to refresh the .SE-zone by issuing the command "rndc flush" or by doing a restart. Similar commands exist for other resolver softwares.

"Right now is an ongoing internal investigation to find out the cause of the faulty software update so that we can improve our procedures further."

Pingdom also notes that "Pingdom monitors the uptime of tens of thousands of websites for our customers, and we often see downtime due to DNS problems. These problems are very common all over the world, but usually it's a single domain name that has been incorrectly configured or the DNS servers of a single web host having problems. An entire top-level domain breaking is exceptionally rare."

"Imagine the same thing happening to the .com domain, which has over 80 million domain names."

One can only hope that all other domain registrars are reminded of this event and re-double their checking and testing procedures to ensure they too do not succumb to this kind of error.