How two of the world's largest websites use Linux for high availability

By David M Williams
Monday, 10 November 2008 16:53

Opinion and Analysis

Pop quiz: you have a web site and you want it to be popular. It must scale to tens, hundreds of thousands, even millions of visitors. It has to be snappy and responsive. What server platform will you host it on? Here’s what two of the world’s most popular sites – Wikipedia and Digg - went with, and it wasn’t Windows.

I have no doubt whatsoever you know about Wikipedia and an introduction is probably unnecessary. This is a comprehensive online encyclopaedia that is freely editable, which ensures currency of content while also being controversial for potential abuse.

Wikipedia is rated by Alexa as having a traffic rank of 8. Alexa is an independent website monitoring company who provide site ranking information based on a variety of sources over a rolling three month period. Primarily, they measure the number of individual pages visited by individual people.

By rating Wikipedia an ‘8’ Alexa are saying that Wikipedia is the 8th most popular web site out of every site the company collects data on. For contrast, my personal blog – low traffic, little content, updated infrequently – has a paltry rank of 7,476,670. This site, iTWire.com, is ranked 39,326. If you’re interested, here’s an interesting article comparing Wikipedia’s Alexa ranking against many, many other web sites.

If you have a site as massive as Wikipedia, hit as many times a day by as many people as it, what are you going to do? We’re talking more than 10 million distinct articles, in 250 different languages, served to over 684 million people per year.

The answer is that Wikipedia have a massive server farm driving their web site – some 400 servers, in fact. And each and every one runs Linux.

Previously, Wikipedia had a mix of Red Hat and Fedora Linux installations. These aren’t unsurprising choices. Red Hat Linux is a very well known and popular enterprise-grade server product. Fedora is also by the same company; it is Red Hat’s community-supported distribution which omits some of its big brother’s grunt.

You’ll appreciate that the systems administration burden of managing upgrades and patches and software compatibilities is made easier if every server has the exact same platform. Therefore, Wikipedia decided to migrate every server to a single Linux distribution across the board. They chose Ubuntu Linux, which is arguably the most popular Linux release today.

Now, here’s something I really want to hit on: how many systems administrators does it take to manage 400 Linux servers? Let me tell you!

CONTINUED