(Disclosure: I have been a Debian user for the last eight years and currently run the AMD64, x86 and MIPS ports of the distribution).
What made the situation even worse was the fact that the bug was introduced as a result of a Debian-specific change made in September 2006.
This change resulted in the random number generator in the OpenSSL package being predictable. Key generation was limited to about 32000 different unique keys, a rather small space when it comes to brute-force searches.
Within OpenSSL, the valgrind memory management profiler can use uninitialised memory as a potential source of entropy/randomness; the change introduced by the Debian developer removed two lines of code, with his intention being that the profiler would stop complaining about the improper use of uninitialised memory.
While this was achieved, the removal of the second line also removed all sources of entropy apart from the process ID which limited the number of unique keys to that given above.
There are a few things to be noted here. The Debian developer in question, Kurt Roeckx, sent a message to the openssl-dev mailing list on May 1, 2006, titled "Random number generator, uninitialised memory and valgrind", proposing the changes which he wanted to make - the commenting out of the two lines of code. He also mentioned that he had no idea what effect this would have on the random number generator.
(It must be noted that the two lines of code were similar - and the removal of the first actually did away with mixing uninitialised memory into the pool. The developer assumed that the second occurrence did the same function as the first and removed it as well. That caused the problem. Neither line of code was commented in the original source.)
In reply, OpenSSL developer Ulf Moller responded that if it helped with debugging, then he was in favour of removing the two lines.
Moller's reply can be interpreted two ways - one, that this meant that an OpenSSL developer was okay with the change. A second school of thought, which includes long-term Debian developer Russell Coker, says that since Roeckx had begun his message by saying, " When debugging applications that make use of openssl using valgrind...", Moller may well have understood his (Roeckx's) reference to removal of code as meaning that the removal was only for the purpose of debugging and not as a final change.
There were three responses to Roeckx's post; apart from Moller, a second OpenSSL developer, Geoff Thorpe, suggested that compiling the package with the -DPURIFY option would remove the unnecessary warnings generated by valgrind.
It turns out that Roeckx had sent this message to the wrong mailing list - but nobody can blame him for doing so, for the OpenSSL website states that this list (openssl-dev) is for "Discussions on development of the OpenSSL library. Not for application development questions!"
A post by Ben Laurie (see comment 43), a member of the core OpenSSL team, stating that if Roeckx wanted to communicate with OpenSSL developers he should have sent his message to the openssl-team mailing list, would have had some merit if it had been indicated anywhere on the OpenSSL site that this (openssl-team) was the mailing list that would ensure communication with the OpenSSL developers.
As former Debian project leader, Branden Robinson, pointed out, nowhere, including in the OpenSSL package itself, is there any mention of the openssl-team mailing list as being the one which ensured communication with developers. In fact, there is no mention of the mailing list at all.
However, Laurie did point out one mistake made by the Debian project - that the changed OpenSSL package, which fixed the bug, was committed to a public repository on May 7, nearly a week before the advisory about the vulnerability was issued. As Laurie pointed out; "This gives alert attackers a big hint (and only one needs to take that hint) without warning defenders of the problem (all of whom have to know)."
Russell says this happens periodically when there are upstream issues and cross-distribution advisories are synchronised.
The appearance of a report on May 13 titled "Brute-Force SSH Server Attacks Surge" may not be unrelated to Laurie's comment.
The implications of the bug are huge and systems administrators are not to be envied at this point in time. Craig Sanders, an extremely competent sysadmin based in Melbourne and a long-term Debian developer, said that he had all of his 15 hosts updated in half a morning - but this was because the non-signed certificates he had were generated on a non-Debian webserver.
For those looking after hundreds of servers, the process will be much more time-consuming and nerve-wracking; and depending on the number of certificates generated using vulnerable systems, they may be looking at months or even years of work to track down and change certificates.
"The process is essentially to upgrade to the latest OpenSSL and OpenSSH packages; regenerate any vulnerable keys; fix any known_hosts and/or authorised_keys files; replace any vulnerable SSL certificates (e.g. for SSL web server, TLS mail server, SSL IMAP and POP, etc) and repeat for every machine/service that is affected," Sanders said.
Peter Giorgilli, a longtime UNIX developer and systems engineer, said the whole process had shone a light on what had happened in an open source shop. "At least, you know how the sausages are made," he said.
"On the other side, there could be similar, or worse, things happening in a closed source shop. You never hear about them - and you still end up eating the sausages," he observed.
The Debian fix, issued on May 13, also included a link to a detector for known weak material and also a link to detailed instructions on how to roll over keys generated by the vulnerable versions of OpenSSL.
Russell, who is well known for his work on the security-enhanced Linux project, said the new Debian OpenSSH packages would check for and reject broken keys. "I am not aware of other distributions doing this, so if you run Linux servers that are kept up to date then running Debian may be regarded as being more secure because of this," he added. Upgraded servers also refuse log-ins from unsafe client keys.
However even in the darkest of moments, geeks do have a sense of humour - and this bears testimony to that.
That said, there are some lessons from this episode. The corporate world benefits enormously from the labours of a great many talented hackers, who churn out software at no cost and then give it away. It is high time that some of these leeches look at giving something back so that these hackers are able to devote some more time to their pet projects. After all, even the most motivated software developer has to eat. Projects like OpenSSL provide good quality software, as good or better than commercial alternatives. More time means the chance for more QA and more code audits; it means better quality and better security.
The same goes for the Debian project too. It could do with some more corporate involvement in the shape of jobs for developers, the same developers who keep the distribution going year after year.
And, finally, the OpenSSL fiasco is a reminder to all that a very simple thing can defeat the best of technology - remember, once a man with an ordinary hoe cost a goodly portion of the US its internet access. And it only took one anchor to cut a cable and keep most of South-East Asia off the net for more than a week.