Germaine's posting is available online. Yet, the underlying story presented by the secretary reveals a level of organisation perhaps rivalling the Keystone Cops.
Linux Australia is populated with a membership that may contain some hobbyists and beginners, but for the most part it would be reasonable to think has a sizeable proportion of smart, experienced technology professionals.
Yet, despite this, and contrary to the organisation's desire to be seen as the "peak body" for Linux and open source in Australia, the web hosting was not managed to any standard the members would apply to their daytime jobs.
In November 2015 Linux Australia learned — how is not specified — VPAC's parent body, V3 Alliance, a Victorian eResearch co-operative, was closing. Linux Australia contacted VPAC — you will note VPAC did not contact Linux Australia — and identified that VPAC's funding ran to the end of the current financial year, namely June 30, 2016.
Linux Australia's Council and Admin teams determined that new hosting was needed, and the Admin team commenced discussions with a Victorian university seeking hosting space.
In early December, VPAC reached out to Linux Australia, and advised that most staff had moved on and VPAC would be auspiced by the University which had hosted its offices and equipment from January 1, 2016.
The Admin team sought to have VPAC identify the servers used by Linux Australia in the hope these would not be prematurely turned off by the University.
The Admin team purchased new server hardware, yet discontinued its search for a new hosting donor until March 2016. Germaine does not state why this work paused for two months despite the organising committees knowing the urgency of the matter. It is also not stated why Linux Australia did not seek commercial hosting space, and why it purchased server hardware — with no home — when virtualised hosting options exist.
At 3pm on April 19, 2016 disaster hit and Linux Australia's Admin team received notifications that two servers at VPAC, and the virtual machines running on them, were no longer available. It was considered the servers had been powered off.
It seems to be evident that Linux Australia, despite knowing the situation at VPAC, did not actually engage with the University itself at any point.
A former VPAC employee was approached and he explained the site was in the process of being decommissioned. This former employee was able to provide contact details for the contractor; the contractor was reached, and this person agreed to make the server available for two hours.
Linux Australia used this window to commence transferring services from these servers to alternates. The detail is described in the mailing list posting. Yet, it is noted that at the time of posting some services were still not online, and some services were still dependent on the goodwill of others providing hosting.
Germaine states that this happened despite Linux Australia undertaking "all possible means" to ensure the equipment was identified as belonging to the organisation.
However, this does not stand to reason. Firstly, as Germaine notes, the ownership was mistakenly attributed to a Victorian Linux user group. Secondly, Linux Australia did not have any contacts at the University but only ex-staff at VPAC. It is as if Linux Australia, in a fit of digital autism, felt that putting a sticker on a server would save them having to actually speak to any human at the University or the V3 Alliance, choosing to solely work through staff whom they knew would soon be no longer in any position to assist. It is telling that neither the University nor the contractor reached out to Linux Australia, and is likely neither party were aware — at least by Linux Australia's hands — that they were hosting such equipment.
Germaine also does not explain why the search for a new host was abandoned for two months, despite knowing how perilous the situation was, with the Admin team consciously choosing to leave its valuable infrastructure in a rack in an organisation that was knowingly being dismantled, guarded solely by a sticker on the server.
At the time of writing Germaine was also unable to state if data has been lost, raising doubtsabout the integrity of back-ups.
Now, Germaine does rightly raise the question why this keeps happening to Linux Australia, with the explanation that the organisation relies on the goodwill of the community or commercial organisations to host server infrastructure, noting that these are not binding and may change as commercial drivers change for those businesses on which Linux Australia relies.
While one can appreciate Linux Australia is not at heart a commercial organisation, it seems an unacceptable risk and even very cavalier that such an attitude toward hosting vital services has developed at all.
Ultimately, despite the rhetoric of goodwill, of adopting a policy of owning hardware (but not paying for hosting?), and so forth, the picture that emerges smacks of a scenario where Linux Australia had its servers in unknown hands and only initially found out about it by accident. Once knowing for certain, no serious action was taken. Oh, yes, the Admin team determined to find new hosting. Then they didn't do so. The servers were left in an environment where it was predictable they would be powered off, but actions were not taken at that time, nor for two months, and ultimately services were lost.
The organisation has chosen, despite the skillbase of its members, to set something up with band-aid solutions in the name of keeping costs low, while at the same time making capital purchases of hardware. These solutions have depended on specific individuals being employed for the continuity of the hosting, and when those persons have moved on, there would appear to be little or no documentation. There appears to have been no communication with the parent organisations when key contacts were leaving. There even appears to be uncertainty over the reliability of back-ups.
Given Linux Australia largely exists for the sole purpose of running its annual Linux.Conf.Au conference (and giving some grants to various purposes through the year, but certainly not for anything like, say, seeking to influence business or Government), and given the organisation always make a profit from the conference, it seems unbelievable the concept of paying for a reliable virtual hosted server environment is totally dismissed by the organisation.
Financial statements for the 2015 financial year are also publicly available and reveal a profit of $143K over the last year. Interestingly, the organisation pays for "storage", but not the electronic storage of its own servers. Indeed, year after year, Linux Australia spends little — because let's face it, it sees its mission as fundamentally to run an annual conference — and raises more money from that same conference.
Why professional, paid hosting is ruled out — in favour of capital expenditure and communication-lacking documentationless favour-reliant hosting — seems a bizarre and risk-laden omission.
This is not the first occasion Linux Australia has experienced reliability issues, as indicated by Germaine's rhetorical question "Why does this keep happening to Linux Australia?" and as documented by iTWire colleague Sam Varghese.
Under the organisation's current thinking about server hosting it will not be the last, either.