Third time unlucky: Primus data centre down again

James Riley
Monday, 10 August 2009 14:47

Your IT - Home IT

Primus Telecom has again been forced to apologise to customers as its Melbourne data centre crashed for several hours Sunday evening – the third time the facility has failed this year.

In an email statement to customers, Primus said the problems started with a fuse failure at the CitiPower substation that feeds the facility, but conceded its own diesel-powered back-up generators also failed to start.

The outage affected business and residential customers – including meant internet service providers like Internode, iiNet and WestNet were also offline. Affected customers were primarily in Melbourne and Tasmania.

Ironically for Primus – and just plain bad luck for its customers – the data centre’s CitiPower substation is scheduled for upgrade this weekend in preparation for what is expected to be a difficult summer for the Melbourne power grid.

Primus chief executive officer Ravi Bhatia told iTWire the substation upgrade had been planned as a direct result of a previous data centre failure in February . The centre also failed in April.

“We had a problem back in February, so we did a complete audit of the system and decided then to add another diesel generator, and to double the capacity of the substation," Bhatia told iTWire.

“And these things take time. The substation isn’t ours, it belongs Citipower. You have to really push, and we've paid a lot of money. “But (the failure) simply should not have happened. We have spent a lot of money upgrading the facility and come next Monday, it can’t happen – because the whole substation will have been upgraded.

The company said in a statement that high tension fuses in the CitiPower substation that feeds the data centre failed late Sunday afternoon at about 3.45pm.

“We have experienced a similar issue several months ago and are in the process of executing a plan to resolve this issue,” the statement to customers said.

“One of the emergency diesel generators failed to start due to a synchronization processor failure. Contents of the synchronisation processor's register are being analysed to ascertain the true cause of failure.”

After the failure of its synchronisation processor, Primus technical staff said they had successfully tested a “controlled failover” to the generators after implementing a temporary work around.

Primus says the substation upgrade on August 14 and 15 will include the installation of both new high-tension switchgear and transformer.
 
It says the data centre will switch to its diesel power prior to the substation work commencing, and “an additional emergency diesel generator which is capable of carrying the entire load by itself will also be a part of the system.”

“The facility will be fully staffed around the clock during this procedure.  Customers have previously been informed of this work and ongoing communications with the customers will continue,” the company said.

Primus said the data centre was offline for two hours, although customers in online forums say it was closer to four hours.

Clearly unhappy about the outage, Bhatia told iTWire: “We believe in excellent, we will provide excellence, we are committed to excellence, and I will leave no stone unturned to achieve that.”

Bhatia had announced a multi-million expansion and upgrade of the Melbourne Data centre in early April.