Human error increasingly causes enterprise data ‘disasters’

Peter Dinham
Wednesday, 18 November 2009 08:32

Human error is responsible for an increase in enterprise data ‘disasters’, a trend which data recovery vendor, Kroll Ontrack, attributes to more complex storage systems coupled with depleted resources to replace equipment, train IT staff and maintain optimal staffing levels.

According to Kroll Ontrack general manager APAC, Adrian Briscoe, the company has found that human error is responsible for an increasing number of incoming enterprise data recovery requests, and he says that while advanced storage options such as virtualisation and cloud computing offer corporations storage optimisation, “human processes are still at the root of these solutions, instructing the technology as to how to perform.”

“The complexity of these systems often require a steep learning curve.  Human error is increasingly common.”

Kroll Ontrack says the most common enterprise human error cases include:

•    Pulling the wrong drive. While trying to replace a failed disk in a RAID array, a healthy disk is accidentally removed

•    Reformatting a disk. During a server migration, the wrong SAN LUN is accidentally reformatted

•    Restoring corrupt/old backup data. A server containing a business-critical database is deleted by mistake and is restored with a corrupt or incomplete backup prior to realising the backup is not sound

•    Rebuilding a bad array. Following a multiple drive failure in a RAID array, an attempt to force the failed drives back online and rebuild the configuration is made, whereby damaging or corrupting the data on the array

•    Deleting data. Files, volumes, virtual machines or a SAN LUN is deleted by accident and there is no backup or the backup is old or corrupt.

Kroll Ontrack lists a number of examples of human error cases and subsequent enterprise data recoveries this year, which Briscoe says included:

•    A support engineer forgot to turn off his replication software before formatting the volumes on the primary site. Unfortunately, this mistake resulted in overwriting the backup.

•    An entity with a 10 drive Raid 5 array had a drive die without notice for three months. When a second drive died, the server crashed, deeming all data unavailable. Before attempting to force the drives back online and ultimately corrupt all the data on the volume, the customer called Kroll Ontrack. Ontrack Data Recovery engineers rebuilt the array, making a full recovery possible

•    An organisation accidentally ran a script during a test project that deleted all 38 virtual machines from two arrays. Kroll Ontrack was able to connect to the organisation remotely and recover the deleted data while also copying the virtual machines to a new array

•    And, a company leasing cloud computers accidentally detached a “virtual” storage volume in the cloud environment – similar to pulling a cable from an operational volume. When it was reconnected, Windows reported the volume as unallocated space, so the volume could not be mounted. Using Kroll Ontrack’s proprietary Remote Data Recovery™ service, Ontrack Data Recovery engineers were able to repair the damaged cloud storage volume, getting the development company back up and running in four hours.