Home Business IT Technology CSIRO wrangles extra 5TB a day
Get all your tech news delivered to your mail box five days a week
iTWire UPDATE - it's FREE!

On a typical day the CSIRO adds 5 terabytes of data to its advanced scientific computer centre’s storage systems; it’s now up to 2.5 petabytes of data stored in 23 million files. To cope with the growth – but also rein in costs – the research organisation has just installed an SGI MAID (Massive Array of Idle Disk) which has the capacity to hold 1.7 petabytes of data.

Accounting for just 260 working days a year, 5 terabytes a day quickly mounts up to 1.3 petabytes a year – meaning CSIRO’s data bank will double in less than two years. While it seems an enormous growth rate – it’s smack in the middle of Gartner’s forecast for all industry sectors which is 40-60 per cent data growth per year.

Dr Robert Bell, technical services manager for the CSIRO’s advanced scientific computer centre, says that at present the CSIRO’s storage systems have enough headroom to handle that level of growth. The SGI Copan MAID system was brought into production at the CSIRO with 870 Tbyte formatted capacity in September – with the ability to build that out to 1.7 Petabytes.

Dr Bell said the MAID layer of storage helps to keep running costs under control – but also provide much faster access to data than is possible using tape. (CSIRO has access to 30 Petabytes of tape storage should it need it).

Two decades ago CSIRO started using Cray’s Data Migration Facility – a system designed to allow data storage to be sensibly tiered. That technology was inherited by SGI when it bought Cray, and since been redeveloped, but remains a fixture at the CSIRO.

Dr Bell said that it allowed the organisation to store its data collections across four levels of storage – fast expensive disk; slower cheaper disk; the new MAID; and tape. Sitting in front of these four levels of storage was an SGI UV system with 512 cores and 4 terabytes of memory to perform the initial data analysis.

One of the clear benefits of the tiered approach was the reduced power costs according to Dr Bell.

He said that the storage collection, housed in its Melbourne Docklands facility, today consumed about $13,000 worth of electricity each year. Had the entire collection been housed on fast expensive disk the electricity bill alone would have blown out to $500,000 a year.


Avoid War Room Scenarios and improve handling of critical application problems:

• Track all transactions, end-to-end, all the time and know what your users experience 24/7

• View code level details with context and repair problems quickly

• Fix problems in minutes before they wreak havoc

• Optimize your most important applications, Java, .NET, PHP, C/C++ and many more

Start your free trial today!



Where are your clients backing up to right now?

Is your DR strategy as advanced as the rest of your service portfolio?

What areas of your business could be improved if you outsourced your backups to a trusted source?

Read the industry whitepaper and discover where to turn to for managed backup


Beverley Head

my space counter

Beverley Head is a Sydney-based freelance writer who specialises in exploring how and why technology changes everything - society, business, government, education, health. Beverley started writing about the business of technology in London in 1983 before moving to Australia in 1986. She was the technology editor of the Financial Review for almost a decade, and then became the newspaper's features editor before embarking on a freelance career, during which time she has written on a broad array of technology related topics for the Sydney Morning Herald, Age, Boss, BRW, Banking Day, Campus Review, Education Review, Insite and Government Technology Review. Beverley holds a degree in Metallurgy and the Science of Materials from Oxford University and a deep affection for things which are shaken not stirred.