Big data is the new black

Beverley Head
Thursday, 23 February 2012 17:13

IT Industry - Development

Big data is going to prove the next major frontier for ICT researchers who stand poised on the 'cusp of a new age of wonder' according to Hugh Durrant-Whyte, chief executive of NICTA. The national research organisation is currently refining a tool, called Scoobi, now being tested by a range of organisations, including eBay, which allows rapid analysis of massive collections of distributed data.

Scoobi logically sits on top of the Hadoop big data platform. Ben Lever, senior research engineer on the project said that NICTA has been working with local banks which wanted to make sense of the data collections they were accumulating.

In terms of the scale of the data involved, Mr Lever said that a typical big bank might have a Terabyte of customer information on its databases, but internet banking could add as much as 500 Gb a day to the data collection. To make sense of that scale of information banks needed a completely different breed of data analysis.

Mr Lever said that the NICTA team had done a number of proof of concept engagements, although declined to say which local institutions had been involved. Besides the banks Mr Lever said that retailers and telecommunications companies were likely candidates to use such systems.

The open source tool which is currently in release 0.3 is suited at present to early adopters said Mr Lever who acknowledged there were still some bugs to be ironed out - eBay was one company which was currently 'playing with it' he said. Although NICTA would like to commercialise the tool, Mr Lever said it was not yet clear whether that might be through spinning off a company, or alternative means.

Speaking at the opening of the seventh annual NICTA TechFest, which showcases the organisation's research programmes, Professor Durrant-Whyte said that 'Big data is the new black.' The main challenge he acknowledged was to make sense of that scale and range of data types.

But he remained confident that; 'ICT will be the most important contributor to the efficiency and productivity of all industries.'