A survey released earlier this week by IDC in association with EMC revealed that the world is now generating data at a rate of 5 exabytes every two days. But at present only a half of one per cent of the world’s available data is being mined for value.
It’s a wasted opportunity according to research published mid-year by the Economist Intelligence Unit which claimed that there was a 26 per cent business improvement to be reaped by organisations which deployed big data analytics.
Franz Aman, chief marketing officer for supercomputer company SGI, this week said that although big data analysis had been championed by Government agencies “analysing information on the internet, a lot of signal processing and video processing” for anti terrorist and security purposes, big data was now making its way quickly into the corporate realm. Gartner has already forecast that in 2013 big data will drive $US34 billion worth of IT investment.
Mr Aman said that while data analysis provided valuable insights in 2012 he expected to see technology installed in the future which would “allow you to transact on big data the way you transact on unstructured data.” He said he was aware of a company in the financial services sector which had built applications for traders which analysed commentary in social networks and allowed traders to act based on what the customers of companies were saying in real time.
One of the challenges that organisations may face however is access to skills to analyse and harness insights from big data. Anil Kaul, chief executive officer of AbsolutData which is a marketing analytics firm based in San Francisco, has located 250 of its 300 staff in India due to a paucity of skills elsewhere.
Dr Kaul said that data analysts emerged from the STEM – science, technology, engineering and mathematics - disciplines, and in many countries access to those skills was increasingly limited. There remained however a good supply in India he said.
He pointed to a McKinsey & Co report which had identified as much as a 190,000 data analytics skills shortfall in the US alone.
The company, which is currently in discussions with an Australian telecommunications company about a big data project, has recently secured $US20 million funding from Fidelity which it will use in part to fund an international expansion.
Starting with Singapore and London in January, Australia is one of four countries being considered for expansion beyond that.