According to analyst firm Gartner, the continuing global expansion in big data technologies indicates organisations are starting to “get off the fence” about their big data investment, with the number of organisations indicating they had no plans for big data investment falling from 31% in 2013 to 24% in 2014.
The Gartner survey, conducted in June, was designed to explore organisations' technology investment plans relating to big data, stages of big data adoption, business problems solved, data, technology and challenges and compare the results with those from previous years.
"Big data investment continues to be led by North America, with 47% of organisations reporting investment, up from 37.8% in 2013," said Nick Heudecker, research director at Gartner. "All other regions experienced increases in investment over the last year."
"Last year, we said 2013 was big data's year of experimentation and early deployment," Heudecker said.
"So is 2014. In 2013, only 8%of organisations reported having big data projects deployed to production. This has increased to 13% in 2014, and while still relatively small, represents a sizeable increase. However, the 6% drop in organisations still gathering knowledge about big data and the 7% increase in pilots and experiments indicate that organisations are evolving in their understanding and willingness to explore big data opportunities."
According to Lisa Kart, research director at Gartner, big data can help address a wide range of business problems across many industries and “for the third year in our study, both enhancing the customer experience and improving process efficiency are the top areas to address."
"The most dramatic changes are in enhancing customer experience, especially in transportation, healthcare, insurance, media and communications, retail, and banking. Another area where we see an increase is using big data to develop information products, where organisations are looking to monetise their data. This is especially true among IT vendors, government and manufacturing."
Gartner reports that it continues to see strong investments and planned investments across all vertical industries with communications and media continuing to lead the pack with 53% of organisations surveyed having already invested and a further 33% planning investments in big data technology.
And, Gartner reports that the other year-to-year changes in the survey findings are a function of the adoption stage - as organisations move beyond knowledge gathering and developing a strategy to making investments, piloting and deploying, the challenges they face become more practical.
Those organisations with no big data plans, according to Gartner, feel the big hurdles are determining how to get value from big data, defining a strategy, leadership or organisational issues, and even still trying to understand what big data is.
The research firm says that in the planning stages, beyond determining value, the top challenges are obtaining skills and capabilities needed, defining strategy, obtaining funding, and beginning to think about infrastructure issues, and that companies that are further along with investments “must begin to address risk and governance issues, data integration and infrastructure”.
According to Gartner, the challenges introduced by analysing a variety of data sources may explain why most organisations are studying traditional data sources for their big data projects:
“Those organisations analysing transactions increased from 70% in 2013 to 79% in 2014, while those analysing log data fell slightly by 2%. Interestingly, both types of social media data sources — profiles and interactions — fell over the last year, which may be caused by the difficulty in integrating social media data sources with other types of data, such as transactions,” Gartner explains.
"We got a surprising result when we asked respondents which data sources they planned on adding in the future," Heudecker says.
"Every data source received roughly 30% to 40% of responses, including extremely challenging data sources like audio and video. This overly optimistic and apparently random nature of future data sources for analysis indicates two things. First, organisations don't have a plan for what they're going to do next. Picking everything isn't a strategy. It indicates a fear of missing out on an opportunity yet to be defined. Also, there may be a certain amount of hubris at work. If organisations can 'do big data' on transactions and log data, they may assume they can also leverage more challenging data sources as easily."