CA Technologies senior vice president for product management David Hodgson said there is a massive amount of data available that can be leveraged to reinvent a business. It may be data that is already being collected in-house, data that can be collected from outside the organisation, or data that can be purchased.
One example is Twitter traffic: "there's going to be some gold," he said, such as early warnings of problems with one of your products.
With 200 billion connected devices expected within the next several years, there's going to be "a tidal wave of data," according to CA executive vice president for enterprise solutions Amit Chatterjee.
All this data needs to be stored and processed, said Hodgson, and Hadoop "has got to be one of the game-changers." Hadoop has already gone from exotic to mainstream, and will be a US$50 billion market by 2020 according to one estimate, he said.
A survey carried out in April found that 55% of small and medium businesses (using the US definition of less than 1000 employees) and 70% of enterprises had either implemented at least one big data project or were planning one.
Hodgson thought this was probably an underestimate, as respondents would not necessarily have known about all the projects within their organisation - there is a lot of "rogue IT" in this area, he said.
Multiple projects and the natural evolution that occurs as experience is gained leads to a significantly complex big data environment: "There's a lot of moving parts here," he observed.
So CA has announced CA Big Data Infrastructure Management (BDIM), a tool that works across multiple vendors and tools. "It's a classic CA play," said Hodgson.
It provides a single pane of glass to manage all big data environments, because "you're not going to be standardised."
Hodgson showed how BDIM could be used to identify and fix the reason why a Hadoop job failed.
"We're halfway through the development of this," he said. "It promises to be a very practical tool … there's nothing like it on the market at the moment."
CA Technologies CTO John Michelsen said "managing big data is our birthright," in the sense that it is an area where the company's experience in mainframes and distributed systems can be applied to providing customers with the tools they need.
One way this will be achieved is by applying analytics to the "enormous amounts of data" collected by CA's tools, he said.
CA has no intention of competing with Hadoop or Cassandra, but will provide the management part just as it has for mainframes, Linux and other systems.
Disclosure: the writer attended CA World '14 in Las Vegas as a guest of the company.