Big data coming to everyday enterprise apps

Stephen Withers
Monday, 01 August 2011 15:35

Business IT - Technology

The term 'big data' is generally associated with science and a very few specialised areas of business. But one database company is predicting it will find a home in a much broader range of activities.

The emergence of 'big data' in areas such as astronomy, financial services and utilities is "changing profoundly" the world of database management, said Robert Nagel, vice president of software development at InterSystems. What's needed are systems that can handle the ingestion of massive amounts of well-structured data, persistence, rapid logging and checking, and then allowing analysis to begin while the data keeps accumulating.

Unlike traditional databases, InterSystems' Caché can ingest data at very high rates with low latency, and allow SQL processing while the data accumulates, he said.

Examples of its use include an ESA project to map one billion celestial objects that will require the processing of tens of thousands of 600-byte records each second. According to Nagel, Caché has been demonstrated to handle 90,000 records per second.

Another example is the third largest trading platform in the world (behind the New York Stock Exchange and NASDAQ), which runs at a large US investment bank. The bank previously used a system that was developed in house, but it sometimes failed near the end of the trading day, requiring a considerable recovery effort to identify which trades had been committed.

It was replaced with a Caché based system that delivered better than required latency, halved the infrastructure costs, and - most importantly -= removed the uncertainty associated with system failures.

CONTINUED