"Big data is happening right now," the Asia Pacific director of Oracle's specialist architecture team Babar Jan-Haleem told a Melbourne audience this morning. Around two-thirds of data growth is unstructured rather than structured, but the ability to analyse it is what makes it possible for IT to change a business.
Big data technologies can quickly deliver answers to new questions, provide more accurate predictions, allow the construction of a 'data reservoir' to retain the maximum amount of data for possible future use, and accelerate the adoption of data-driven architectures, he suggested.
It can be difficult to cost-justify the storage of massive amounts of data without an immediate purpose, but some of the value of big data comes from "letting the data tell you what it can do for you," Mr Jan-Haleem said.
Ambulance Victoria enterprise architecture manager John Dousset outlined the way his organisation is using Oracle products to make use of the data collected by the tablets used by paramedics and from other sources.
Putting responsibility for enterprise architecture, enterprise integration and business intelligence into one department proved very successful, he said, as it made it easier to take a very strategic approach ensuring that every project was done properly from an organisation-wide perspective, even if that seemed like overkill from the point of view of the particular department concerned.
For example, three departments each had a project involving in-memory analytics, and they were each considering a different vendor. Mr Dousset opted for a single enterprise data warehouse, and to stay within his limited resources carried out a narrow but deep implementation for one department at a time, an approach he described as 'laying the train tracks'.
Page 2: more local big data projects
It is important to be disciplined so that everything you do moves the organisation in the right direction, he told iTWire. Creating a data model for the whole of Ambulance Victoria and building a single data warehouse meant more work than was necessary for any single project, but it reduced the overall effort.
Another part of the story was the implementation of the Oracle SOA suite to provide a service bus supporting integration with other organisations, in line with state government policy about making data publicly available. The same service bus is also used for internal application integration.
The outcomes included a dramatic improvement in the time taken to generate reports. For example, one that took 12 minutes on the previous system could be produced in 12 seconds from Oracle Business Intelligence Enterprise Edition, and that was reduced to less than 2 seconds once the Oracle Exalytics In-Memory Machine was installed.
Other benefits included 70% faster report deployment (partly because Ambulance Victoria can now store data at the transactional level instead of aggregating it), an 80% reduction in the number of basic reports through consolidation ("that's really important," Mr Dousset said), a 66% saving in IT project costs, and staff are able to do 'what if' analysis on the main system rather than using Excel and other tools.
Consequently, some 500,000 'events' are logged by Ambulance Victoria every day, and there are currently 71 research and development programs in progress - "we're world leaders in research into pre-hospital outcomes," Mr Dousset said, pointing out that where some ambulance services are primarily concerned about getting patients to hospital within a certain time, Ambulance Victoria's key performance indicator is patient outcome.
One way that live data is being applied is to provide hospitals with information about ambulances and patients that are heading their way. The ambulance arrivals board in hospitals is now a smart TV that displays a web page. Rather than having to create a separate application, Ambulance Victoria simply generates an HTML report for each hospital showing the required information.
Ambulance position data is also used for route optimisation, along with live feeds from Google and Nokia, he said.
Page 3: more big data projects from the Victorian public sector
How much data? 32 million records, 30 million emails, 25 million documents, and 70,000 web pages.
"Problems reveal themselves in the data" so it can be difficult to prepare a business case in advance, Mr Harridge told iTWire, echoing Mr Jan-Haleem's observation. If predetermined outcomes don't provide sufficient justification, case studies from other organisations with similar problems may tip the balance in favour of proceeding. But insights come from data, he said, so an agile and innovative approach is needed with big data projects.
This might take the form of a cloud-based pilot, he suggested. Such an experiment can validate a proposed project, and with little as two weeks work might reveal that there really is a business case for funding the full project.
Endeca made it relatively easy to build a single, searchable index across all the data sources, while preserving the integrity of access controls. Mr Harridge described it as "search technology and business intelligence merged together."
The ecommerce heritage of Endeca means it provides 'guided navigation' - just as an online store lets you drill down to say men's long-sleeved shirts made from pure cotton that are available in size 89, the Department of Justice system lets users select for example a particular name, date range and contact type.
The system is also serving as a foundation for other projects, such as exploring patterns of mobile phone use in the hope of finding ways to reduce the total cost, and determining whether certain infrequently used data sources can be retired completely or migrated to another platform to save money.
Page 4: Big data gets wet - and why being rich can mean a shorter queue at the ATM
The use of multiple independent systems led to problems with manual consolidation and inconsistent reporting, so Goulburn-Murray moved to an Oracle-based data warehouse and business intelligence system, explained information systems manager John Weber.
This system is also being used to store and process a large volume of data from irrigation systems. Goulburn-Murray is heading towards full automation and central control of irrigation, and the data can be used for day-to-day operations as well as predictive analytics, said customer support manager John Vise.
The use of big data tools will allow Goulburn-Murray to concentrate on business issues rather than the underlying technology, he said.
Around 1.5 million records are collected annually, and they are used, for example, to document exactly what was done in the event of a complaint about spraying from a landholder.
Finally, Oracle's Asia Pacific sales consulting leader Craig Han described an example of a regional customer using social data as part of its big data efforts.
A Singapore bank watched for tweets about queues at its ATMs, and then correlated those messages with ATM transaction records to determine exactly who was complaining. If high-value customers were being inconvenienced, the bank considered installing an additional ATM at that location.