Splunk has traditionally been a tool to ingest, index and enable searching across vast bodies of data from multiple disparate sources.
Rick Fitz, vice-president of IT Markets, explains the driving forces behind the introduction of machine learning, announced this week at the 7th annual Splunk conference, .conf 2016.
Data doesn't live in a vacuum. Fitz states. "The goal of the data is how do I make better business decisions?"
Machine learning, Fitz explains, adds context and meaning to all this data that has been gathered through the use of algorithms.
"The machine learning that's best should be just natural and all around you. Users don't necessarily need to know the algorithms in use, they can just observe and use the results. There's machine learning, anomaly detection, modelling, and so on," he states.
"Most of the data scientists of the world waste 85% of their time just trying to find where the data is they want, get it in some format, and write 4GL apps to try and work with that data. With Splunk Enterprise 6.5," says Doug Merritt, Splunk chief executive, "the data is just there and the machine learning toolkit allows users to craft apps in whatever language they are comfortable with."
"We're on a journey, and that journey is one where we're trying to leverage algorithms, Artificial Intelligence, to just do what computers do best, figure out patterns of information, and surface those so humans can solve problems," Fitz states.
"In the future, we won't talk about machine learning or Artificial Intelligence and just talk about how systems solve problems. It will become something we use every day and rely upon."
The machine learning toolkit enables a data scientist to leverage any relevant available open source algorithm to do something with Splunk data sets. "That really opens up the ecosystem for problems we can have data scientists solve," Fitz says.
Yet, "Data scientists are in scarce supply," he adds. "So what we have done in the new IT Service Intelligence (ITSI) premium product is to apply some of these algorithms, allowing an operator to simply point-and-click and configure some of these algorithms for use, such as dynamic thresholds."
Setting static thresholds is something IT operators have been doing for years – the typical CPU usage, disk usage, and so forth. Yet, Fitz explains, "the world of compute storage today is pools of resources and they collectively work together to solve problems, so you can add and remove compute based on needs. What you find is your performance can actually change and your thresholds need to change. They become dynamic, your CPUs will climb during the day, and you need to dynamically change your thresholds – what was green at one point is not green at another."
"We're using a machine learning algorithm to learn from prior behaviour and if the behaviour deviates moving forward we would notify after that fact. Our machine learning algorithms built into ITSI remove false positive alarms and will be much more useful to an operator. It is a good example of us packaging algorithms into our solutions so people who aren't data scientists can leverage them."
"We will continue to add algorithms into our premium solutions such as ITSI. The analytic theme will continue for many years for us."