Splunk Delivers Insights from Machines

I recently attended .conf2016, Splunk’s seventh annual user conference. Splunk created the market for analyzing machine data (shorthand for machine-generated data), which consists of log files and event data from various types of systems and devices. Our big data analytics benchmark research shows that these are two of the most common sources of big data that organizations analyze. This market has proven to be fertile ground for Splunk, growing steadily with revenues more than doubling over the previous two fiscal years. Machine data is also the backbone for the Internet of Things (IoT) and operational intelligence, which form the basis of forthcoming benchmark research from Ventana Research.

At the event, Splunk announced general availability of Splunk Cloud and Splunk Enterprise 6.5. The company also announced new versions of Splunk IT Service Intelligence, Splunk Enterprise Security and Splunk User Behavior Analytics. These new versions incorporate machine learning capabilities to help organizations analyze the massive volumes of machine data they collect with more advanced analytics and in a more automated manner. Machine learning has become a hot topic lately; it was also a popular subject at Strata+Hadoop World, as I wrote recently.

The machine learning capabilities, which arose in part from Splunk’s July 2015 acquisition of Caspida, have been added to Splunk Cloud and Splunk Enterprise 6.5. Machine learning is a method used to develop predictive analytics without explicitly programming the models. In effect the algorithms are designed to sift through the data, learn from it and make predictions. With Version 6.5 Splunk also has simplified its data preparation capabilities and enhanced its user interface to appeal to more types of users. The company also offers tighter integration with Hadoop in this version.  Storing historical data in Hadoop can help lower costs, and the Hadoop data can be combined with data in Splunk Enterprise using the Splunk query capability for a single unified interface.

Splunk IT Service Intelligence (ITSI), an application built on the Splunk platform, provides a view of how critical IT services are operating as well as an environment in which to investigate and triage incidents when they occur. The latest release of ITSI, 2.4, includes machine learning capabilities to perform anomaly detection, identifying unusual system activity to help prevent outages and service degradations. The system can learn what the pattern of normal operations looks like and then establish thresholds for alerts that adapt to cyclical changes in usage. Adaptive alerts help reduce “alert fatigue” when so many alerts are issued that they overwhelm the recipients.

Splunk Enterprise Security (ES), a security information and event management (SIEM) application, provides real-time monitoring of security threats and an environment to support incident response teams. Splunk ES 4.5, the latest release, provides a similar adaptive alerting feature based on machine learning as described above. ES 4.5 now includes the Glass Tables feature that has been available in ITSI, which allows users to create custom visualizations and KPIs. Splunk User Behavior Analytics (UBA) complements ES by analyzing longer periods of history to create a profile of normal user behavior and comparing it with peers to provide more advanced detection of security threats. UBA 3.0 incorporates more than 40 machine learning models, which cover a combination of streaming and batch analytic scenarios. Splunk in 2015 received the Technology Innovation Award for CIO for its innovation in advancing cybersecurity through these products.

Splunk has followed a unique path. While a pioneer in the big data market, it built its products on a proprietary big data architecture rather than open source technologies as others did. In recent releases, however, it has broadened its support for Hadoop. Splunk focused on one subset of big data – machine data – and based much of its user interface around search. Rather than expand into the horizontal business intelligence market the company has chosen to tackle the IT service market and the SIEM market. This focus appears to have been successful so far. It’s hard to argue with its success. If you are looking for a way to manage and analyze the machine data in your organization, including IT service applications or enterprise security, I recommend you consider the offerings from Splunk.

Regards,

David Menninger

SVP & Research Director

Follow Me on Twitter @dmenningerVR and Connect with me on LinkedIn.

Information Management Technology Revolution and Research Agenda for 2012

In our definition, information management encompasses the acquisition, organization, dissemination and use of information by organizations to create and enhance business value. Effective information management ensures optimal access, relevance, timeliness, quality and security of this data with the aim to improve organizational performance. This goal is not easily met, especially as organizations acquire ever more data at an ever faster pace. In our business analytics benchmark research of more than 2,600 organizations, almost half (45%) have to integrate six or more types of data in their analyses. More than two-thirds reported that they spend more time preparing data than analyzing it. To assist in dealing with these sorts of issues and others, we’ve laid out an ambitious information management research agenda for 2012.

In recent years the complexity of information management has risen dramatically. The volume of information being processed has increased exponentially and so have the challenges of ensuring consistency and quality and managing governance and the information life cycle. New data types and sources such as comments on social media have emerged and must be integrated into an organization’s information assets. Moreover, in many cases the boundaries between organizations and the outside world with which they interact have become far less distinct, leading to the need for a more expansive understanding of information management. Our Business Data in the Cloud research shows that data is seldom stored in only one repository; the majority of organizations (86%) need to bring together cloud-based data and on-premises data.

We will provide new insights on the dynamics of the information management market as we complete research on Information Management Trends. This research will illuminate the priorities organizations place on data quality, master data management and data governance. It will also explore ways in which organizations are incorporating virtualization and replication for broader and faster data access. The growing volumes and sources of data will require data integration that can help facilitate better linkages across IT and into business. We will assess the vendors and products in a Value Index for Data Integration that will determine what suppliers can be best fit for your enterprise.

Our research will also help organizations facilitate adoption of and use of big-data technologies. Our recently published Big Data research highlights the role of various technology alternatives for managing data on a large scale. More than 80 percent of organizations utilize more than one technology to tackle their big-data challenges, but organizations lack maturity when incorporating these data sources.  Specifically, our research shows that business have not adapted many of their standard processes to deal with big data. We’ll follow up this research by looking at specific vendor capabilities and how they can help extend information management processes to support big data.

Data is increasing not only in volume but in velocity as well – the speed with which data is generated and communicated. Technological developments such as smart meters, RFID, sensors and embedded computing devices for environmental monitoring, surveillance and other purposes are creating demand for tools that can derive insights from huge, continuous streams of event data coming into systems in real time. Traditional database systems are geared to manage discrete sets of data for standard BI queries, but event streams from sources such as sensing devices typically are continuous and their analysis requires different kinds of tools that enable users to understand causality, patterns, time relationships and other complex factors. These requirements have led to innovations in complex event processing, event stream processing, event modeling, visualization and analytics. We’ll be exploring how organizations can capitalize on real-time data collection and analysis in our benchmark research on operational intelligence and complex event processing. We will also assess vendors and products in a Value Index to determine the value of vendor offerings in Operational Intelligence to harvest the events from these streams of data.

Information management continues to be a strategic business imperative. It can help organizations improve their understanding and use of enterprise information and to establish governance of it. To accomplish these aims they must manage the flow of information throughout the full life cycle of data and provide proper data stewardship to support the business while minimizing risk. We need to better use the information through a simpler means of being able to assemble and deploy it to those in business who might even want to receive it through mobile technologies. This is what we call information applications that can help in timely access to information and should be coupled with an information management discipline. Our research will deliver education and best practices that can help you understand how to reduce the costs, time and risk of delivering these capabilities to your organization.

It will be a big year for information management in the forms of technology but also the methods and processes for which to manage and utilize the full value of it within organizations. I look forward to connecting with all of you on LinkedIn or following me on Twitter.

Regards,

David Menninger – VP & Research Director