You are currently browsing the category archive for the ‘Governance, Risk & Compliance (GRC)’ category.

It has been more than five years since James Dixon of Pentaho coined the term “data lake.” His original post suggests, “If you think of a data mart as a store of bottled water – cleansed and packaged and structured for easy consumption – the data lake is a large body of water in a more natural state.” The analogy is a simple one, but in my experience talking with many end users there is still mystery surrounding the concept. In this post I’d like to clarify what a data lake is, review the reasons an organization might consider using one and the challenges they present, and outline some developments in software tools that support data lakes.

Data lakes offer a way to deal with big data. A data lake combines massive storage capabilities for any type of data in any format as well as processing power to transform and analyze the data. Often data lakes are implemented using Hadoop technology. Raw, detailed data from various sources is loaded into a single consolidated repository to enable analyses that look across any data available to the user. To understand why data lakes have become popular it’s helpful to contrast this approach with the enterprise data warehouse (EDW). In some ways an EDW is similar to a data lake. Both act as a centralized repository for information from across an organization. However, the data loaded into an EDW is generally summarized, structured data. EDWs are typically based on relational database technologies, which are designed to deal with structured information. And while advances have been made in the scalability of relational databases, they are generally not as scalable as Hadoop. Because these technologies are not as scalable, it is not practical to store all the raw data that come in to the organization. Hence there is a need for summarization. In contrast, a data lake contains the most granular data generated across the organization. The data may be structured information, such as sales transaction data, or unstructured information, such as email exchanged in customer service interactions.

Hadoop is often used with data lakes becausevr_Big_Data_Analytics_21_external_data_sources_for_big_data_analytics it can store and manage large volumes of both structured and unstructured data for subsequent analytic processing. The advent of Hadoop made it feasible and more affordable to store much larger volumes of information, and organizations began collecting and storing the raw detail from various systems throughout the organization. Hadoop has also become a repository for unstructured information such as social media and semistructured data such as log files. In fact, our benchmark research shows that social media data is the second-most important source of external information used in big data analytics.

In addition to handling larger volumes and more varieties of information, data lakes enable faster access to information as it is generated. Since data is gathered in its raw form, no preprocessing is needed. Therefore, information can be added to the data lake as soon as it is generated and collected. This approach has caused some controversy with many industry analysts and even vendors to raise concerns about data lakes turning into data swamps. In general, the concerns about data lakes becoming data swamps center around the lack of governance of the data in a data lake, an appropriate topic here. These collections of data should be governed like any other set of information assets within an organization. The challenge was that most of the governance tools and technologies had been developed for relational databases and EDWs. In essence, the big data technologies used for data lakes had gotten ahead of themselves, without incorporating all the features needed to support enterprise deployments.

Another, perhaps more minor controversy centers around terminology. I raise this issue so that, regardless of the terminology a vendor chooses, you can recognize data lakes and be aware of the challenges. Cloudera uses the term Enterprise Data Hub to represent essentially the same concept as a data lake. Hortonworks embraces the data lake terminology as evidenced in this post. IBM acknowledges the value of data lakes as well as its challenges in this post, but Jim Kobielus, IBM’s Big Data Evangelist, questioned the terminology in a more recent post on LinkedIn, and the term “data lake” is not featured prominently on IBM’s website.

Despite the controversy and challenges, data lakes are continuing to grow in popularity. They provide important capabilities for data science. First, they contain the detailed data necessary to perform predictive analytics. Second, they allow efficient access to unstructured data such as social media or other text from customer interactions. For business this information can create a more complete profile of customers and their behavior. Data lakes also make data available sooner than it might be available in a conventional EDW architecture. OurVentanaResearch_DAC_BenchmarkResearch data and analytics in the cloud benchmark research shows that one in five (21%) organizations are integrating their data in real time. The research also shows that those who integrate their data more often are more satisfied and more confident in their results. Granted, a data lake contains raw information, and it may require more analysis or manipulation since the data is not yet cleansed, but time is money and faster access can often lead to new revenue opportunities. Half the participants in our predictive analytics benchmark research said they have created new revenue opportunities with their analytics.

Cognizant of the lack of governance and management tools some organizations hesitated to adopt data lakes, while others went ahead. Vendors in this space have advanced their capabilities in the meantime. Some, such as Informatica, are bringing data governance capabilities from the EDW world to data lakes. I wrote about the most recent release of Informatica’s big data capabilities, which it calls Intelligent Data Lake. Other vendors are bringing their EDW capabilities to data lakes as well. Information Builders and Teradata both made data lake announcements this spring. In addition, a new category of vendors is emerging focused specifically on data lakes. Podium Data says it provides an “enterprise data lake management platform,” Zaloni calls itself “the data lake company,” and Waterline Data draws its name “from the metaphor of a data lake where the data is hidden below the waterline.”

Is it safe to jump in? Well, just like you shouldn’t jump into a lake without knowing how to swim, you shouldn’t jump into a data lake without plans for managing and governing the information in it. Data lakes can provide unique opportunities to take advantage of big data and create new revenue opportunities. With the right tools and training, it might be worth testing the water.


David Menninger

SVP & Research Director

Talend recently announced version 5 of its information management platform, which emphasizes unifying its various components. Through a combination of development activities, acquisitions and partnerships, Talend has been steadily building its portfolio of information management capabilities. In addition to its core data integration capabilities, it has added data quality, master data management, application integration and with this release business process management (BPM).

In its progress, Talend has exploited the open source model well. Not only has it developed a large community of users for its free version, but it has successfully used awareness and interest in the open source product to build a community of 2,500 paying customers for the commercial versions of its products. The open source concept also has influenced Talend’s acquisition and partnership strategy. Sopera, acquired by Talend last year, was an open source vendor, and the BPM capabilities in version 5 are based on a partnership created 18 months ago with open source vendor BonitaSoft.

Like other software vendors in the information management market, Talend developed its portfolio as independent products, which resulted in an array of somewhat disjointed capabilities. This problem was made worse with the acquisition of a completely independent set of products from Sopera. Customers should welcome Talend’s efforts in version 5 to create a unified information management platform. Our ongoing research in information management investigates the importance of capabilities such as data integration, data quality, master data management and application integration.  Other vendors are combining these capabilities as well, but Talend has chosen to go further by adding BPM capabilities. Too many software vendors and enterprise IT departments expect customer organizations to adapt their business processes to how technology products work rather than the other way around. This misguided approach is the central theme of my post “What Is Wrong with Business Intelligence?”, and these same issues affect information management. I expect more vendors to embrace BPM as a way to combine business intelligence and information management into business processes.

With this release Talend also extends its big-data and cloud capabilities, which can be complementary. Our Business Data in the Cloud benchmarkresearch shows the importance of cloud-based data sources. Within 12 months 40 percent or more of eight different lines of business will be using cloud-based applications. Talend’s new cloud capabilities include connectors for Amazon RDS instances of Oracle and MySQL. Talend 5 also includes a REST interface for applications that require or support it. Our Hadoop and Information Management benchmark research shows that over half the organizations tackling big-data issues are using or evaluating Hadoop to meet those challenges, and in this release Talend extends its big-data capabilities to support HBase, a database component for Hadoop, and continuous streaming of information into or out of the Hadoop Distributed File System (HDFS).

On a minor note, Talend has renamed its open source products around the “Open Studio” theme. So now each of the four products are named “Talend Open Studio for…”. The open source version of BPM is available from BonitaSoft, so it doesn’t follow this same convention.

Now that Talend has unified its information management products, one of its remaining challenges will be to compete with integrated BI and information management suites. End users need information such as metadata and lineage to flow through both sets of products freely. But can an independent vendor provide the same level of integration as a suite vendor? Informatica, one of Talend’s competitors in the information management market, has shown that it is possible to succeed as an independent vendor. It will be important for Talend to maintain close partnerships with as many of the BI vendors as possible so it can fully support the features end users require.

Based on customer growth, it appears that Talend is succeeding with its current strategy. The new release should make the products both easier to use and more broadly applicable. If you are looking for an information management platform independent of your database, applications and BI environments, I recommend considering Talend.


David Menninger – VP & Research Director

Follow on

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 22 other followers

RSS David Menninger’s Analyst Perspective’s at Ventana Research

  • An error has occurred; the feed is probably down. Try again later.

David Menninger – Twitter

Top Rated

Blog Stats

  • 46,527 hits
%d bloggers like this: