Opinion: Battle for data warehousing supremacy

The next couple of years will prove a major inflection point in the data warehouse market primarily due to the growing need for diverse types of information used in business decisions. An indication of this major shift are the five major acquisitions over the past 12 months: Aster Data, Greenplum, Netezza, Sybase and Vertica by Teradata, EMC, IBM, SAP and HP respectively. The database management system (DBMS) world is changing and will drastically affect the way we think about, create and use the data warehouse.

Hardware systems are faster, less expensive and have increased in size from one CPU per server with small memory sizes to over 64 CPUs and greater than 1TB of memory in a single server. Storage costs have plummeted. Add to this advances in software for storing and retrieving data, new methods of parallel processing of data (e.g. Apache Hadoop MapReduce), and the emergence of the in-memory DBMS, all of which make possible the storage, retrieval and analysis of all kinds of data, from normal, structured data to complex forms of unstructured data, such as text, email, audio, images and biometrics.

Finally, we need to process data more quickly, allowing us to use continuously-generated data, such as clickstream data, stock-market trade data, retail market basket data, smart metering data for utilities and phone-call records. This pronounced change in the data we use is the “Big Data” movement, denoting the large size and complex nature of the data.

Technology change alone does not drive adoption. New tools and applications are emerging that allow organisations to demonstrate business value, such as predictive analytics, performance management, pattern-based discovery and other real-time applications. Data warehouse infrastructure will need to change to support adoption of these.

By 2013, data warehouse DBMS vendors will combine their offerings to create information management platforms. The DBMS will become an execution platform, supporting and performing data management and integration tasks (such as transformation and loading) as well as processing queries and performing analysis.

The DBMS will graduate to the role of optimising the use of in-memory operations, in-memory database management and mixed storage tiers (e.g. disk, solid-state storage, Flash and memory). The repository function of persistence will become a secondary functionality - even in the case of managing and supporting metadata capabilities.

Many traditional data warehouse DBMS vendors already offer both the DBMS and execution platforms - but the independents may surprise the mega vendors, leading to acquisitions. By all indications, these acquisitions as well as research and development will be the deciding factors for the big winners and losers as the battle for supremacy in this market begins.

Donald Feinberg is a vice president of analyst firm Gartner