Artificial intelligence

Knowledge is power – but only if it can be extracted quickly and efficiently from an ever-growing mountain of data. It is projected that in just four years the world’s information base double in size every 11 hours.

The data explosion is affecting the whole industry – from manufacturing to financial services. The problem at the moment is not in storing the data – millions of pounds have been spent over the last 10 years investing in data warehouses to contain the ever-growing data pools.

However, it is becoming harder to retrieve information promptly, and networks and application performance is suffering from excess traffic as users search, and then search again, for the material, statistics and knowledge they need. Our terabytes are turning toxic.

Driven by the ascendance of business intelligence (BI) into a priority investment area, the data warehouse market is booming. According to Datamonitor, 32 per cent of companies plan to invest in BI for financial analytics with further BI spending planned for other business areas over the next two years.

The role of the database is changing, in particular how we use data. From the business perspective it has become about building adaptability and change into IT. According to analyst Gartner business logic should not necessarily be stored in a database, but should be closer to, and relevant to, the business where it is to be used. A central repository of information, such as a data warehouse, will not serve real-time applications well.

Charles Nicholls, chief executive at BI consultant SeeWhy, says that part of the problem is the focus and application of data warehouses. ‘The challenge is that data warehouses are data centric, storing data ready for analysis and reporting are not aligned to the process at hand,’ he says.

‘Moreover, since the data warehouse often does not hold process-state data, let alone real-time process state information, it does not usually have the information you need to make a decision.’

In many organisations the data warehouse is perceived as the sole central repository for historical data. However, the information needed for day-to-day operational decisions needs to be held in a way that is centred on the business process, perhaps even as part of the applications.

The goal for companies is to find a new approach to data warehousing infrastructure that is both specific and flexible – suited to handling vast amounts of data and compatible with existing BI applications and infrastructure. It also needs to be straightforward to deploy, easy to manage and affordable.

The traditional answer to the problem has been to continue to add more hardware, increasing the patchwork of hardware, software and storage that is growing ever more complex. Terabyte-scale databases that continue to grow steadily put tremendous strain on these systems, and often fail to meet the basic BI goals of delivering vital business information to make decisions in a timely manner.

In addition, the vast array of extra equipment is difficult, time-consuming and costly to manage and maintain.

The data warehouse appliance is designed specifically for the streaming workload of BI and is built using commodity components. Sarah Burnett, analyst at Butler Group, believes that the advent of the data warehouse appliance will address the issues of speed and accessibility.

‘Appliances are dedicated solutions that combine software, server and storage into a single pre-configured package, such as offered by Netezza and DATAllegro,’ she says.

‘The different components of the appliance are optimised to make the most of their joint capabilities, making these dedicated solutions very fast.’

A data warehouse appliance is architected to remove all the bottlenecks to data flow so that the only remaining limit is the disk speed. Standard interfaces ensure that the appliance is fully compatible with BI applications, tools and information.

Simple to use, it also has a low cost of ownership through its use of commercial, off-the-shelf components. Offering superior performance at a significantly lower price, data warehouse appliances deliver convenience, predictability, speed and price. Scalable, reliable and easy to administer, the integrated nature of an appliance means that there is very little to do.

The increased performance is one of the most significant benefits of the data warehouse appliance – reports that took days can be compiled in minutes. Data such as call details records, individual web clicks, itemised point-of-sale transactions and hyper-detailed customer activity can all be readily analysed in near real-time.

Using high-speed analysis has allowed holiday resort chain Center Parcs to review and amend its web site to improve usability and also increase bookings.

Richard Verhoeff, director of IT services at Center Parcs, says that the Netezza appliance has allowed the company to perform cluster analysis on all its web click stream information, with interesting and profitable results.

‘We knew that our home page is used a lot, but through cluster analysis we discovered that other pages were being used almost as much, which led us to place offers on those pages too,’ says Verhoeff.

‘In a shop you put attractive items at eye level but there you have the added advantage that you can watch the customers. On an internet site you have to watch the data to find out where eye level is, and for that you need the power of an appliance because of the sheer volume of records.

See next page for what the experts say about data warehouses