Mind the store

If your top brass thinks that its data warehouse - stacked to the rafters with 'valuable' information - is its most important strategic asset, then you'd better think 'storage'.

'Storage is key to a warehouse, it's where information lives,' says Astley Gayle Jr, EMC's product marketing manager for the UK and Ireland. 'Storage impacts performance and data availability.'

Enterprise information, including data warehousing, is set to become the most important storage application over the next five years, ahead of storage applications for client-server systems and multimedia. 'The explosion in demand for information storage to meet the needs of data warehousing and data mining is largely responsible for the problem of storage inflation,' says Andy Walsky, Quantum Corporation's European Analyst.

Most organisations face dramatically increased storage demands, particularly large companies, where the volumes of data can be overwhelming. 'Decision-support applications are soaking up storage capacity at huge rates,' says Tom Lahive, IDC's Senior Storage Analyst. 'Intranet and Internet applications made a relatively small dent in the storage market during 1996, but this will change during the second half of 1997 and in 1998.'

To meet this demand and to avoid problems early in the life of a data warehousing application, businesses can no longer treat storage as an afterthought. Instead, IT managers should ensure computer storage is a strategic part of IT systems from the outset, and make it a central resource, independent of a particular platform.

'A sensible archive strategy should also be part of a warehouse. As data gets older it's less important to hold it online, and if it's online it's often not necessary to hold it in its full form,' says Richard Neale, SAS Institute's Data Warehouse Product Manager.

'It may mean summarising weekly or monthly transactions or archiving old data onto tape. It's important, though, to ensure all data is still accessible with minimum disruption for the user.'

The demand for data warehousing has created an unholy mixture of different technologies. This is also forcing companies to rethink how they should manage the whole of their storage options to optimise performance and guarantee security.

'Although the technology is becoming relatively inexpensive, it's not clear it's being managed and protected as well as it should. Once you've bought all that storage and you realise you've got a lot of important data on it, there are lots of other issues you've got to look at,' says Mike Casey, Gartner's chief storage analyst.

These choices are usually application-dependent - What kind of application are you running? Do you need high bandwidth, high transaction rates? Do you really need fault tolerance or just need to be able to recover data if you lose it? Usually, the creation and management of a storage architecture is the answer. 'We tell clients to take a considered, architectural approach to storage, and where it should be located and managed,' adds Casey.

Too many companies have bought a lot of systems that are then scattered all over the enterprise with different kinds of storage inside them. Companies are beginning to realise these devices manage important customer data, yet they are not being managed and backed up as thoroughly as in the traditional days of 'everything-on-the-mainframe'.

They now know they don't have the tools, disciplines, policies and procedures in place out in the departments to safeguard this data and manage it well.

It's inefficient to have many devices scattered around the building, being only partly-used. So there's a certain amount of consolidation going on, as firms put storage in one big box and manage it from the data centre.

With users demanding ever-faster access to data, a storage management software solution is necessary to anticipate bottlenecks. Hierarchical Storage Management (HSM) is used not only in mainframe environments, but also in client-server networks to manage all storage media on the network.

The data warehousing software also needs to be able to read and write to tape and disk as appropriate, and to access both seamlessly. Unfortunately, most data warehouses don't currently support archiving, whether online or not, so archived information must be restored for user queries to become effective. Seamless archiving should therefore be an integral element of the data warehouse architecture, not an afterthought.

Looking ahead to the next five years, data warehousing storage will be affected by the debate over the network computer (NC) and the development of Internet and intranet computing. In a distributed, client-server architecture, some data is often stored locally on the client. In an NC architecture, the server will have to take on board the application and 100% of the application logic.

Any Web-enabled data warehouse will therefore need to be bigger in terms of both data and hardware, although its cost and size should be offset by the reduced cost of ownership of the NC environment. And as not all business intelligence applications are suitable for Web deployment, there will be a significant number of 'power users' still using PCs.

As datamarts (specialised data warehouses) expand, there is a danger of creating even more isolated pockets of information throughout an organisation - something which destroys what the data warehouse is setting out to prevent.

'The same problems can occur with PC storage in the data warehouse,' says SAS Institute's Neal.

'At the moment, the danger of using PCs for strategic storage in a client-server data warehouse architecture is that this is often overlooked. The data warehousing process must ensure all relevant information is accessible, whether or not the data resides on the central server.'

Success will depend on good answers to all these questions being built into the planning of your data warehouse and accompanying storage strategy.

'It's important to have planned growth - don't be tempted to buy lots of hardware up front at a discount before the value of the data warehouse to the organisation is proven,' continues Neal. 'Many pioneers are stuck with increasing amounts of hardware to store data, with no coherent strategy to ensure it provides value to an organisation.'

According to Gayle, storage often represents more than 65% of the platform acquisition costs, yet it is often not given the same level of focus as, say, the server platform when choosing which 'best-of-breed' products most effectively address a customer requirement. 'That said, a growing number of customers do recognise the benefits of making a separate, external, storage choice, just as they would for database software,' says Gayle.

Primary storage is almost always disk these days. The main benefits of using disk for larger database applications are speed and cost. 'Although the price of disk is falling, installed capacities are growing at a rate exceeding the fall in raw megabyte costs,' says Mel Taylor, StorageTek's marketing manager. He also claims disk performance has increased about fourfold over the last four years whereas capacity has increased by a factor of 100. 'That's why storage management is becoming the key issue to be addressed by data warehouse users.'

Storage management's costs include backup and recovery, archiving, capacity planning, performance management, data distribution, presentation and security: 'Productivity is increased by reducing the size of disk storage, reducing the time taken for data back-ups and any recovery that may be required,' adds Taylor.

According to Peter Martin, marketing manager of Sterling Software's systems management group, US companies are mixing and matching different storage devices so they can take advantage of technology advances and competitive pricing. 'The number of organisations buying storage from only one vendor is shrinking rapidly,' he says.

Martin recommends IT managers focus on data. 'That's where the business value is,' he says. So the IT manager needs to build a data/storage management environment which is able to:

- Accommodate any new devices as they come along.

- Provide management and performance windows into all the major Raid offerings.

- Allow you to group, view and manage the data by application.

- Use these logical views to interface to the various storage management solutions.

He also recommends that the IT manager keep vendors on their toes by demanding keener pricing in what is a competitive environment. However, there are drawbacks:

- More vendors in the market translates into greater complexity and the need to take care choosing where data should be stored.

- It takes some effort to keep up to date with technology and make the right purchasing decisions.

- Storage management is becoming a burden. Devices, using a number of interfaces, need the vendor's own software to work effectively - for example, to monitor performance.