Conquering the data mountain

Bearpark: The value we add to the application makes the utilisation of the SSD phenomenally cost effective

Businesses today need to store and manage massive amounts of data as the explosive growth in digital content shows no sign of slowing.

The rapid obsolescence of content combined with the need to retain data long term is forcing organisations to look for effective ways of accessing this information, while trying to control the overall cost of storage and manage data efficiently. Unstructured data is also adding to the management challenges and storage overheads as Andrew Gorton, technical co-ordinator at Greater Manchester Police, has discovered.

“The demand for applications that use more rich content is accelerating storage growth,” he says. These include images, audio and video records, automated number plate recognition data, CCTV and digital voice recordings.

“At the same time, we are seeing growth in the structured data produced by central business systems and unstructured data produced by office applications. A key consideration for us is to address the long-term capacity risks of data residing on inappropriate storage. We need to plan ahead to make sure that data does not grow out of control and exceed physical capacity,” he says.

For some organisations, the economic climate is also putting pressure on storage systems. Dave Allerton, IT director at architect RHWL, uses large, mirrored central network attached storage (NAS) devices to support the work of staff located in London, Germany and Quatar.

“In leaner times, when we are hungry and looking for new work, we produce lots of proposal documentation, which far outstrips the regular construction records,” says Allerton. “The guys are creating 200MB architectural models on a daily basis and sometimes working on two or three iterations, so we quickly get into gigabytes of data being created every day.”

For RHWL, a tiered storage system from F5 has reduced the risk of running out of space while satisfying the legal requirements of retaining data for up to 12 years.

“A lot of our data needs to be kept live because of the needs of ongoing projects,” says Allerton.

“However, 60 per cent of our data hadn’t been touched for more than six months, so we were able to move it onto cheaper tier two storage. Overnight we regained 60 per cent of our tier one storage without users noticing, except that it worked faster because all of a sudden they had all this additional storage to play with. Currently, we have 7TB of data and that is growing by about 2TB a year.”

For many businesses the benefits of adopting a virtualised storage environment are becoming more compelling. With the move away from a silo-based infrastructure model of storage towards a centralised strategy of tiered storage and information lifecycle management (ILM), virtualisation adds flexibility, management control and cost effectiveness.

According to John Grove, senior technical architect at utility group Scottish and Southern Energy (SSE), the continuing trend of server virtualisation is having a significant impact on decision-making around storage.

“SSE has made a significant investment in virtualisation technology and the net effect of this is that all the storage that used to be internal to the server has moved into the SAN [storage area network] environment,” he says. “This shift has meant that more blade servers are connected to the SAN, increasing the number of physical switches and naturally the volumes of data moving within the SAN fabric.”

Managing a storage estate fast approaching a petabyte in size is also particularly challenging.

“Storage management tools are still immature; vendors do produce tools that have good capabilities when monitoring their own kit, but monitoring heterogeneous environments is a real challenge as there is not one tool that provides enough depth across multiple vendor products,” says Grove.

Landmark Solutions uses enormous amounts of data to supply a wide range of services around geospatial data, geographic applications, web mapping and geocoding. It has just upgraded its entire storage infrastructure to be more scalable, virtualised and to remove vendor lock-in restrictions.

“We wanted a platform to build multiple solutions without creating islands of storage and a need for constant upgrades ­ a system that would scale out to a relatively large size but in small increments,” says the firm’s Unix systems administrator, Vic Cornell.

“OCF designed for us a vendor-independent, scalable, flexible virtualised storage environment that has reduced administrative overheads and yet allows us to grow our business without requiring additional staff and resources.”

But for Landmark it is not just a case of throwing more disks at the solution; speed of operations is also important. “While the cost of disk space may be falling, the cost of IOPS [input/output operations per second] is not,” says Cornell. “Having sufficient space isn’t the problem –­ it’s more about the speed at which you can access your data.”

The cost of storage is not just the price of the drives ­ – what really digs into budgets is the management of the data, which is where those pricey systems from specialist storage companies come into the equation. You may think you are paying over the odds for disk capacity, but what you are really buying is the software that provides the intelligence to manage the storage space effectively.