Imperial War Museum battles data storage

Details of every item in the Imperial War Museum will be included in the digitised system

Not every organisation can rely on a government grant to help it expand its data storage capacity, or make money by selling the same content online.

However, the recent revamp of IT infrastructure at the Imperial War Museum (IWM) provides a blueprint for an efficient upgrade of storage, network and disaster recovery facilities at multiple sites which makes the best of legacy equipment to keep costs down.

The project, aimed at digitising and cataloguing the whole collection, saw the implementation of a mass storage system able to support a bespoke digital asset management system (DAMS), developed using a combination of software from museums and library application specialist Adlib, and image and video archiving management specialist Cambridge Imaging.

However, the project had unexpected results, making efficient data storage even more important. Ian Crawford, IWM head of ICT, said: “We have actually generated more data during the digitisation process. With DAMS we create a new digital object, such as an analogue photo of a spitfire, for example, create a high-res TIF as a master image, then smaller JPGs that can be accessed via the web front end.”

The IWM owns five sites spread over London, Duxford in Cambridgeshire and Manchester.

When Crawford took the job three years ago, the museum in Lambeth Road was running a single OnStor Bobcat network attached storage (NAS) gateway with 5TB of direct attached storage behind it.

This has been retained, with additional NAS gateways from LSI Storage Systems installed at the IWM’s London datacentre and Duxford site, backed by an LSI Engenio 3994 RAID array providing up to 80TB of capacity to host the DAMS system.

The IWM also continues to run two Nexsan SATABeast RAID arrays at Duxford and some DotHill NAS gateways, all of which are connected into the fibre channel (FC) SAN. Crawford also plans to bring in a Spectralogic T950 tape library with a capacity of up to one petabyte at Duxford to house its archived files.

“One of the key drivers for using the LSI storage arrays and NAS gateways is that, as long as there is an FC interface, we can put any type of technology behind it, and SATA hard disks are a good compromise between cost, capacity and speed,” he said.

The storage overhaul, implemented with the help of reseller Eurotech, was partly funded by a public sector research exploration grant from the Department of Education, which was matched by the IWM, bringing total investment to about £2.5m.

The bill covered a simultaneous network and server revamp including a wide area network (WAN) connection upgrade to the joint academic network, new 10GbE local area networks at the London and Duxford sites, and building a new computer room in London to house the additional servers.

“We are on Gigabit [Ethernet] tails at Duxford and All Saints, though we use 100Mbit/s normally and burst to 1Gbit/s in increments of 100Mbit/s if we start serious file transfers between sites,” said Crawford.

The extra bandwidth and storage capacity have strengthened the IWM’s disaster recovery strategy by providing continuous data backups to locations across the WAN.

“There is a complete mirror between the two sites, so data is written to primary storage in London and replicated to Duxford,” said Crawford. “Once the tape library is fully functional, the idea is to back all that up to a tape archive at another small site a few miles down the road from Duxford.”

“The aim is to finish digitising the entire collection, then use the web site as a film sales site, where buyers can look, select and order them electronically,” he said. “We have done all the hard work getting in the infrastructure and the DAMS system; now we can concentrate on e-commerce activities and let customers access it.”