Disaster recovery: senior management not interested

managing data recovery is more important than ever with growth in data storage requirements

Computing: How do you justify the concept of business continuity management to senior executive and other departments?

Roger Bearpark, assistant head of ICT, London Borough of Hillingdon:

We don’t anymore. There are plenty of other things I can do and would not waste my time trying to sell it. Hillingdon tried very hard to get the interest of senior management in the past, but unless you have something practical such as scary cost figures to grab their attention, or talk about the potential loss of reputation, it is impossible.

Neil Edmonds, systems specialist (operations), the Salvation Army:

9/11 and 7/7 made no impact at all on our senior managers, who tended to think "it will never happen to us". One question was "if we have a DR site how long does it take to get a server in there?". So we had to take a group down to the data centre and show them six racks and ask: "which part of the infrastructure do you want to retrieve?" That got people thinking, but when we have an issue that affects the business, the first thing people do is go home. It is only on returning to work that they ask how are we going to get our systems back.

David Weston, IT infrastructure manager, Oxford University Press:

Business continuity is a major part of our role, but not [necessarily including] computers. We have a lot of academics in senior management who think IT should just work while they focus on writing and selling books. IT is becoming a business function within which the IT department is just a participant. We invite business users to come along to tests designed to make sure they can recover their systems [in the event of failure], which is a major step forward, but it is a lot of hard work to pursuade people to attend.

Computing: Do you keep DR in-house or outsource it?

Weston: We use [managed availability services provider] Sungard, which is more cost effective than doing it in-house. Our warehouse in Corby is far enough away to be a good DR site for us, but the cost of putting the hardware in, keeping it patched and maintaining it was far higher than using a third-party provider.
Also, Sungard provide annual DR testing as part of the service, and if there is a major incident in Oxford, we can get the [backup] tapes to Sungard and they will recover the systems for us.

Bearpark: We have both a live and secondary site that help us deal with momentarily increased requirements such as council-tax demands. The local authorities are not in competition with each other, so there is an opportunity to look at mutual aid and support around DR. If you have arrangements around third-party support rather than in-house provision it is difficult to set that up. The strongest assets at Hillingdon are the in-house IT staff who really understand the business and can implement their own resiliency and DR plans.

Edmonds: One of our third-party providers gave us consultancy, did an impact analysis with our business units and found that the only thing we could not exist without is the email system - but there is still no [in-house] business continuity plan for that.

Computing: How fast are your data volumes growing, and what challenges does that present?

Weston: We are dealing with data storage growth requirements of around 100 per cent per annum, and have about four times the volume we had two years ago. The Compellent kit we recently introduced maintains that data growth and has reduced the overall cost of storage.

Our data centre is in a listed Oxford building so there are some limits on what we can do there, especially when the power utilisation is running at 95 per cent. Switching off redundant kit is hard work, but we are in the early stages of implementing server virtualisation.

Edmonds: We use HP StorageWorks EVA for shared storage, which we can expand easily using extra disks. There is continuous access between our data and our DR site, the data is asynchronously replicated to that site so we can do failover at the press of a button. We are also trying to tier our storage, using slower systems as dumping ground areas for un-important data.

Bearkpark: We have a project looking at tiering storage, because data is currently held on one level. Our databases are growing by 2GB a week so we need to keep them at high level recovery, while simultaneously reducing the costs and the amount of equipment we have in the computer room.

Computing: Have you had to make any compromises due to budget limitations?

Bearkpark: If I had the budget I would buy booze for senior managers and make them loosen up a bit and tell me what it is they actually want from storage.

Edmonds: It all comes down to budgets – from an IT perspective we are looking at resources that we can use in-house to improve business continuity, like backup. But we have to make it clear at board level what we do and do not have, and how much it would cost to provide what they want. We try and sell storage and virtualisation projects to the business, and are always looking for cheaper ways of doing things, like making do with local storage. The trouble is you can sometimes end up storing up future problems by taking the cheapest option at an early stage.

Weston: We always look at the most cost effective way of doing things, but we want an environment where we have what we need for the business and we don't need instant failover. We are not a bank, there are budget restraints of course.

Computing: When you think about disaster recovery, is if failover, recovery from backup, or both?

Weston: In this day and age, it is just failover though some organisations do go through recovery without failover – throwing planning and testing to one side and dealing with the situation there and then. There is not too much in previous DR plans that have helped, but the technology now available offers a much better facilities for maintaining core [storage] infrastructure so it is less about quickly getting the tapes back and restoring them, and more about failing over to those systems.

Edmonds: It is a question of where disaster happens and where you can build a recovery from. For example, is there finance and the timescale for doing it? You have to look at what is going to fail and what will happen if those systems are not backed up within a certain amount of time.

Bearpark: Different situations require different responses, whether it is a major incident, like fire which prevent a site being used, or another situation that demands a different set of plans.

Computing: Do you replicate to the DR site and backup that site onto tape?

Edmonds: We back up all our servers to tape on a daily basis, and move some but not all of that data onto a DR site on a daily basis as well.

Bearpark: We have portable volumes that allow us to carry copies of the data on the SAN and rebuild it. We have done disaster recovery from tape for the last ten years and every single time we do it there are issues. Something has always changed meaning the procedures need updating.

Weston: All our servers are backed up once a week and the tapes are sent to a secure location. We have tapes going back six or seven years. We test recovery from tape, but as part of an outsourced service.

Computing: Do you trust cloud service providers enough to outsource your DR strategies to them?

Bearpark: I would not rule that out in the future, but private cloud services may be more appropriate for us than public cloud. The industry is still looking for a use for the technology and is trying all sorts badges to start selling it.

I am not totally convinced by the idea of everything as a service, cloud this and cloud that. It is a bit like the story of the emperor’s new clothes – it could be old or new, but might be nothing at all. I am also sceptical about putting that trust in somebody else’s hands.

Edmonds: Private cloud is where the Salvation Army is looking to go.