How the DWP is ridding itself of 40 years of technical debt

The DWP's IT estate had become convoluted and complex after four decades of outsourcing, but virtualisation and the public cloud is cutting through the tangle

The Department for Work and Pensions (DWP) has massive technical debt. It offers over 100 products, holds data on every UK citizen and runs more than 850 branches - 700 of which are customer-facing job centres. When it comes to IT, the DWP still runs many systems and apps on legacy mainframes.

All of this means that moving to the cloud presented a significant challenge for John Keegan, Head of Infrastructure Services at the DWP, who spoke about the organisation's ongoing digital transformation at Computing's Cloud and Infrastructure Summit North this week.

"Historically, we've had a lot of big contracts with big SIs," Keegan told delegates. "We've been looking to combine those services together, take a lot of services back in house and then transform the way that we can do stuff, so that can get it out to public clouds and more COTS-based services."

DWP has spent the last two years leaving those big hosting contracts, but in the process realised that it wasn't ready to make the full transition to the public cloud - which would have meant dealing with four decades of technical debt.

The next step was the introduction of Universal Credit: a new service that replaces more than 12 existing benefits payments, including Jobseeker's Allowance and Income Support. It is a modern product based on microservices, with no legacy: "Instead of having to re-engineer 40 or 50 million lines of code, we started from scratch," said Keegan.

DWP is running Universal Credit today on AWS, and the consumer-facing side is a web-based app. This can be accessed at home or on the new guest WiFi and self-service ‘fat client' kiosks (replacing the existing 90,000 thin clients) at all of DWP's job centres.

Replacing old tin with new

"We've been moving a whole bunch of applications across [to the cloud]…and managing traditional applications and moving those to new data centres and hybrid services." The legacy tin still exists though, including VME and MVS mainframes. "We've still got those today, serving somebody's key systems that's paying out £170 billion. We can't turn those things off overnight."

Instead, DWP took its apps and ascertained which ones it could and couldn't virtualise, then lifted and shifted the former into new data centres, with a gateway into AWS, Azure, ServiceNow and "all the big SaaS cloud providers."

Virtualising the applications took "about a year" and involved consolidating down from "lots of old tin" to "newer, smaller tin". Some of the batch jobs that run overnight are now up to 40 per cent faster as a result.

"Because we had so many old applications...we couldn't just migrate it all in one weekend. It took us about a year once we'd built the infrastructure to migrate those applications.

"To do that, and to replicate 10 petabytes of data - the silly amounts of data that we've got are key, because a lot of legislation means we've got to keep it for a long time... - [required] a dedicated optical network, and then making sure...[that] all the things it talked to - and some of them talk to 20, 30, 40 different things - still work.

"That's really hard, but we've done it. Now we understand those, so the journey to cloud - proper cloud, not just hybrid cloud - is a lot easier."

DWP is still migrating services from its new data centres to the public cloud "week by week"; Keegan said, "AWS has been key for us [in this migration]."

Keegan was critical of the state that DWP's IT infrastructure was in, and blamed it on the public sector penchant for giving work to contractors:

"A lot of this has got into this mess because of 20 years of outsourcing. We couldn't just award a contract to somebody else to go and fix it all - we had to bring it in-house to understand it [and] rationalise it. Going forward, we're moving things to SaaS-based services [and] managed services. It is part of that journey to make sure that we're doing the right thing for these applications."

As well as more flexibility and reliability, DWP is aiming to get to the elusive goal of ‘zero service hours lost' with its cloud transition. "That's a hard thing to do," said Keegan. "We're tracking the applications, so that we can make sure that we're monitoring [the right ones]; we've got the right resilience...[and] recovery built into things as well. Lots of automation is part of that journey."