How HMRC did DevOps

Andrew Sheppard, product owner, digital tax platform at HMRC explains that the taxman never set out to work in a DevOps way, it just happened...

HMRC is a huge organisation, with annual revenues of £536.8 billion generated from what Andrew Sheppard, product owner, digital tax platform at the organisation describes as "tens of millions of reluctant customers".

In the past the organisation managed several large, unweildy long-term outsourcing contracts, and ran quarterly release cycles.

"Things had to change," said Sheppard, speaking at Computing's recent DevOps Summit. He explained that outsourcing contracts were reduced to a maximum of two years, with an option of a third.

"And we stopped doing Waterfall," he added, referring to the project management methodology. "But we do still work with third party suppliers who work in a waterfall way, so there's some friction there," Sheppard admitted.

He described the organisation today as "fully agile", and explained that HMRC now engages wherever possible with SMEs.

"But it doesn't always work, as there isn't always the right technology available."

Going back to the release cycle, there were only two windows for major software released each year, in April and October.

"These releases would knock out major systems for up to five days at a time. We had a legacy Java app on the HRMC website, and any update to it would cause all its services to shut down.

"We didn't know what the user needs were, we had no concept of the user," Sheppard continued. "We realised we needed to tackle some problems. We needed to get changes in front of users asap, thus shortening the feedback cycle.

"We had to convince the business of the value of getting them emebedded with developers, and talking to users, not locking down requirements 18 months in advance, but instead get out there on the street, drag people into user testing labs and talk to them."

HMRC also needed to understand the concept of the minimum viable product.

"We had to convince the business that it's okay to put something out there. They were reluctant to put anything out which wasn't already fully formed, but the more quickly you can have users interacting with it, the faster it turns into something that's feature-rich," he said.

The most common stumbling block with DevOps transformations is cultural resistance. Sheppard said that this fact was recognised at HMRC.

"Without senior support you can't affect cultural change. We had digital leadership from the board-level down. So everybody was pushing the message fromt he top, then we developed a few exemplar services, proving it from the ground up too."

He explained that senior leaders had Sheppard team "on a long lead", with the rest of IT watching the process and learning from it.

"And the more trust and confidence we built, the longer the lead got," he added.

"Automation is key for us," Sheppard continued. "We forced service teams to do their own deployments into production. It's all underpinned by loads of micro services and APIs. The API platform is there for third party providers to build their services onto," he said.

This strategy of devolving responsibility continued when HMRC disbanded its support teams in digital operations earlier this year.

"The best way to instil ownership is to break teams down into specialisms. Now they're responsible for the full-lifecycle of their technology stack," he said, explaining that developers never just hand their code over once its live.

All of which serves to make HMRC more efficient, and certainly faster at deploying code. But is it still secure?

"We store everything in MongoDB, and our protected zone is not exposed to internet," said Sheppard. "The public zone [which is exposed to the internet] has no customer data in it. Sometimes service teams inexplicably store large amounts of data in the protected zone, which I strongly discourage. But there's nothing dangerous in the public zone. Our use of Mongo is there to be purely transient, the data then goes to systems of record, where it's stored," he concluded.