HMRC takes 'grow your own' approach to overcome DevOps skills shortages

HMRC Digital interim head Andrew Sheppard explains how the taxman shifted to DevOps - and takes on apprenticeships to fill the skills gap

HMRC has taken a ‘grow your own' approach to DevOps engineers with an apprenticeship scheme that takes students straight from school, as well as older career switchers.

Andrew Sheppard, the interim head of digital operations at HMRC, detailed the organisation's strategy for overcoming DevOps IT skills shortages at Computing's DevOps Live 2019 event last week.

"We've had limited success over the years in bringing permanent people into those really niche IT roles, like DevOps engineer and, to a lesser extent, software developers," said Sheppard.

HMRC, he continued, is reliant on contractors to cover many specialist elements, but added that the organisation needs to shift the balance more in the direction of full-time permanent staff.

"I think it's absolutely essential to hang on to a core of very experienced contractors who are coming in to the organisation and bringing fresh ideas in; that's not all they do, of course. They are also there to uplift and help the development of permanent staff."

Line managers are expected to act as talent spotters and to prepare permanent staff for the step-up if they feel that they ought to be promoted.

In addition to the Civil Service Fast Stream, HMRC also uses industrial placements - people who have finished the second year of their computer science degree who go to work at HMRC for 12 months. "Hopefully, they will remember us fondly," said Sheppard. "We give them lots of interesting and challenging work. They go back to university, finish their degree and then we keep in touch with them and try to bring them back."

However, apprenticeships have been a particular focus for Sheppard, both in terms of taking teenagers interested in a career in IT - the scheme is primarily aimed at school leavers with A' Levels under their belt - as well as some career changers.

"We have just started dabbling in apprenticeships. About two years ago, because of the difficulties we'd had in bringing people in at higher grades, we thought we'd create our own capability within Digital Operations, mainly as a feeder to get onto the platform team.

"What we were looking for, therefore, were infrastructure engineers and maybe the odd software developer. When we started looking into this we realised pretty quickly that there is no such thing as a Level 4 DevOps engineer DevOps standard [for apprenticeships]. So we thought we would put in a bid for people on the Level 4 software developer scheme. We got our first four in December 2017 and took-on another five last December," said Sheppard, explaining how HMRC's apprenticeship scheme got going.

"So, we thought of it is a good source of infrastructure, provided that the individual hasn't got their heart set on ‘just' being a software developer. We would have a quiet word with them before they turn up to explain that, although they are on the software developer scheme, we want to turn them into something else."

No-one has turned down the opportunity so far. Of the first four, two remain - the other two didn't have an interest, after all, HMRC Digital found.

Two, however, transferred-in early on from other parts of HMRC at the same time, so the organisation still has four students in their second year. All five of the second cohort, however, have fitted in well.

"They are all very good. The ones we brought-in in 2017 were all straight out of school after A' Levels, they had not had any work experience before, apart from working in a shop. The five we brought in last year were all mid-to-late 20s. So they had tried stuff, tried and failed in some respects. But they've made the decision that they are going to take this very seriously - this is their career; this is what they want to do.

"None of the people we've brought in so far has a history of coding. None of them had written any code before they turned up. Yet, two of the four brought-in in December 2017 have got that aptitude and have already left ‘team apprentice' and moved into the platform teams. The other five are all working now at around the same level," said Sheppard.

And this is despite the relatively lack of knowledge, as well as experience, that some of the apprentices came in with. "One of them literally did not know where all the keys on the keyboard were that he needed to use."

Sheppard attributes the success of the scheme to the work of the trainer brought-in to develop the apprentices. Initially, he said, the first tranche of apprentices were spread out among different teams. "That was a mistake because they couldn't do anything, because they were literally taken straight after their A Levels."

Instead, the trainer was brought in to put-in place a curriculum and design a training regime for them. They were therefore pulled out of their teams for training, with the tranche the following year also put through the regime. Although the apprenticeship is two-years long, within 12 months they were ready to join one of the teams. "They needed less than 12 months before they were ready to actively contribute and ‘pair' with experienced engineers and developers," said Sheppard.

The curriculum itself includes everything you would need to "build a cutting edge public cloud", he added. The first month involves learning the basics, with an ‘outcome' at the end of each month's learning. "They literally have to demo - not just to the trainer and each other, but to the platform teams as well - that they have met the learning requirements for that month."

February focused on NoSQL database technology - in particular, because HMRC makes heavy use of MongoDB. They also have licences for various online training websites, such as Linux Academy, which they are expected to make use of on their own account - not expect to be spoon-fed everything by their trainer.

Since the project was started - because there was no such thing as a Level 4 Apprenticeship for DevOps engineers - other organisations have followed suit, added Sheppard. "We've got the Trailblazer Group, which I'm the chair of, we have 12 government departments and now more than 20 private sector organisations, including a healthy mix of SMEs, such as Zen Internet, right up to Capgemini, Credit Suisse and IBM."

[Next page: More on HMRC's digital platforms and its three core 'platform principles']

Miss DevOps Live 2019? Don't miss out on any future Computing events: check out Computing's dedicated events website - all events are free to qualifying CIO, IT directors, managers and senior IT pros, but places always go fast

HMRC takes 'grow your own' approach to overcome DevOps skills shortages

HMRC Digital interim head Andrew Sheppard explains how the taxman shifted to DevOps - and takes on apprenticeships to fill the skills gap

Digital platforms

HMRC Digital is behind the organisation's various digital platforms, including HMRC's self-assessment tax portal. The introduction of DevOps has brought with it a shake-up in HMRC Digital, said Sheppard.

"There are six core platform teams. The seventh team is ‘team apprentice'. Back in April 2017, we disbanded the platform-support team. We formed into technical specialisms so that every single specialism fully owned the tech' stack. So there's nobody cleaning up their [developers'] mess any more - they have to do it themselves.

"So we have a team that provides all of the collaboration tooling. We have another team that provides a lot of expertise on software development and frameworks needed to do that. A build-and-deploy team that owns the CI/DC pipeline, a telemetry team that owns the login and metrics stacks, and also an infrastructure team responsible for building the vast majority of the infrastructure and platform.

"There's also a platform security team - a DevSecOps capability that works across all of HMRC Digital, they also touch on traditional security… underpinning everything that our teams do is providing a consultancy service across, not just the digital community, but as many as 4,000 users across HMRC's estate, particularly of our collaboration tools like Jira and Confluence. We open those up to non-digital projects for them to build team spaces in there as well," said Sheppard.

HMRC Digital, obviously, counts "tens of millions of customers" in the real-world that need to be served, but also many parts of HMRC internally, too.

"There are around 150 customer facing digital services on the platform at the moment. I can't quite keep up with the numbers. We reached a point about two years ago when we no longer need to increase the size of the teams or people within the team - we've now reached an equilibrium. So it doesn't really matter how many more services land on the platform now, we can cope with it."

"Underpinning it is a typical micro-services architecture with around 400 production micro-services. There are about 63 that are old Java ones, which coalesce into those 25 customer-facing ones. It's MongoDB on the backend. We have some Oracle on RDS, but otherwise it's MongoDB all the way. And, of course, backend connectivity through to the legacy estate."

This all exists for one main end: "On the 31^st January this year, the culmination of everybody's favourite business event - the self-assessment filing deadline where we collected one-and-a-half billion logs of metrics in a 24-hour period."

At this time, everything needs to stay up for an ‘event' that is every bit as demanding on HMRC's infrastructure as Black Friday is on Amazon's.

NHS Digital, Sheppard continued, employs three broad ‘platform principles' that all staff are expected to buy-in to.

#1, ownership. "That's the DevOps, mantra, of course - you build it, you run it. We obviously practice that on platform. Although we don't control or direct the work of the service teams - we just provide them with, more or less, a black box to drop their services onto - we do insist that they take ownership of their services… We don't provide any level of application support. We just make sure that the platform is entirely available, performant and secure."

#2, collaboration. "We do a lot of work on close collaboration… Slack has been a game-changer for us. We used to use Skype, years ago, then we tried HipChat, then Slack and haven't looked back. Pairing is encouraged… remote pairing or face-to-face. We also encourage teams to do tech talks and demos. It's also a good way for permanent staff, when they start, to help them to build their confidence."

#3, self-service. "We do this by a number of means. About four years ago, I got two of the engineers in my team to start building something that last year became a more-or-less feature-complete user management portal. Up until that time, platform teams had to raise Jira tickets in order to get somebody on-boarded onto their project, to get somebody access to tools, or to off-board somebody. That's all now self-service."

"We disbanded our platform support team in 2107 as part of the move for everyone to take full responsibility for their tech stack. At the same time we enabled self-deployment and self-shuttering. It seems rather archaic now, but if service team wanted to do a deployment into production… when they got to production, they'd have to raise a Jira ticket. Then a platform engineer would have to deploy their code into production for them, while that service team sat on Slack."

"We used to do those deployments, 9am-4pm, Monday to Thursday and some critical bug fixes on Friday. It's 24/7 now - we don't place any restrictions on our teams. But in reality they are all very sensible and nobody deploys at 3am on Christmas day. They have got used to deploying a little earlier in the morning and sometimes deploying as late as 5pm in the afternoon."

"Shuttering a service usually happens because something on the backend has fallen over. It used to be that someone would have to raise a Jira ticket or they would ring or Slack us and ask us to shutter their service, and then we would have to stick around until they were ready to unshutter the service."