How LV= rolled out DevOps across IT

"Drink beer" - part of LV='s Anthony Hardy's advice for adopting DevOps

"I've always sat in the middle of development and operations even though I've never been called DevOps," says Anthony Hardy, infrastructure platform principal at insurance company Liverpool Victoria, which is probably better known today as LV=. "Now, it's funny to hear this word 'DevOps' because that's the kind of thing I've been doing for a long time," he adds.

According to Hardy, the company has been using DevOps principles for some time. It was an early adopter of Puppet Enterprise, one of the key tools behind the DevOps "revolution", in 2013 and had been using the open source version of Puppet for a long time before that. Today, it uses Puppet to orchestrate the operations of more than 1,000 Linux services, most running on virtual machines, across the organisation.

Hardy was explaining how LV= shifted to DevOps at Computing's IT Leaders' Forum this week - a task complicated by the fact that it is heavily regulated by the Financial Conduct Authority (FCA) and needs to adhere to other standards, too. "There's lots of constraints working in financial services. It would be great if we could develop stuff and put it into production however we want. But we have audit and PCI-DSS and all sorts of [regulatory] things that get in the way," says Hardy.

For Hardy, DevOps doesn't come down to a toolset, such as Chef and Puppet. "For me, it's a cultural change, involving communication, collaboration and mutual respect - just talking to each other. The new technologies come along as part of the consequences, but you don't need the whizzy new technologies to implement DevOps," says Hardy.

One of the key mistakes many organisations make, he adds, is that they simply don't plan for operations tasks when developing new apps and systems, and especially the consequences of shifting to an agile development methodology.

"On the development side, probably a majority of the developers have moved to agile. The business loves the flexibility, they love the 'show and tells', and they can see that stuff is getting done.

"What we have seen is that developers pick nice and easy targets for agile. They will typically build a nice, new shiny front-end with a couple of micro-services that has to connect to a legacy back-end. It's great, it looks good and the business likes it.

"But then integration infrastructure becomes an operations problem. This typically happens when we've had external consultants do the development bit. They don't quite get some of the operations side of things, and they go, 'What do you mean, three weeks for a new database? We can code a micro-service really quickly'," says Hardy.

However, this underplays the level of work that operations need to do in order to integrate those micro-services into the existing infrastructure, and to do so compliantly.

"Or there's 'our production launch was delayed because of operational issues'. Actually, there weren't actually any issues, you [developers] just never asked what needed to be done to put it into production. Why don't developers realise all the stuff that operations need to do?" he asks.

He continues: "It's easy to say, 'It's an ops problem because they didn't prepare for the production launch', but that's just taking the easy way out. Ops wants to build production-ready systems. Yes, developers don't ask the ops people, but ops people don't ask the developers either.

"I've seen ops people build what they think the developers will want for the final production launch. But all the developers want really is a VM to play with to build a micro-service on, they don't really care if it's production ready at that stage," he says.

That's why Hardy believes that communication is one of the keys to DevOps - and, indeed, one of the skills that he screens for when hiring.

"For me, DevOps is a cultural change, involving communication, collaboration and mutual respect - just talking to each other. The new technologies come along as part of the consequences, but you don't need the whizzy new technologies to implement DevOps. Play to people's strengths and encourage the sharing of knowledge, ideas and progress," he advises.

Hardy continues: "Practical steps that have helped us: encourage mutual respect, openness and honesty. Learn from failures; don't blame individuals. I've tried hard with my team where I've limited change reviews. I don't let individuals get blamed any more, and I very much try to understand and learn if something hasn't worked as it should."

Also, he suggests, planning for operational tasks in development. "Include those in the sprint backlog. Make sure that when you're planning an agile project and, as we have at LV=, still have a notionally separate operations team, that the operational tasks are included upfront," he says. That means bringing operations into the development process.

However, while DevOps isn't simply about tools that can help automate operations tasks, using a tool like Puppet, adds Hardy, also helps in terms of re-assuring regulators that the organisation is well on top of compliance and security issues. "I can say we use Puppet and I don't need to go into a great deal of detail about what we do. I can reassure the auditors that Puppet is going to keep things consistent," says Hardy.

That means, for instance, enforcing a particular build for servers upon which systems are developed across the organisation, and ensuring that they are appropriately patched. A tool like Puppet, furthermore, can prevent "rogue" engineers from knocking holes in systems with individual configuration changes - because the tool will simply put them back again.

It can also be used to enforce particular versions of software so that, for example, integration with legacy mainframes isn't inadvertently 'broken' by an enthusiastic engineer rolling up updates.

However, arguably Hardy's best suggestion is not just to bring the teams together so that they can exchange expertise, but to "drink beer" - to build links between teams, as well as mutual respect and understanding.

Computing's DevOps Summit 2016 takes place in London in July - and places are going fast. Learn more about how other end-user organisations are approaching DevOps, and their advice for doing it right. Places are free to qualifying IT professionals, so register now.