Industry Voice: How to advance your DevOps capability and measure what matters

Darren Sharp
Industry Voice: How to advance your DevOps capability and measure what matters

There is a temptation, once operating under the DevOps model, to rest on your laurels, content that you are now working in a modern development environment. The upshot is many organisations don't measure the outcomes brought about through DevOps. They've done the hard bit, undergoing the initial technology, people, and process changes that the model demands, but are leaving unknown benefits on the table by not measuring how the approach is now performing.

By not measuring DevOps performance (or even knowing what KPI's are important in the first place) you have no way of knowing whether you're equipped to deliver the improvements that will realise the greatest impact to your business. It's a common failing, highlighted by Automation Logic's Kris Saxton at Computing's DevOps Live conference earlier this year,

"Many of our clients, when we join them, aren't really monitoring anything relating to DevOps performance. They usually have some fairly large, abstract goal relating to the thing they're trying to deliver - be it a programme or a service - but none of them are systematically measuring whether they're getting better at the act of delivery itself.

"To me that seems really odd. It would be like taking up a sport - or any activity that involves skill and practice - and not measuring performance to understand whether you're getting better or worse at it.

"Doing DevOps without measuring is a bit like playing archery in the dark. Not only are you less likely to hit the target, but you take your shots and the darkness can last for several months in terms of understanding whether you got better or worse. You've got no way of correlating the actions that you took with the outcomes."

Measuring DevOps success

The DevOps approach makes demands across technology, people, and process. This complex interaction means measuring DevOps success across the board requires soft metrics such as people-oriented questionnaires and workshops, alongside quantitative hard metrics. A successful DevOps setup will promote and track performance across the following areas:




Continuous improvement


Deployment automation



Environment automation



Continuous integration



Infrastructure as a Code

Roles & expectations


Test automation


Quality assurance

Metrics & monitoring





These metrics are crucial for motivating high-performance across a team, demonstrating a return to your organisation's leaders, and being able to share progress with stakeholders to unlock investment.

Focus on transparency

Automation Logic Head of Delivery Nick Barton's approach to ensuring a motivated and productive team involves tracking problematic symptoms back to their root causes.

"When assessing a team's performance, I generally start with a list of the symptoms being experienced, perhaps track a list of points raised in retrospectives (you are doing them, right?), and all the times where something has needed escalation. Look at long-running tickets and perform an analysis of the workflow. Are tickets getting blocked? Is the team always waiting for responses from key individuals? It's important at this stage not to jump straight to resolving these symptoms, because as the name suggests they are just symptomatic of underlying issues. Look for common themes and try to understand what the root causes could be.

"Some areas to look at are; How is the work coming into your team, is there a clearly defined process? Do the team understand the overall importance of what they are doing? Do they even understand the aims of the programme or project they are working on?

"If engineers are not following good agile processes, do they understand the benefit of the ceremonies being performed? What is the point of a daily stand up, or sprint planning? Are you, as a manager, constantly bypassing the agile process by consistently changing priorities, adding work into a sprint or distracting the team with non-essential, non-sprint related work?"

Inspect & adapt

This need for transparency and understanding extends to your interpretation of performance measurements too. It is important to understand the results of your metrics and what they mean in context. Automation Logic tries to plot the impact on DevOps capability/performance from focusing on one area over another - this helps to build a roadmap by finding areas to prioritise.

The scores, analysis and roadmap of measurements must be tied together into a framework, which requires an approach to change known as empirical process control (EPC). The three pillars of EPC are transparency, inspect and adaptation. You must be transparent about what you're measuring and the results; inspect and draw insights from the results gathered; and use the insights to adapt and plan.

Head to Automation Logic's dedicated benchmarking site DevOps Advance, where you can gain a better understanding of your DevOps performance today, across technology, process, and people, and how to take it to the next level. Get the results you need to:

- assess capabilities

- identify weaknesses

- adopt best practices

- accelerate speed, agility and quality of delivery

Take the digital assessment today to discover how you fare, what might be holding you back and, ultimately, where to focus to improve your DevOps performance.

More on Networks