Measuring the Success of DevOps

Prashant Arora DevOps Metrics ITIL Blog

DevOps, the sizzling hot buzzword, is being used in many organizations today as they plan towards the roadmap or already in the midway of implementation. If you are one of those early movers, one question you will likely to face if not today then tomorrow, How to Measure the Success of DevOps?

While dealing with the conflicting priorities of stability versus responsiveness, people, processes and technology play a vital role. Leveraging the best in class tools for code reviews, builds automation and rapid deployments can give you some great tangible benefits but you must have right metrics in place to measure and improve your DevOps capabilities.

Needless to say, baselining the current metrics is very important and should be early in your DevOps roadmap to have comparisons later in time.

Here are few DevOps Metrics which can be used to see if it is worth spending time and money in your DevOps journey:

Broken Branches

Code compilation issues or code conflicts can result in broken branches. In an environment where multiple development teams are writing and merging the codes, there are greater chances of such issues which are uncovered only during the integration testing or may be later. As a successful implementation of DevOps tool set and processes, the Broken Branches can go all the way to zero! Measure it to see the decreasing trend.

Code Coverage

While you are moving away from time taking peer review process for code review to automated ones, it is important to track the Code Coverage percentage.
Code Coverage is a measurement of how many lines of your code are executed while the automated tests are running. An increase in percentage shows that it is comparatively safer to commit. Leverage some good tools like Phabricator, Gerrit or Review Board to achieve this.

Time to Deploy

Overall time taken to deploy should ideally be reduced if you have right tools and right skills in place. Time reduction in every activity from development to deployment results in the overall lead time reduction and enables more number of releases per day, week or month.

Shorter the time to deploy, greater the deployment frequency!

Deployment Frequency

A healthy CI/CD* delivery pipeline and less lead time from development to deployment can facilitate increase in the deployment frequency. This rapid delivery can never go unnoticed from the stakeholders and specially your customers.
Change Volume is another metrics to be tracked. A significant increase in total number of change tickets can tell your success story. This is your steps towards better business agility!

*CI/CD = Continuous Integration / Continuous Delivery

Failed Deployment

Increase in change and deployment is one metrics but capturing the failed deployment as a stress metric is perhaps a good idea as it keeps a check on stability. We all are looking forward toward responsiveness WITH stability, isn’t it?
A decreasing or neutral trend is way to go!

Defect Density

Well, no defect at all is an ideal state but kind of unrealistic too. So let’s accept the defects but at the same time measure the defect density and look for a thinner percentage.

Lower the number, better the product; happy customers, happy you!

Mean Time to Resolve (MTTR)

Mean time to resolve represents the average time required to fix an issue. While your DevOps efforts have ensured early engagement of support teams, it is the time to see if that has benefited or not. A gradual dip in MTTR shows that your teams have understood the concepts and actually collaborating in real time to give better customer experience.

In addition to these, there are some other metrics worth mentioning such as, CI Seepage, Regression Defects, Incident Volume, Frequency of Major Incidents and Response Time.

In my next blog, I will be mapping some of these metrics to the Business Value Metrics to see what we get in return, in terms of time, cost and quality. Stay tuned…

Meanwhile, if you believe there are some other metrics which is worth measuring, feel free to mention in comments below or tweet with #DevOpsMetrics.