Of course, those of us with roots in product development and engineering have read the Agile Manifesto and from it we’ve learned many values, including “Working software is the primary measure of progress.” But that is quickly changing, given today’s business trend to scale Agile across the organization. The intention of companies to become more Agile to fuel Digital Transformation initiatives is growing and making mainstream business headlines. (See Daitan’s CEO, Augusto Cavalcanti’s latest blog: How the C-Suite can Scale Agile from an Engineering Practice to Business Value and Harvard Business Review’s Agile at Scale).
The movement to scale Agile across organizations has led to lots of focus on, not only best practices and methods for implementation, but also the right way to measure success. In this article, we want to discuss the distinction between Agile analytics that quantify “output” versus those that measure business “outcomes.”
"At Encora, we believe output measures are helpful, but outcome measures are truly where businesses gain insights that guide senior executives on changes and investments needed to grow."
Agile Output Analytics Quantify
A large part of Agile best practices is to use analytics to quantify what’s been delivered to the business; and from the engineering/product development perspective, those analytics have been largely focused on the “output.” There are a variety of examples—such as, how many lines of code, the velocity of sprints and production releases, percentage of automated test coverage, control charts for cycle times, epic and release burn downs, release frequency or time-to-value, post production defects and support calls and so on. They are the tangible proof points demonstrating progress.
But, these types of output measures look at what was produced, they don’t measure a business outcome. While, this information is certainly useful and important—particularly when Agile is new to a company and you want to understand how you have accelerated development and product releases—the business needs more.
Agile Outcome Analytics Make the Business Value Clear
It’s been said that output measures what is produced, whereas outcomes measure the business result. Outcome analytics speak to product lines, business lines and the C-suite because they show the value of Agile in terms that relate to the customer experience and business performance—such as, more sales, or more customers, or more sales per customer, longer customer retention, and so on.
Too often we see Agile Teams attempt to present the benefit of Agile to the C-Suite, focusing too much on output analytics, which falls on deaf ears because senior executives think and operate based on business outcomes.
When you want the C-Suite to understand the value of investing in Agile adoption, the analytics are far more impactful if you correlate that an increase in release velocity by 200% (output) during the quarter yielded increased revenues by 40% (outcome) and an increase in new customer sales by 20% (outcome).
"Understanding the customer needs and creating a delightful customer experience enables Agile organizations to correlate output productivity to business value outcomes, which provides a complete picture around the ROI from being Agile."
Analytics can also relate to internal business measures such as developer retention—a key concern among many tech companies. In this example, investment in DevOps improves automation and accelerates release cycles (output), which correlates to happier developers and improved retention.
Thinking about analytics in terms of business outcomes also helps the organization shift their mindset from internal to external—in other words, about the customer. Let’s take another example, improvements that streamline the user experience by reducing the steps (clicks) to make a purchase (output) correlates to a lower incidence of shopping cart abandonment (outcome) leading to increased sales (outcome). Or, accelerating time-to-recovery from failures (output) leads to less revenue lost due to lost transactions (outcome).
Analytics Should Be Central to Every Agile Deployment
No matter how small or big your Agile implementation may be, analytics matter. Analytics help drive communication, collaboration and orchestration within individual Agile teams, across teams and to team-adjacent roles such as, product owner, DevOps and portfolio management—as well as with other departments and executives.
Having no measures leads to negative siloed behaviors and old school, command-and-control style management that will likely fail to achieve the most important benefits of being Agile—rapid iterations and continuous delivery.
"When analytics are purpose built to quantify the outputs, as well as reveal business outcomes, they truly fuel the expansion of an Agile mindset across the organization because the analytics make the value obvious and relevant."
Analytics are not a one size-fits-all answer, but there are excellent resources available to help you get started. Gene Kim’s book, The DevOps Handbook has excellent discussion on the concept of value streams, which is integral to understanding business outcomes. It describes metrics from development through operations, including team metrics, portfolio/application metrics (which includes those parameters tied to customer experience), infrastructure metrics and business metrics.
We like the author’s approach, because it’s important to not only understand performance for each area, but also recognize interdependencies across them. For example, teams that are fully Agile and delivering new releases with greater frequency and quality, can be impacted if the infrastructure is unable to handle the higher load so production releases don’t increase, and that ultimately ripples up to the business failing to yield any benefit from their Agile investment. It all relates.
We find that the best approach is to start with practical measures that analyze related outputs and outcomes; and then expand based on cross-functional input around what provides the most meaningful insights about the customer experience to all stakeholders. If you have questions or comments, feel free to contact us and let Encora help you expand your Agile initiatives.
