The Importance of KPIs
Every project we work on, whether it is Quality Assurance, Development, or User Experience, is measured with Key Performance Indicators (KPI). Without KPIs, we are working without a direction, or at least without a way to measure our direction and effort.
How to Evaluate KPIs
To evaluate the data captured as a KPI, it is necessary to be able to look at the progression. For example, the number of Test Cases (TCs) that are not being executed on time becomes a KPI when transformed into a percentage and compared to the total number of TCs that are being executed, or when comparing numbers from previous executions.
Time is one of the most valuable indicators related to measurement, especially as a way to measure deviation from the initial deadline estimates. This indicator will evolve throughout the execution of the project.
For example, the following graph represents the deviation between Total Planned and Total Real, though the situation was even more serious at the end of the month when the Total Real was not even close to the Total Planned.
This case cannot tell the whole story. We don’t know if the execution was affected by an external factor from the beginning, for example. That is why it’s necessary to use this in conjunction with other indicators and with practical information that we gather from conversations with the team members.
How to Convert Indicators and Data into Useful Metrics
Let’s look at a method that we can use to convert indicators and data in useful metrics that help us make better decisions in QA projects.
We want to create a metric that measures the difference between estimated and actual in a typical QA Project. Focus on the TC estimation for a new project. First, we need to start by asking the right questions:
- What is the daily goal of estimated TCs?
- What is the real quantity of estimated TCs per day?
- What is the gap between estimated and real?
In order to make sure we are on the same page, here’s some definitions of concepts we are using.
Planned TCs
The number of TCs planned to be created per day on the Test Case management tool that will be ready to be run on the next cycle planned.
Real
The actual number of TCs created on the Test Case management tool per day that will be ready to be run on the next cycle ready done.
Total Planned
The total number of TCs planned to be created per day on the Test Case management tool that will be ready to be run on the next cycle planned.
Total Real
The total number of TCs actually created on the Test Case management tool per day that will be ready to be run on the next cycle ready done.
How to Calculate Metrics
Here’s how the metrics will be calculated, by defining the functions:
Once we have the concepts and how to calculate the metrics clear, it's time to define the Measurement Methods.
Defining Measurement Methods
Important information you need to include:
- Where is the previous value going to be calculated? Excel - Quip - etc.
- Person in charge of collecting data
- Location and link to the compilation document
- Frequency/time of data collection i.e. every morning, this person will update the recollection document with the data from the day before.
Once we have the metric design, the next logical question is: What do I have to do with all this information? How or when do I make decisions?
It's the moment to define the decision criteria, i.e.
Defining the Decision Criteria
If the Gap is higher than a 5%, the team will need to review the situation and define a strategy to recover the gap
It's crucial for the team that the person in charge of collecting data pay attention to the numbers and be aware that any deviation in the gap can have a positive or negative impact on the team results.
Reporting Mechanism
Let’s define how we will present the data, distribution methods, metric availability, i.e.
Additionally, we can add additional qualifiers, like By resource, By component, By Subproject, or even a metric by department, location, team, etc.
Calculating the metric
Following our initial idea, we want to track the test case estimation for a new project. We have 8 days to estimate 500 TCs for the existing components, with 2 resources estimating TCs cases 4 hours by day.
We notice how the gap was under 5% from the decision criteria. Always remember to ask your team members what is blocking or impeding the progress. During the following days, the team recovered the gap and ended up above the planned, keeping the gap over 5.
Conclusion
Metrics are crucial in keeping teams and customers informed and aware of how a project or company is performing. A comprehensive business dashboard is an effective way to stay on top of your project performance.
