Let’s start by asking a simple question: Why is measuring important? By measuring our actions we are able to know how we are doing on our tasks, how long they are taking, and how much they are costing. This information can be used to improve our processes and quality. Plus, we can apply these metrics and techniques to our activities to get important information, which we can use to make better decisions.
These decisions lead us to take actions. Quality level improvement depends on the metrics and is why we should choose ones that help judge productivity and efficiency.
“If something cannot be measured, it cannot be managed”
Best practices must always be included in the tasks of a good QA team, which includes continuous learning, improving processes, giving insights into test progress, productivity and quality. Some common questions that a QA team has to deal with are: How long will it take to test? How much software was tested? How good were the tests? In order to answer these questions, a good measurement is needed. Using metrics a QA team can track their own progress and analyze not only if the goals are met, but also if all the resources are being used to their maximum capacity.
In today’s world, we do not need to only execute test cases, but also communicate progress throughout the software development lifecycle and the metrics used for executive decisions. The metrics provide guidance about time, cost, resources and are also used for team decision making, evaluating the strengths, weakness and capabilities of each team member, and motivating the QA team to maximize productivity. Additionally, metrics help us understand problems in the process and customer expectations.
Each metric has an objective or goal, and something that will be measured. For instance, the two metrics below explain their objectives clearly:
- Metric One: “Number of defects found on a specific build” - Measures the stability of builds or compares various builds. If number of defects decreases, it means build is becoming stable.
- Metric Two: “Time to execute regression test cases”- Measures testing efficiency. The first time should be higher than all the following tests.
|
Metric |
Purpose |
|
Number of Defects |
The total number of defects found in a given build that resulted in software or documentation modifications. Only accepted defects that resulted in modifying the software or the documentation are counted |
|
Time to execute regression test cases |
Effort required to execute test cases on a regression cycle |
Once the metrics are chosen, the team needs to interpret them so they can be managed correctly and carefully since big data is a big responsibility. Always make sure that you get some value from whatever measurement you are using. Metrics are helpful tools, however they can be used for good and for evil, to tell you the story you want to hear. The interpretation gives the information, the information gives the knowledge, and the knowledge is essential to make the right decisions.
Number of defects found – let’s consider a team that has executed two cycles of testing and the number of defects logged in each of these cycles is displayed below:
|
Number of defects |
||
|
Defect Severity |
Cycle 1 |
Cycle 2 |
|
s1 |
5 |
5 |
|
s2 |
10 |
20 |
|
s3 |
30 |
10 |
|
Total |
45 |
35 |
Looking into the number of defects logged and their severity, we see the total number of Severity 1, Severity 2 and Severity 3 defects for cycle 1 is 45, and the number of Severity 1, Severity 2 and Severity 3 defects for cycle 2 is 35. In terms of quality, cycle 1 is more favorable than cycle 2 as cycle 1 has a lower number of high severity defects, even when the total number of defects logged in cycle 1 is more than the defects in cycle 2.
There are some characteristics that metrics must have. They should be simple to collect, interpret, maintain, and should be sensitive to changes while not being ambiguous. Also, the return from the metric should be far greater than the cost of the data collection, calculation and interpretation.
Conclusion
To summarize, measuring is the secret ingredient for a successful QA process. It tracks progress versus actual progress, costs (money spent), defects, test cases, man-hours, etc. QA metrics definitely helps us understand the processes, the test reporting, and also visibility shows the quality of the software product. It is effective at managing the software development process.
References
https://www.qasymphony.com/blog/64-test-metrics/
https://blog.qasource.com/performance-metrics-of-your-qa-team/
http://www.mindlance.com/documents/test_management/testing_metrics.pdf
