Several project managers hope that software testing automation will be the miraculous recipe to solve recurring problems of the testing phase: phase shifts, long waiting times for version delivery and high costs.
The problem arises when very simple cost models are used to make decisions about test automation. We present a guide to understanding the costs associated with this type of project.
Costs, risks, and benefits
In any business case analysis, costs, risks, and benefits must be considered. In our experience with cost cases, they can be grouped into two categories: initial costs and recurrent costs.
Initial costs:
- Evaluate and select the appropriate tool. In many cases, this step is ignored and then the associated problems manifest.
- Buy the tool or adapt open source tools, or develop a customized one. In the automated testing process, tools (for example, Selenium or QTP) only play a supporting role. The design of a test automation framework has a major role. It’s important to understand that the standard used to design this framework has a direct relationship with the success of the automation project.
- Prioritize test cases to be automated. For example, test cases that have a higher rate of reuse can be considered first in the prioritization (they are executed with the highest frequency in the current process: in smoke tests, in regression tests, in sanity tests, etc.)
- Generate test data to consume by the test scripts.
- Learn to use the tool properly. This includes the costs of knowledge transfer within the organization and the generation of new knowledge; for example, designing and documenting new test architecture.
- The cost of integrating the tool with the current testing process, the integration with other existing tools (for example, the tool for managing test cases or defect management, or continuous integration) and the integration with the current testing equipment.
Recurrent Costs:
- Maintenance of tool and test scripts. This cost is associated with the durability of a script, that is, how long the script lasts before it is modified. It can soon become a big problem if you do not think about minimizing this cost as you design. By making sure to establish an appropriate architecture (for example, the well-known design pattern Page Object Model) or by appropriately selecting the test cases to automate. Even when eventually starting the automation process in a system that receives constant changes in its functionality or user interface design.
- Maintenance of test data generation scripts.
- Execution cost of the test scripts and analyze the results. This cost can be estimated by multiplying the time in running the scripts by the frequency of execution and by the rate per unit of time.
- License renewal fees.
- Support rates for tools.
- Ongoing training, for example: new resources that are added to the team or tool updates.
- The cost of migrating the test scripts to new platforms.
- Extending the coverage to new application features that are being tested, or new applications.
- Dealing with tool availability problems, limitations, and dependencies.
- Establishing a continuous improvement of test scripts even though, like the first initial cost mentioned, this can be tempting to ignore. This is even more evident when you have a team of several people automating the test scripts of an application, which tends to increase the likelihood of tool use and script development evolving in incompatible ways and decreases the probability of reusing scripts.
When to automate the tests?
Another way to analyze costs is to think about fixed costs and variable costs. From the previous list, the fixed ones will be those that will remain regardless of the number of test cases that are automated. For example, tool purchase, training, licenses, etc.
Variable costs are those that change according to the number of test cases at a given time. For example, the development of test scripts or the development of test data.
The previous list will allow us to determine, with a very good level of detail, the cost related to a test automation project and the recurrent use of scripts generated in future test cycles.
However, when asked: when to automate the tests? We still have to analyze other costs associated with the project, such as the cost of opportunity in the automation of tests regarding manual tests or the recurrent cost of automated execution versus manual execution.
These topics will be addressed in future articles in order to determine when is the ideal moment to obtain maximum benefits from the process of testing automation.
