In a previous article (Efficient API Test Automation: Exploring the power of GenAI and Schema-Driven Test Generation), we explored the potential of employing Generative AI tools to generate API automated tests based on a provided API Schema. In this piece, we are going to dive into the practical approach, hands-on methods, prompt techniques, and how they impact the quality of the results and, ultimately, we’ll determine the real-world usefulness of the outcomes generated by the tool.
The tool:
We decided to do a first proof of concept using one of the most popular and powerful Generative AI tools, ChatGPT.
The schema:
We are using a popular testing API from Swagger https://petstore.swagger.io/v2/swagger.json
This is a JSON file that contains the API schema of a virtual pet shop. The API contains some of the most typical endpoint's operations. To maintain brevity in this article, we will show the results of using just one endpoint:
The prompts:
A prompt guides the direction of an AI's response, serving as the catalyst for relevant and contextually appropriate output. In essence, a prompt acts as both a directive and a filter. It directs the AI on what to focus on and filters out unwanted or irrelevant information, ensuring that the response is in line with the user's intention.
We conducted experiments using various prompts to assess how ChatGPT responds to each, ranging from basic prompts with minimal information to more detailed prompts with additional context and instructions. For the purpose of this article, we will only present the results from the more complex prompt.
Act like an SDET. Analyze https://petstore.swagger.io/v2/swagger.json. You want to focus specifically on the endpoint “/pet”. Create tests using Jest and Supertest. Additionally, provide valid and invalid request payloads, add assertions based on the expected response of the endpoint, and consider any negative and positive scenarios.
The results:
First, it provides a step-by-step guide on how to set up the project.
Then, it generates a set of tests to cover different scenarios for that endpoint
Findings:
- It provides a step-by-step guide to set up the project.
- It generated a few tests for the requested endpoint and included test payloads for valid and invalid scenarios.
- Overall, the tests generated are of good quality, but there is room for improvement. This can be achieved either by instructing the AI to make enhancements or by allowing an automation engineer to modify the code as required.
- It also specifies that the code generated it’s “a simplified example, and in a real-world scenario, you would want to cover more edge cases”
Link to the chat: https://chat.openai.com/share/41930e16-a9d8-49c0-8492-f598e782a112
After observing these results, let’s explore whether ChatGPT can generate nonfunctional tests, such as Security tests.
It starts by identifying the potential security concerns for the given endpoint.
Then, it generates a set of security tests based on the risk analysis it made.
Finally, it explains the tests created and makes a recommendation about security assessments.
Findings:
- ChatGPT was able to identify the security concerns of the given endpoint.
- It gives a detailed explanation of the risk assessment, which is helpful for the user to understand the potential vulnerabilities.
- ChatGPT generated a set of security tests based only on the analysis it made of the given schema.
Link to the chat: https://chat.openai.com/share/548e1318-5dc0-4fc5-9b50-860beb756253
Key takeaways:
- When a user provides a JSON schema or the specific data for the endpoints they wish to test within a single prompt, ChatGPT has the capability to generate an API test automation project from the ground up. This project includes clear instructions and step-by-step guidance for the user to follow. Additionally, it incorporates illustrative test cases, complete with their respective payloads and assertions, along with valuable recommendations for project improvement.
- Upon attempting to execute the provided code, it is fully functional. However, there may be some updates required, which are thoughtfully detailed by ChatGPT within comments embedded in the code itself, as well as in the final comments of the output.
- It's important to note that there will always be some level of work for the automation engineer to undertake in order to review and tailor the code to meet specific requirements.
- Naturally, in addition to the mentioned tests, there exists a plethora of other test types, such as performance, edge cases, and negative tests, that can be implemented as needed. As you can see in the Security example, ChatGPT is capable of generating nonfunctional test scenarios as well.
- The quality of the prompt has a direct impact on the outcomes produced by the tool, and there is room for enhancement by making it more specific. In this proof of concept, our intention was to assess result quality through the utilization of a straightforward set of instructions.
About Encora
Fast-growing tech companies partner with Encora to outsource product development and drive growth. Contact us to learn more about our software engineering capabilities.
