Redefine API Testing with the OpenAPI workflows
An ever-changing game — like football
Football, similarly to many other sports, has been continuously changing. The beautiful game had to adapt to changes in the players’ abilities, the clubs’ ambitions, and the demands and expectations of its supporters and lovers.
But there have been, at times, changes that have drastically transformed the sport. Not many will remember the times before the back-pass rule: the goalkeeper was allowed to handle the ball even when this was a pass from one of the teammates. That led to time-wasting and slowing down the rhythm, significantly affecting the speed of the game.
Nothing has been the same since that day: goalkeepers learned to master the ball with their feet, and the game has become much faster while matches are more exciting.
And the best part of it is that we all have won, no matter which team you supported or which player you would dream to be.
Can the API ecosystem undergo a similar transformation? Will API workflows create opportunities that benefit everyone — be it applications, tools, or users?
In this blog…
…read about the OpenAPI Workflow specification, its use cases, and the opportunities it can bring to API Testing.
API Testing
APIs are the backbone of any modern platform. They enable seamless communication between users and applications, secure data exchange, and connect systems and partners. But, as the communication patterns become more complex and the use cases more advanced, API testing is more crucial than ever.
API testing is not the newest topic. We have all been doing this for a while, engineering the approach as we go, trying tools, and learning best practices. However, the focus has always been on ensuring an endpoint respects its contract. While this approach is valuable for validating the correctness of individual API calls, it falls short of validating the holistic behaviour of the API.
Contract Testing
Take Contract Testing. It’s a well-established discipline in the API testing arena. The API team provides a contract (such as an OpenAPI definition), and both the consumer and producer mutually agree upon it. No exception, not even a small one.
The way this works is pretty simple. The OpenAPI standard lets you define examples that provide a request-response interaction: when providing a specific JSON payload for an operation, it would return a given response. The more examples, the more interactions you can test.
For example: the Adyen Checkout OpenAPI file gives you an exhaustive list of payment scenarios. Developers look at those examples to understand the type and format of the data being exchanged, while tools use these expectations to verify the behaviour (contract) of the API.
Several tools are doing a great job in this domain. Some of them also work without OpenAPI, allowing us to define the expectations differently through some meta language or custom code. They are all good at what they do. Unfortunately, that is testing a single API call at the time.
Enter OpenAPI workflows
If you haven’t heard of it yet, watch now the 𝗧𝗵𝗲 𝗔𝗣𝗜 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀 𝗦𝗽𝗲𝗰𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻: 𝗧𝗵𝗲 𝗺𝗶𝘀𝘀𝗶𝗻𝗴 𝗽𝗶𝗲𝗰𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗔𝗣𝗜 𝗽𝘂𝘇𝘇𝗹𝗲 talk https://www.youtube.com/watch?v=NdlhxIVlSEM&t=2s given by Frank Kilcommins (SmartBear) at API Days Paris 2023.
In a nutshell, the goal of the OpenAPI workflow specification is to describe, in a machine-readable format, a chain of API requests. It is no longer about sending some input and receiving an output. It’s about representing a sequence of API calls that collectively fulfil a business process. Each API call is one of those steps towards delivering the intended business value.
OpenAPI workflow files can be written in JSON or YAML, and include several elements:
workflowSpec: the version of the workflow specification the file implements.info: the usual (essential) metadata like title, version, and description.
sourceDescriptions: the sources that will be used in the workflow specification file. Think of the OpenAPI document that describes the operations invoked by the workflow.workflows: one or more workflows. Each workflow specifies the steps (API calls) and their input, output, and expectations.
components: re-usable parameters and objects
Use cases
Some use cases are immediately visible, and surely new ones will emerge as the adoption starts.
Start with the visual representation of API-driven processes: this makes clear to consumers how they will interact with multiple services. That is a significant advancement in developer experience, as developers no longer need to figure out (reading the docs or the OpenAPI endpoint specs) the order, data format, and expectation of each step.
I look forward to seeing how the standard will help AI agents better understand services that can be plugged into a conversation (e.g., “Hey chatbot, find me the best deals! “). AI agents are quickly becoming the largest consumer of APIs; chatbots engage users, perform actions on their behalf, or retrieve data during the conversation. The OpenAPI workflows can play a significant role here, providing the semantics necessary to use the APIs.
Finally, the API testing use case is where the OpenAPI workflow can bring real innovation. The API users should be able to define the sequence of API calls (mimic the way they are going to use them) and make those flows testable, repeatable and automated.
Consumer-Driven API Testing
Let’s consider an e-commerce application with functionalities such as user authentication, product browsing, placing an order, and checkout. Instead of testing each API endpoint in isolation and verifying its contract, developers can define an OpenAPI workflow encompassing the entire shopping process. This holistic approach allows for identifying integration issues, data inconsistencies, and side effects that may arise only when multiple APIs are involved.
OpenAPI workflows will facilitate consumer-driven testing, where the consumers can define machine-readable test scenarios that have all the information: operations, order, data format, and expectations.
As a developer consuming multiple APIs, I want to be in the position to script the API flows that I support. I want to test those in an automated fashion. I need to make sure they function as expected from the end-user perspective. And I want to run scenarios that are supposed to fail because of the input, the order of the steps, or other factors.
I compare (in a way) the OpenAPI workflows to a test plan that can assert different scenarios, starting from the happy flows as well as exploring the potential errors that might arise.
Assertion Verification
In the context of API testing, how the OpenAPI workflow specification supports data-driven verification is crucial. This is achieved by letting developers specify the validation criteria within each workflow step.
The SuccessCriteria element is a list of assertions that establish whether the step has succeeded. Each assertion is defined as a Runtime Expression, a condition computed using the information available as the step is executed.
In the example above, the step must:
- return a response with HTTP Status Code 200
- find a specific reference in the response body
- match a second reference with the initial data inputs defined in the workflow.
Let’s go back to the e-commerce application mentioned earlier. Developers will be able to define assertions to validate the correctness of product prices, the details of the order being placed, and the payment outcome. By embedding such assertions into the OpenAPI workflow, developers can automatically detect discrepancies from the expected behaviour and pinpoint exactly where the issues occur within the user journey.
Conclusion
In summary, the advent of the OpenAPI workflows can fundamentally evolve API testing practices. By moving beyond testing individual endpoints to embracing a scenario-driven approach, developers can verify their APIs against real-world scenarios. Being in a machine-readable format the OpenAPI workflow can become effective test plans that are automated to ensure continuous verification.
As software systems continue to grow in complexity, orchestrating multiple APIs and re-using services from different platforms, the OpenAPI workflows can create the framework to validate the correctness and effectiveness of these interactions.
I am looking forward to seeing if and how tools, vendors, and the whole API ecosystem will understand the massive opportunity and adopt a new approach to API testing. The OpenAPI workflow specification is here, and the first step is taken. Let’s see what unfolds next.
