Testing OpenAPI specification compliance in Quarkus with Prism

Quarkus is a good framework choice when developing RESTful APIs in an API-first manner by using the OpenAPI generator to generate models and API interfaces. That way, larger parts of compliance with the API specification are already handled by the compiler. However, not every inconsistency can be detected this way. In this post, I demonstrate how to integrate Stoplight’s Prism proxy into the test infrastructure of a Quarkus Kotlin project for validating OpenAPI specification compliance automatically as part of the API tests.

Motivation

In API-first development with Quarkus and Kotlin I have shown a basic setup for Quarkus to support API-first development. As a short recap, API-first development is an approach of developing (RESTful) APIs, where a formal specification of the intended API (changes) is created before implementing the API provider or consumers. That way, we can make use of the specification for code generation, parallel development, and for verification purposes.

A common issue seen when using APIs based on their documentation is that the actual API implementation differs from what is documented. The setup shown before prevents larger parts of this problem by leveraging code generation through the OpenAPI Generator. Moreover, the compiler catches a few error cases when generated interfaces are not implemented properly. However, not every aspect of API compliance is validated this way and we would still be able to provide an implementation that deviates from the specification. Here’s a short example demonstrating one of the more obvious ways we can still deviate:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# ...
paths:
  /pets:
    get:
      summary: List all pets
      operationId: listPets
      responses:
        '200':
          description: An array of pets
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Pets"
# ...
components:
  schemas:
    PetId:
      type: integer
      format: int64
    Pet:
      type: object
      required:
        - id
        - name
      properties:
        id:
          $ref: "#/components/schemas/PetId"
        name:
          type: string
    Pets:
      type: array
      items:
        $ref: "#/components/schemas/Pet"

The GET /pets should clearly return an array of Pet objects. However, we could easily (and hopefully accidentally) implement it this way without the Kotlin compiler detecting the problem:

override suspend fun listPets(): Response {
    return Response.ok(listOf("not", "valid")).build()
}

Being able to do this is definitely not ideal and we need to improve the test harness of the API implementation to prevent such deviations from being possible.

Validating OpenAPI compliance with Prism

Fortunately, tools are available to check actual API requests and responses for compliance with the specification. Probably the best option to use right now is Stoplight’s Prism.In addition to being a mock server, which would be useful for implementing a consumer for a yet to be implemented API, Prism can also act as a validation proxy. In this mode, Prism compares requests and responses to the OpenAPI specification and flags invalid requests or responses.

Let’s explore for a moment how that works. First, we spin up a stub implementation providing an invalid response. The quickest way I could come up with is by using the Python http.server module:

from http.server import BaseHTTPRequestHandler, HTTPServer

class BrokenServer(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "application/json")
        self.end_headers()
        self.wfile.write(bytes('["not", "valid"]', "utf-8"))

if __name__ == "__main__":
    with HTTPServer(("localhost", 9000), BrokenServer) as server:
        server.serve_forever()

Once this script is launched, we can verify that it provides a reply for GET /pets:

$ curl -4 -v http://localhost:9000/pets
* Host localhost:9000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying 127.0.0.1:9000...
* Connected to localhost (127.0.0.1) port 9000
* using HTTP/1.x
> GET /pets HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/8.11.0
> Accept: */*
>
* Request completely sent off
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BaseHTTP/0.6 Python/3.12.7
< Date: Thu, 28 Nov 2024 21:59:32 GMT
< Content-type: application/json
<
* shutting down connection #0
["not", "valid"]

We can now launch Prism with the known OpenAPI specification and our broken implementation as the upstream server:

❯ ./prism-cli-linux proxy openapi.yaml http://127.0.0.1:9000
[11:03:46 PM] › [CLI] …  awaiting  Starting Prism…
[11:03:47 PM] › [CLI] ℹ  info      GET        http://127.0.0.1:4010/pets
[11:03:47 PM] › [CLI] ▶  start     Prism is listening on http://127.0.0.1:4010

When we repeat the curl request against the proxy, we receive the following:

$ curl -4 -v http://localhost:4010/pets
* Host localhost:4010 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying 127.0.0.1:4010...
* Connected to localhost (127.0.0.1) port 4010
* using HTTP/1.x
> GET /pets HTTP/1.1
> Host: localhost:4010
> User-Agent: curl/8.11.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: *
< Access-Control-Allow-Credentials: true
< Access-Control-Expose-Headers: *
< sl-violations: [{"location":["response","body","0"],"severity":"Error","code":"type","message":"Response body property 0 must be object"},{"location":["response","body","1"],"severity":"Error","code":"type","message":"Response body property 1 must be object"}]
< server: BaseHTTP/0.6 Python/3.12.7
< date: Thu, 28 Nov 2024 22:03:48 GMT
< content-type: application/json
< Content-Length: 15
< Connection: keep-alive
< Keep-Alive: timeout=5
<
* Connection #0 to host localhost left intact
["not","valid"]

Prism has added an sl-violations header with information on how the response deviates from the API specification. For the sake of completeness, this header is absent in case of valid responses:

< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: *
< Access-Control-Allow-Credentials: true
< Access-Control-Expose-Headers: *
< server: BaseHTTP/0.6 Python/3.12.7
< date: Thu, 28 Nov 2024 22:07:08 GMT
< content-type: application/json
< Content-Length: 2
< Connection: keep-alive
< Keep-Alive: timeout=5
<
* Connection #0 to host localhost left intact
[]

It’s still somewhat easy to miss the violations header. We can make violations a bit more obvious by using the --errors command line flag. This instructs Prism to respond with an internal server error in case of violations, which is a lot less likely to be missed:

$ curl -4 -v http://localhost:4010/pets
* Host localhost:4010 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying 127.0.0.1:4010...
* Connected to localhost (127.0.0.1) port 4010
* using HTTP/1.x
> GET /pets HTTP/1.1
> Host: localhost:4010
> User-Agent: curl/8.11.0
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 500 Internal Server Error
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: *
< Access-Control-Allow-Credentials: true
< Access-Control-Expose-Headers: *
< sl-violations: [{"location":["response","body","0"],"severity":"Error","code":"type","message":"Response body property 0 must be object"},{"location":["response","body","1"],"severity":"Error","code":"type","message":"Response body property 1 must be object"}]
< content-type: application/problem+json
< Content-Length: 483
< Date: Thu, 28 Nov 2024 22:09:38 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
<
* Connection #0 to host localhost left intact
{
  "type": "https://stoplight.io/prism/errors#VIOLATIONS",
  "title": "Request/Response not valid",
  "status": 500,
  "detail": "Your request/response is not valid and the --errors flag is set, so Prism is generating this error for you.",
  "validation": [
    {
      "location": [
        "response",
        "body",
        "0"
      ],
      "severity": "Error",
      "code": "type",
      "message": "Response body property 0 must be object"
    },
    {
      "location": [
        "response",
        "body",
        "1"
      ],
      "severity": "Error",
      "code": "type",
      "message": "Response body property 1 must be object"
    }
  ]
}

Integrating Prism into Quarkus

In API-first development with Quarkus and Kotlin I have set up a Quarkus + Kotlin example project for API-first development. I will now show how to integrate Prism into the test setup of that project so that we get automatic response validation whenever we test one of the API endpoints. That way, it becomes even less likely that the API implementation is not compliant with the OpenAPI specification.

Implementing a Quarkus test resource

For validating responses with Prism, we need a running instance of Prism that tests can use for requests instead of the actual Quarkus app. The most portable way is to start Prism as a container, which avoids developers from having to provide an installation of Prism on their machine. External services such as the Prism proxy are usually spun up as "test resources" in Quarkus. Therefore, we first need an implementation of a QuarkusTestResourceLifecycleManager, which we call PrismProxyResource. This class spins up a Prism container using testcontainers. The container is configured to proxy requests to the started Quarkus app used by the Quarkus test infrastructure:

class PrismProxyResource : QuarkusTestResourceLifecycleManager {

    override fun start(): MutableMap<String, String> {
        Testcontainers.exposeHostPorts(testPort()) 1
        proxyContainer.start()
        return emptyMap<String, String>().toMutableMap()
    }

    override fun stop() {
        proxyContainer.stop()
    }

    companion object {
        private var proxyContainer: GenericContainer<*> =
            GenericContainer<Nothing>(DockerImageName.parse("stoplight/prism:5")).apply {
                withAccessToHost(true) 2
                withCopyFileToContainer( 3
                    MountableFile.forClasspathResource("META-INF/openapi.yaml"),
                    "/tmp/openapi.yaml"
                )
                withCommand(
                    "proxy",
                    "--errors", 4
                    "-h", "0.0.0.0", 5
                    "/tmp/openapi.yaml", 5
                    "http://host.testcontainers.internal:${testPort()}" 6
                )
                withExposedPorts(4010) 7
            }

        private fun testPort(): Int = ConfigProvider.getConfig().getOptionalValue(
            "quarkus.http.test-port",
            Int::class.java
        ).orElse(8081) // The default as documented by Quarkus

        fun proxyPort(): Int = proxyContainer.firstMappedPort
    }
}

Prism needs access to the Quarkus app used during tests execution. Quarkus opens an app on a well-known port when running tests. The Prism container needs access to this app. Therefore, testcontainers is configured to expose this well-known Quarkus test port from the host system (where the test app is running) to containers.
The Prism container also needs to declare that it needs host access for being able to reach the Quarkus test app.
Prism has to know the OpenAPI specification to validate against. Therefore, we copy it to the container.
We instruct Prism to generate errors, which will surface automatically in most test cases. That way, we cannot forget to check the sl-violations header in tests.
Prism must listen on all interfaces so that requests from outside the container – ie., the host system executing the tests – can be received.
The upstream for Prism is the test Quarkus app running on the host system. The URL constructed here uses the platform-agnostic host exposing feature of testcontainers.
Finally, we also need to expose the Prism port so that it can be reach from the host system running the tests and thereby performing API requests.

As this new resource uses testcontainers, the dependency has to be declared in pom.xml:

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>testcontainers</artifactId>
    <scope>test</scope>
</dependency>

Redirecting test requests through the test resource

With the test resource in place, we can adapt the existing API test to perform requests through Prism:

@QuarkusTest
@QuarkusTestResource(PrismProxyResource::class) 1
class PetsApiTest {

    @Inject
    lateinit var petRepository: PetRepository

    @BeforeEach
    fun clearRepo() {
        petRepository.clear()
    }

    @BeforeEach
    fun usePrism() {
        RestAssured.port = PrismProxyResource.proxyPort() 2
        RestAssured.enableLoggingOfRequestAndResponseIfValidationFails() 3
    }

    @Nested
    inner class CreatePets {

        @Test
        fun `can create new pets`() {
            val pet = Pet(42, "new name", "some tag")
            Given {
                body(pet)
                contentType(ContentType.JSON)
            } When {
                post("/pets")
            } Then {
                statusCode(StatusCode.CREATED)
            }
            assertTrue(petRepository.listPets().contains(pet))
        }

    }

    // ...

At least one test class in Quarkus needs to declare PrismProxyResource as a QuarkusTestResource so that the proxy container is actually started before tests launch.
We need to convince REST-assured to use the forwarded proxy port instead of the test port of the Quarkus app. Unfortunately, I could not find a way to prevent this manual configuration step. The Quarkus test infrastructure resets the REST-assured port before each test. So we need to overwrite this manually before each test case is run.
When REST-assured expectations fail, its usually a good idea to see the data that caused the failure. Could/should be set up once globally for all tests.

We now automatically validate all requests and responses against the OpenAPI specification.

Detecting and fixing a bug in the example project

Running the existing tests in the example project that previously ran green through Prism immediately brought up a violation that was unnoticed before. Two tests, one for listPets and one for showPetById, failed with the following error reported by Prism:

{
    "type": "https://stoplight.io/prism/errors#VIOLATIONS",
    "title": "Request/Response not valid",
    "status": 500,
    "detail": "Your request/response is not valid and the --errors flag is set, so Prism is generating this error for you.",
    "validation": [
        {
            "location": [
                "response",
                "body",
                "0",
                "tag"
            ],
            "severity": "Error",
            "code": "type",
            "message": "Response body property 0.tag must be string"
        }
    ]
}

Pets can have tags according to our specification:

Pet:
  type: object
  required:
    - id
    - name
  properties:
    id:
      $ref: "#/components/schemas/PetId"
    name:
      type: string
    tag:
      type: string

However, tag is not declared to be nullable but our serialization adds a null value when the tag is not set in our data class for Pet, which is generated as:

data class Pet (
    @JsonProperty("id")
    val id: kotlin.Long,

    @JsonProperty("name")
    val name: kotlin.String,

    @JsonProperty("tag")
    val tag: kotlin.String? = null
)

Probably, the kotlin-server generator of the OpenAPI Generator should have added an explicit @JsonInclude annotation to each field to prevent this issues in the first place. I could change the generator templates to add these annotations. But then I would be deviating from upstream and wouldn’t get updates for the affected templates anymore with version bumps of OpenAPI Generator. Therefore, for now, an easier option to fix this problem is to configure the Jackson ObjectMapper used by Quarkus to always exclude null values:

@Singleton
class ObjectMapperCustomizer: ObjectMapperCustomizer {
    override fun customize(objectMapper: ObjectMapper) {
        objectMapper.setSerializationInclusion(JsonInclude.Include.NON_NULL)
    }
}

After adding this class to the project, tests execute successfully and we know that all parts of the REST API covered by tests are complying with the OpenAPI specification.

Using the test resource in integration tests

For now, we have covered typical @QuarkusTests, which are often used for tests that are at least close to the unit test level. However, the same setup can also be used for integration tests so that we also gain confidence regarding OpenAPI compliance for the integrated product. The following example shows how to use the new test resource also for integration tests. Please note, in Quarkus integration tests, tests do not have access to the dependency injection container. Therefore, we cannot directly modify the contents of the PetRepository. Instead, we need to use the REST API to test the business logic, which is a good thing to do in integration tests in any case. With these types of tests we want to gain confidence that the integrated software works as intended and support the business cases.

@QuarkusIntegrationTest
@QuarkusTestResource(PrismProxyResource::class)
class PetsApiIT {

    @BeforeEach
    fun beforeEach() { 1
        RestAssured.port = PrismProxyResource.proxyPort()
        RestAssured.enableLoggingOfRequestAndResponseIfValidationFails()
    }

    @Test
    fun `can list created pets`() {
        val pet = Pet(42, "new name", "some tag")
        Given { 2
            body(pet)
            contentType(ContentType.JSON)
        } When {
            post("/pets")
        } Then {
            statusCode(StatusCode.CREATED)
        }

        val responseBody = When {
            get("/pets")
        } Then {
            statusCode(StatusCode.OK)
            contentType(ContentType.JSON)
        } Extract {
            body().`as`(petListTypeRef)
        }

        assertTrue(responseBody.contains(pet))
    }

    @Test
    fun `can receive created pets`() {
        val pet = Pet(17, "new name")
        Given {
            body(pet)
            contentType(ContentType.JSON)
        } When {
            post("/pets")
        } Then {
            statusCode(StatusCode.CREATED)
        }

        val responseBody = When {
            get("/pets/${pet.id}")
        } Then {
            statusCode(StatusCode.OK)
            contentType(ContentType.JSON)
        } Extract {
            body().`as`(Pet::class.java)
        }

        assertEquals(pet, responseBody)
    }

    companion object {
        private val petListTypeRef = object : TypeRef<List<Pet>>() {}

        @BeforeAll
        @JvmStatic
        fun configureObjectMapper() { 3
            RestAssured.config = RestAssuredConfig.config()
                .objectMapperConfig(ObjectMapperConfig().jackson2ObjectMapperFactory { cls, charset ->
                    ObjectMapper().apply {
                        findAndRegisterModules()
                        ObjectMapperCustomizer().customize(this)
                    }
                })
        }
    }

}

Similar to the @QuarkusTest case, we unfortunately need to configure REST-assured before each test case to pass requests through Prism.
Test cases now use the API to set up the pre-conditions instead of manipulating the internal repository.
The integration test code has no access to the ObjectMapper instance configured inside the Quarkus app. Therefore, we need to configure it similar to how it would be configured inside the app so that API requests issued by the test code are valid. For this purpose, we simply reused the ObjectMapperCustomizer from the app code so that we effectively get the same configuration.

With this test in place, we can now launch integration test for the first time:

$ ./mvnw verify -DskipITs=false

...

[INFO] --- failsafe:3.2.5:integration-test (default) @ quarkus-api-first-example ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on -Dswing.aatext=true -Dswing.defaultlaf=com.sun.java.swing.plaf.gtk.GTKLookAndFeel
[INFO] Running de.semipol.PetsApiIT

...

[11:36:59 AM] › [CLI] …  awaiting  Starting Prism…
2024-12-04 12:37:01,201 INFO  [tc.stoplight/prism:5] (pool-3-thread-1) Container stoplight/prism:5 started in PT4.400961939S
[11:37:01 AM] › [CLI] ℹ  info      GET        http://0.0.0.0:4010/pets
[11:37:01 AM] › [CLI] ℹ  info      POST       http://0.0.0.0:4010/pets
[11:37:01 AM] › [CLI] ℹ  info      GET        http://0.0.0.0:4010/pets/-8863377405594787
[11:37:01 AM] › [CLI] ▶  start     Prism is listening on http://0.0.0.0:4010
Executing "/usr/lib/jvm/java-23-openjdk/bin/java -Dquarkus.http.port=8081 -Dquarkus.http.ssl-port=8444 -Dtest.url=http://localhost:8081 -Dquarkus.log.file.path=/home/languitar/src/quarkus-api-first-example/target/quarkus.log -Dquarkus.log.file.enable=true -Dquarkus.log.category."io.quarkus".level=INFO -jar /home/languitar/src/quarkus-api-first-example/target/quarkus-app/quarkus-run.jar"
Picked up _JAVA_OPTIONS: -Dawt.useSystemAAFontSettings=on -Dswing.aatext=true -Dswing.defaultlaf=com.sun.java.swing.plaf.gtk.GTKLookAndFeel
__  ____  __  _____   ___  __ ____  ______
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2024-12-04 12:37:02,899 INFO  [io.quarkus] (main) quarkus-api-first-example 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.16.4) started in 0.917s. Listening on: http://0.0.0.0:8081
2024-12-04 12:37:02,906 INFO  [io.quarkus] (main) Profile prod activated.
2024-12-04 12:37:02,907 INFO  [io.quarkus] (main) Installed features: [cdi, kotlin, rest, rest-jackson, smallrye-context-propagation, smallrye-openapi, vertx]
[11:37:04 AM] › [HTTP SERVER] post /pets ℹ  info      Request received
[11:37:04 AM] ›     [PROXY] ℹ  info      > Forwarding "post" request to http://host.testcontainers.internal:8081/pets...
[11:37:04 AM] ›     [PROXY] ℹ  info      The upstream call to /pets has returned 201
[11:37:04 AM] ›     [PROXY] ℹ  info      < Received forward response
[11:37:04 AM] › [HTTP SERVER] get /pets ℹ  info      Request received
[11:37:04 AM] ›     [PROXY] ℹ  info      > Forwarding "get" request to http://host.testcontainers.internal:8081/pets...
[11:37:04 AM] ›     [PROXY] ℹ  info      The upstream call to /pets has returned 200
[11:37:04 AM] ›     [PROXY] ℹ  info      < Received forward response
[11:37:04 AM] › [HTTP SERVER] post /pets ℹ  info      Request received
[11:37:04 AM] ›     [PROXY] ℹ  info      > Forwarding "post" request to http://host.testcontainers.internal:8081/pets...
[11:37:04 AM] ›     [PROXY] ℹ  info      The upstream call to /pets has returned 201
[11:37:04 AM] ›     [PROXY] ℹ  info      < Received forward response
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.42 s -- in de.semipol.PetsApiIT
[11:37:05 AM] › [HTTP SERVER] get /pets/17 ℹ  info      Request received
[11:37:05 AM] ›     [PROXY] ℹ  info      > Forwarding "get" request to http://host.testcontainers.internal:8081/pets/17...
[11:37:05 AM] ›     [PROXY] ℹ  info      The upstream call to /pets/17 has returned 200
[11:37:05 AM] ›     [PROXY] ℹ  info      < Received forward response


[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

From the Quarkus documentation on launching containers for integration tests one might think that PrismProxyResource would have to implement DevServicesContext.ContextAware. However, this is not required because the Quarkus app itself never has to reach the proxy container. This exact integration test setup also works when running tests against a built container image of the app.

To check a container image with the integration test, the first thing is to enable building container images by adding the following dependency in pom.xml

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-container-image-jib</artifactId>
</dependency>

Afterwards, we can launch the integration tests against a container image of our application, which is built automatically when executing:

$ ./mvnw verify -Dquarkus.container-image.build=true -DskipITs=false

...

[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

Why does this work without DevServicesContext.ContextAware? Even when running the Quarkus app to test as a container, the HTTP port of the running app is published to the container host on the well-defined test port 8081 (if not configured differently). Therefore, Prism can continue to connect that host port and reaches the Quarkus app, despite now being run in a container.

Trading confidence for test execution time

We now have the option to validate API responses (and requests) in all types of tests available in Quarks. However, launching Prism proxy adds a slight penalty on the execution time of the test suite. Fortunately, we can now make trade-off decisions. In case quick executions of things like unit tests are important for you, checking for compliance to the OpenAPI specification can only be used in integration tests. That way, we trade confidence for test execution time. In contrast, if specification compliance is very important to the project, then compliance should probably also be tested in all tests.

Summary

With the addition of a relatively simple Quarkus test resource and a few lines boilerplate in each API test class, we can gain a lot of confidence in whether our REST API complies with the OpenAPI specification. This was one of the missing puzzle pieces for successfully implementing API-first development with Quarkus as described in the previous blog post. What’s now missing for a complete API-first workflow is to lint the API specification so that best-practices and standards are used and to control breaking changes.

Motivation#

Validating OpenAPI compliance with Prism#

Integrating Prism into Quarkus#

Implementing a Quarkus test resource#

Redirecting test requests through the test resource#

Detecting and fixing a bug in the example project#

Using the test resource in integration tests#

Trading confidence for test execution time#

Summary#

Motivation

Validating OpenAPI compliance with Prism

Integrating Prism into Quarkus

Implementing a Quarkus test resource

Redirecting test requests through the test resource

Detecting and fixing a bug in the example project

Using the test resource in integration tests

Trading confidence for test execution time

Summary