Swiftpack.co - Package - mattpolzin/jsonapi-openapi-test-server

JSON:API/OpenAPI Test Server



The test server requires a Postgres database and Redis instace.


The test server and commandline tool will execute tests against an OpenAPI document. Out of box, you get errors for unprocessable OpenAPI documentation, warnings for certain unhandled types of APIs, and a test that every OpenAPI Example parses under the corresponding request/response body schema.

By default the server will assume that all request and response bodies are JSON:API compliant. The server will warn you and fall back to a non-JSON:API (but still JSON) request/response parsing mode if it fails to generate a test based on a JSON:API compliant schema. You can also opt out of attempting to interpret any particular OpenAPI Media Item as JSON:API with the x-not-json-api: true specification extension. This gets added to the OpenAPI document inside the Media Item but outside the schema.

For example,

    "content": {
        "application/json": {
            "x-not-json-api": true,
            "schema": {
                "type": "object"

You can additionally create tests that make API calls and verify that the actual responses from your server are parseable under the corresponding response schema. You do this with the x-tests Specification Extension on the OpenAPI Media Type Object within a Response Object (e.g. responses/'200'/content/'application/json'/x-tests). x-tests has the following structure:

    "test_name": {
        "test_host": "url",
        "skip_example": false,
        "ignore_missing_parameter_warnings": false,
        "parameters": {
            "path_param_name": "value",
            "header_param_name": "value"
        "query_parameters": [
                "name": "param_name",
                "value": "param_value"


  • test_host: optional, if omitted then default server for API will be used.
  • skip_example: optional, defaults to false.
  • ignore_missing_parameter_warnings: optional, defaults to false.
  • parameters values: Must be strings, even if the parameter type is Int or other.

The Commandline Tool

The command line tool's usage can be printed with --help and it is as follows:

VERVIEW: Build and run tests based on an OpenAPI Document.

USAGE: APITest [--dump-files <directory path>] [--fail-hard] [--ignore-warnings] [--openapi-file <file path>] [--override-server <url>] [--parser <parser>]

  --dump-files <directory path>
                          Dump produced test files in a zipped file at the
                          specified location. 
        Tip: A good location to dump files is "./out". For the Dockerized tool
        this will be `/app/out` and when running the tool natively on your
        machine this will be the `out` folder relative to the current working

        Not using this argument will result in test files being deleted after
        execution of the tests.
  -f, --fail-hard         Produce a non-zero exit code if any tests fail. 
  --ignore-warnings       Do not print warnings in the output. 
  --openapi-file <file path>
                          Specify a filename from the local filesystem from
                          which to read OpenAPI documentation. 
        Alternatively, set the `API_TEST_IN_FILE` environment variable.

        Either the environment variable or this argument must be used to
        indicate the OpenAPI file from which the tests should be generated.
  --override-server <url> Override the server definition(s) in the OpenAPI
                          document for the purposes of this test run. 
        This argument allows you to make API requests against a different
        server than the input OpenAPI documentation specifies for this test

        Not using this argument will result in the API server options from the
        OpenAPI documentation being used.
  -p, --parser <parser>   Choose between the "stable" parser and a "fast"
                          parser that is less battle-tested. (default: stable)
        This argument is currently only applicable to JSON parsing. When
        decoding a YAML file, the argument is ignored as there is only
        currently one YAML parser to choose from.

        Not using this argument will result in using the default stable parser.
  -h, --help              Show help information.

Against a URL

You can point the test tool at a URL serving up OpenAPI documentation. The URL can either require HTTP Basic Authentication or no authentication.

The unauthenticated version only requires the API_TEST_IN_URL environment variable.

docker run --rm --entrypoint ./APITest --env 'API_TEST_IN_URL=https://website.com/api/documentation' mattpolzin2/api-test-server

The authenticated version additionally requires the API_TEST_USERNAME and API_TEST_PASSWORD environment variables.

docker run --rm --entrypoint ./APITest --env 'API_TEST_IN_URL=https://website.com/api/documentation' --env 'API_TEST_USERNAME=username' --env 'API_TEST_PASSWORD=password' mattpolzin2/api-test-server ./APITest

Against a local file

You can point the test tool at a local file if you mount that file into the docker container and specify the mount destination with the API_TEST_IN_FILE environment variable or the --openapi-file option for the test command.

# command option
docker run --rm --entrypoint ./APITest -v '/full/path/to/openapi.json:/api/openapi.json' mattpolzin2/api-test-server --openapi-file /api/openapi.json

# ENV var
docker run --rm --entrypoint ./APITest --env 'API_TEST_IN_FILE=/api/openapi.json' -v '/full/path/to/openapi.json:/api/openapi.json' mattpolzin2/api-test-server

Note that you cannot use relative paths with bind mounts but if, for example, your openapi.json file is in the current working directory then you could invoke as:

docker run --rm --entrypoint ./APITest --env 'API_TEST_IN_FILE=/api/openapi.json' -v "$(pwd)/openapi.json:/api/openapi.json" mattpolzin2/api-test-server

API Host Override

You can specify an override test server URL if you want to make API test requests against a different URL than is specified bu the OpenAPI documentation. You use the test command's --override-server option for this.

docker run --rm --entrypoint ./APITest -v '/full/path/to/openapi.json:/api/openapi.json' mattpolzin2/api-test-server --openapi-file /api/openapi.json --override-server https://test.server.com

Dumping test files

The test tool works by generating Swift code to parse examples and test responses. These test files include JSON:API models that could be used as a basis for client implementations. You can dump the test files with the --dump-files argument to the ./APITest test command. You must also mount the output directory (or don't remove the container and then docker cp later) so you can access the generated file from outside of the container.

docker run --rm --env 'API_TEST_IN_URL=https://website.com/api/documentation' -v "$(pwd)/out:/app/out" mattpolzin2/api-test-server ./APITest --dump-files /app/out

You will find the dumped files at /app/out/api_test_files.zip. TIP: You can also find the raw text logs from a test run at /app/out/api_test.log.

The Test Server

You can run an API Test server that accepts requests to run tests at HTTP endpoints. This requires the same input file or URL environment variables explained in the above section but you also must provide a Postgres database for the server to use as its persistence layer. You specify this database using a Postgres URL in the API_TEST_DATABASE_URL environment variable. A Redis instance is required to queue up the test runs. You specify the Redis URL in the API_TEST_REDIS_URL environment variable.

First you need to run the migrator against your Postgres database.

docker run --env 'API_TEST_IN_URL=https://website.com/api/documentation' --env 'API_TEST_DATABASE_URL=postgres://user:password@host:port/databasename' --env 'API_TEST_REDIS_URL=redis://host:port' -p '8080:80' mattpolzin2/api-test-server migrate --yes

Then you can start the server.

docker run --env 'API_TEST_IN_URL=https://website.com/api/documentation' --env 'API_TEST_DATABASE_URL=postgres://user:password@host:port/databasename' --env 'API_TEST_REDIS_URL=redis://host:port' -p '8080:80' mattpolzin2/api-test-server

Jobs Queue

Testing is run in a jobs queue. That queue can be run in the same process as the API server if you specify API_TEST_IN_PROCESS_QUEUES=true as an environment variable but the recommendation is to run the jobs service as its own process.

You start the Jobs Queue using the same docker image as the serve but you specify the queues command.

docker run --env 'API_TEST_IN_URL=https://website.com/api/documentation' --env 'API_TEST_DATABASE_URL=postgres://user:password@host:port/databasename' --env 'API_TEST_REDIS_URL=redis://host:port' mattpolzin2/api-test-server queues

NOTE We must explicitly expose the port to the host device. In this example, http://localhost:8080 will point to the server which is listening on port 80 in the container.

Visit the /docs API endpoint to see what endpoints the server provides.


Note that Vapor 4 (and therefore this server) requires Swift 5.2.

Running and Testing

As of this writing, you need to run swift package generate-xcodeproj and then open that project in Xcode. Using Xcode's built-in Swift Package Manager support is currently broken for libraries like swift-syntax that require dynamic libraries from the Swift toolchain. swift build, swift test, etc. from the command line will work fine, though.

Generating API Documentation

To generate API documentation, run the GenAPIDocumenation target and save the output to Public/openapi.yml. This file is not committed to the repository but it will be picked up by the ReDoc UI served at the /docs API endpoint.

Documentation is generated as part of the Docker image build so you do not need to perform it as a separate step if you are building the Docker image anyway.

Building Docker Image

From the root folder of the repository, run

docker build -t api-test-server:latest .

Once done, you will have the api-test-server:latest Docker image.


Stars: 3

Used By

Total: 0


Error improvements - 2020-10-02 05:21:58

  • Fix bug when parameter definitions are missing.
  • make validation optional but make script exit non-zero if validations fail with fail-hard enabled.

Update to Open APIKit v2 & JSONAPI v5 - 2020-09-28 03:03:52

Update to OpenAPIKit v2 and introduce validation errors to testing output.

You now get parsing errors & warnings, validation errors, and test execution errors as part of a test run.

Updating to JSONAPI v5 comes with much clearer error reporting when JSON:API includes fail to parse.

Fix API Test Properties lookup and creation - 2020-07-26 20:15:28

Add parser option to API Test Properties - 2020-07-26 01:22:38

Expose a new property for tests that chooses between a stable and fast parser. The stable option is the one that was historically available.

⚠️ Breaking Change ⚠️ This is a breaking change for the API because the api_test_properties model has gained a new required attribute.

Support non-GET endpoints - 2020-07-23 01:13:37

Adds support (finally) for non-GET http methods. Also fixes a bug where disk caching of test responses could mess up test results. now ephemeral caching is used.

Support explicitly not-JSON:API endpoints - 2020-07-20 05:11:08

Support the x-not-json-api specification extension on OpenAPI Media Item that makes an endpoint not even warn you if it is not JSON:API compliant because you've stated it is not expected to be.

CLI Improvements - 2020-07-20 02:06:36

  • swap out Vapor command for ArgumentParser library.
  • Also clean up and document command arguments a bit better.
  • Don't bother formatting code that is not being dumped.
  • Offer a good interface for switching between fast and stable JSON parsing.
  • Allow the user to specify the location to dump files if desired.

⚠️ Breaking Changes ⚠️ The --dump-files CLI argument used to be a flag but now it requires the location to dump files as its value. You can specify "./out" to get the same behavior as the flag used to trigger.

Improved messages for network failures - 2020-07-18 08:28:37

When an API Test fails due to a network error, the error is printed out much more clearly now and the test case exits early instead of hitting other errors that are really just because the network failure prevented anything useful from happening in the test.

Message Tweaks - 2020-07-18 01:28:46

A few message tweaks including noting the number of warnings/errors at the end of test building prior to running tests.

Non-JSON:API body parsing - 2020-07-12 20:43:30

This release adds fallback parsing of JSON:API request or response payloads as "generic JSON structures."

Concurrency Wins - 2020-07-11 23:29:01

Fixes https://github.com/mattpolzin/jsonapi-openapi-test-server/issues/7.

Makes concurrent test handling much better.

  • Handle tests as part of a jobs queue.
  • Support jobs queue in-process, but recommend as its own service.
  • Add docker-compose file including queues service setup.
  • Don't hold onto and re-use the same Database for the duration of the long-lived testing process; instead recreate a database once per status update as if handling new requests each time.
  • Put blocking tasks in test running process on packthread threads using the NIO thread pool.

- 2020-07-04 03:10:55

  • Log info messages.
  • Improve warnings for JSON:API resource swift gen failures.
  • Add test names to output of context for tests when available.
  • Fix numerous edge cases around API interpretation by using new resolved (canonical) types from OpenAPIKit.

Fix bugs with source and properties creation - 2020-06-10 03:27:40

Fix bugs where source and property creation resulted in duplicate database records even though there's no reason to have dupes. Fix bug with database field for URL being a TEXT whereas Fluent expected a JSON field.

Add options to command line tool. - 2020-06-07 23:26:09

Add --openapi-file and --override-server command line options to APITest test command. These allow you to specify the input file (alternative to API_TEST_IN_FILE environment variable) and override the server specified by the OpenAPI documentation for test requests.

Update for JSONAPI v4 release. - 2020-06-01 07:10:06

Error logging bug fix - 2020-05-27 06:59:56

Was not logging request errors.

Breaking Changes to API models - 2020-05-27 02:55:11

Add --ignore-warnings flag - 2020-05-06 01:45:44

Add --ignore-warnings flag to APITest test command.

Add --fail-hard flag to `APITest test` command. - 2020-05-04 02:55:14

When running the test service as a terminal utility instead of a server, you now have an additional command line flag --fail-hard which tells the utility to fail with exit code 1 (when there is at least 1 failing test). Otherwise, the script will exit with code 0 even when there are failing tests (which is what it always has done in past versions).

Adds Test Properties - 2020-05-03 03:37:09

Adds a new resource type: test properties.

⚠️ Breaking Changes ⚠️ Test Properties were added to Test Descriptors and now Test Sources are owned by Test Properties instead of directly being referenced by Test Descriptors. It was decided to simply break SQL migrations given the early stage of the project.

Updates to some warning messages. - 2020-04-27 06:20:16

Improve test result context output. - 2020-04-27 03:03:49

Test results now report the context (API path, endpoint, http status) in a much more human friendly way thats they used to.

Fix bug with positional parameter type for request tests. - 2020-04-26 08:06:52

Minor bug fixes and added ignore missing parameter warning option. - 2020-04-26 03:37:01

New "raw logs" route and bug fixes - 2020-04-25 18:01:27

  • Add /api_tests/{id}/logs route to retrieve raw (XCTest) logs from a particular test run.
  • Fix a bug preventing status code checks on response tests.

Add test messages to watch mode - 2020-04-23 02:05:37

Update OpenAPIKit version - 2020-04-22 07:29:46

Add Websocket support for watching test results. - 2020-04-19 22:36:12

The ws route at /watch will push JSON:API api_test_descriptor single-resource documents for each new or modified test descriptor. These updates occur for every test state: pending, building, running, passed, failed.

Update Vapor, Update OpenAPIKit, change docker name - 2020-03-09 05:53:52

⚠️ Breaking Changes ⚠️

  • Updated Vapor 4. Now requires Swift 5.2
  • Updated OpenAPIKit
  • Moved to mattpolzin2/api-test-server on Docker Hub.

Archived test files and multiple runs simultaneously - 2019-12-05 16:53:27

Add support for dumping test files in a zipped format from the APITest executable (APITest test --dump-files).

Add support for retrieving archived test files from server with new api_test/{id}/files endpoint.

Add support for running more than one test simultaneously (or rather coincidentally) to the server. Previously you could request tests in rapid fire, but test files were all generated in the same folder so this was not really supported. Even now, the number of concurrently running tests is 1, but async operations from preparing or tearing down tests will not step on the toes of other test runs.