Json-api: Compliance test suite

[beauby]

The issue

This topic has been brushed a few times already (#1098, #892, #733). The idea is to provide a test suite so that library implementers can 1. prove compliance to the spec and 2. uniformly showcase their abilities.

The solution

I went ahead and created some basic infrastructure: jsonapi-grader, which is mainly a bunch of request specs.

The idea with the validation of server libraries is that each maintainer would set up a service (on a free heroku instance or similar) against which users could run the test suite (and possibly the jsonapi.org maintainers would periodically run it as well). The test scenarii would possibly be documented on the jsonapi.org website.

Making it happen

Sounds interesting? Get involved and help build the actual test suite (suggesting scenarii and/or implementing them).

[beauby]

This could be a helpful reference: https://json-ld.org/test-suite/

[evolve2k]

Since this is the new home for this discussion I'll bring my recommendation here from https://github.com/json-api/json-api/issues/1098

I'm proposing we use 'Author > Books > Reviews' as the basis of an example app.

Lets make it so that all resources in the domain have names that are very easy to grok and can not be in any way misunderstood for code related terms.

[beauby]

@evolve2k Thanks for the input. I agree that we should keep the test scenarii and the tested resources/endpoints simple to reason about.
However, I believe it would be easier for implementers that each scenario tests a specific thing (which is not opposed to having resources/endpoints named as you suggest, but I'd like to avoid having 3 endpoints that have to comply to many different scenarii at the same time).
One possible solution would be to have each test scenario hit namespaced endpoints (e.g. for the first test http://compliance.example.com/test1/books).

[fotinakis]

One thing I think should be in there: a model with a two-word name, like CustomerReviews, so that we can test serializing the type to customer-reviews.

[jaredcnance]

@fotinakis I'm not certain we should be concerned with the casing choice of the Server since, although it is a recommendation, is not required. For example, an implementation _may_ only support camelCased or snake_cased names. Compound resource names introduce, what I would consider to be, an unnecessary complexity. It makes result validation a bit more complicated since now we need to check the keys of the returned document against several different casing options. But, I also see the value in being able to state what casing mode the Server uses by default. If we did use compound names, I believe it should be isolated to a single test, validating ONLY that.

[ethanresnick]

I love this discussion! Don't have the bandwidth to add much, but I would like to point to one of @tkellen's old projects — https://github.com/endpoints/fantasy-database — which could offer a schema/test data to work from easily.

[remmeier]

IMDB is also a popular example dataset.

on the application side I could offer crnk.io as guinea pig to setup and pass those tests. It has both a full server and implementation.

[beauby]

Thanks for the inputs – a dataset will definitely come in handy.
I think the next step should be establishing together a list of specs (sorted by category) that we want for the first version. I'll open a PR so that people can suggest new tests/categories.

[beauby]

Update: I built part of a test suite. Anybody willing to give a hand?

[jaredcnance]

@beauby thanks for leading this. i'll be taking a deeper dive into it this weekend.

[beauby]

@jaredcnance (and everyone, really): I updated the README with a few explanations, and added a (partial) example compliance implementation for jsonapi-rails.

[tkellen]

Partial test suite here as well: https://github.com/endpoints/endpoints/tree/master/test/integration