Graphql-dotnet: Discussion: GraphQL Spec Compliance Test

[pranshuchittora]

Edit

##### Spec discussion -> https://github.com/graphql/graphql-spec/issues/698

Hello maintainers
As a part of research around the simplifying graphql spec tests across implementations, I've stumbled upon the specs written here for graphql-dotnet. I'd really appreciate if I can get an idea around how difficult it has been to maintain this test suite, and how often does it go out of parity with the actual graphql spec? Were there cases when misunderstanding the spec led to issues with the library?

Would having a generic set of compatibility tests help around these issues if any?

As schema definition in SDL format is natively supported by the library then it would be easy to adapt generic testing suite across different languages.

[sungam3r]

Could you tell in more detail what you propose to do?

[pranshuchittora]

Actually I am researching for the project proposal given by the GraphQL Foundation for Google Summer of Code 2020 -> Project Idea

The idea is to implement a Compatibility Acceptance Test Suite which can be integrated into the GraphQL implementation library.

[sungam3r]

The goal of this project is to extract all non-JS specific tests in some language-independent data format (e.g. YAML) and work closely with maintainers to help them to adapt this test suite to their needs.

Where is this test suite to integrate here? Does it exist?

[BenjaBobs]

From reading the proposal, the idea is to extract the tests from being written in a programming language, to being pure data (yaml was proposed, but it could be anything I guess).
Each implementation would then only have to write a parser for that data to have access to all the tests (nearly 2000) and could keep up to date on new spec tests simply by updating the data files.

I think this sounds like a great idea, but it requires quite a lot of coordination.

EDIT:
From proposal:

Build a set of compatibility acceptance tests for libraries that implement the GraphQL specification.

Reference implementation of GraphQL (graphql-js) contains almost two thousand test cases and it constantly growing with every new feature we add in GraphQL. Many GraphQL implementations choose to rewrite these tests in their language of choice but keeping them in sync requires a lot of effort from maintainers. The goal of this project is to extract all non-JS specific tests in some language-independent data format (e.g. YAML) and work closely with maintainers to help them to adapt this test suite to their needs.

Expected outcomes: Extract the majority of tests into language-independent format and have a couple of non-JS GraphQL implementations using them for tests.

[pranshuchittora]

The goal of this project is to extract all non-JS specific tests in some language-independent data format (e.g. YAML) and work closely with maintainers to help them to adapt this test suite to their needs.

Where is this test suite to integrate here? Does it exist?

The idea is to build such a test suite during the course of this summer. Speaking of the integration, the idea is to integrate such a generic test suite in all the implementations of GraphQL (graphql-dotnet, graphql-ruby, etc) with the help of maintainers.

[pranshuchittora]

@BenjaBobs To perform such generic testing the only way is to spawn an instance of GraphQL server (graphql-ruby) and test it on various queries & schemas over HTTP. As all the implementations support running over HTTP.

@ maintainers how does that sound in terms of maintainability and community acceptance. Is there any alternative design that you would like to suggest?

[sungam3r]

Now it’s hard to say how much this will fit here. So far, there are only general theories. When some examples appear, then we can discuss.

[pranshuchittora]

I recommend following this thread -> https://github.com/graphql/graphql-spec/issues/698

[sungam3r]

@pranshuchittora I imagine solving a problem this way:

1. Each implementation provides "standard" GraphQL endpoint via docker container to query GraphQL server. Who should provide this container is a separate issue.
2. Test suite calls those endpoints for different implementations with various queries and builds spec compatibility map.
3. No need in _some language-independent data format (e.g. YAML)_. No implementation needs to support an additional set of tests. Test suite can be implemented for only one platform (maybe JS).

[pranshuchittora]

@pranshuchittora I imagine solving a problem this way:

1. Each implementation provides "standard" GraphQL endpoint via docker container to query GraphQL server. Who should provide this container is a separate issue.
2. Test suite calls those endpoints for different implementations with various queries and builds spec compatibility map.
3. No need in _some language-independent data format (e.g. YAML)_. No implementation needs to support an additional set of tests. The test suite can be implemented for only one platform (maybe JS).

1. The idea is to communicate with the GraphQL server via HTTP. And the major roadblock is to send that schema file to the server from the client, because different languages have different ways to execute this.

2. Ya I agree, and it doesn't matter whether the server is running locally or in docker.

3. As per the project idea writing the tests in language-independent form is the proposed idea.
   I completely agree with you, that the client will be in JS so why not tests in JS too. If the community agrees then we can have tests written in JS.

4. Can you suggest a way to get the queries for testing to cover all the edge cases?

[sungam3r]

And the major roadblock is to send that schema file to the server from the client, because different languages have different ways to execute this.

Sorry, I didn’t understand. What differences of what languages are you talking about? The test client sends requests over the HTTP protocol and receives responses. That is all. All implementation details exist only on the server side.

1. Well, this way is to spend a lot of time and just write a bunch of test requests for each section of the specification :)

[pranshuchittora]

I am thinking of a master HTTP server with a couple of endpoints exposed. This server will listen for events on various endpoints from the GraphQL client. Like

• Update the Schema/fakeData (Given schema/fakeData as payload)
• Spawn the GraphQL server with the updated schema
• Restart, kill the GraphQL instance

The client can run in either CLI (for running test in CI) or on a browser (Given master is reachable).

The master server will be written in the same language as the graphql library, so I am not completely sure about the schema file changes with minimal code. This layer will be the integration layer, therefore making it easy to integrate and disintegrate (like a plug and play model).

[sungam3r]

I imagined it easier.
We come up with a special schema for testing purposes. Schema is given as SDL. Requirements for returned data may also be described. Then each language specific repo implements this schema in some "sample" application and provides docker container with this app. Then test suite runs this container for being used by http client. So no custom endpoints, no additional logic like restart, update for language specific repo, only "sample" WebAPI.

[pranshuchittora]

You mean one schema and some fake data, that sounds awesome as it will reduce a lot of complexity.

Does one schema is enough for such testing?

[sungam3r]

You mean one schema and some fake data

Yes.

Does one schema is enough for such testing?

Why not? The task is to make such an interesting schema on which you can run all types of validations.

[pranshuchittora]

Then tests must be run synchronously as mutations can lead to changes in the data. Therefore sequence must be followed because mutating data in bad order and getting unexpected result my fail the test.

[sungam3r]

Obviously. But these are technical details.

[pranshuchittora]

Obviously. But these are technical details.

True but what happens when a mutation fails in that case, all the tests ran after that will fail as well.

[BenjaBobs]

There is the possibility of resetting the database after each mutation test and not making Test B depend upon data created by Test A.
The test data for queries could consist of an initial database state, a query and a query result. The test data for mutations could be an initial database state, a mutation, a mutation result, a query and a query result.

[shobhitchittora]

Also from how I see it, every implementation of the spec can have a app folder, where a sample server can be spawned. The repo / effort which @pranshuchittora is putting in, can have contribution of small utility libraries ( graphql-spec-sdk ) in respective languages ( c#, rb, .. ), which would help in spawning, initializing, restarting, the test runtime in that language.

A separate process ( can be in any one language ), can host all the schemas and data files to be loaded and this part of the graphql-spec-sdk can then verify if the implementations confirm to the test spec.

Hope this flow makes sense to all.

[sungam3r]

@shobhitchittora This is more complicated and requires more effort. graphql-spec-sdk should be written as well for each language. In fact, this is all the same work, simply scattered in several places.

which would help in spawning, initializing, restarting, the test runtime in that language.

These are all test suite tasks. This work is not for implementations.

[BenjaBobs]

As I imagine it,
the process of doing the test would be something like this:

• Start test server
• Load test schema from graphql-spec/test-schema.gql
• Load seed test data into test database from graphql-spec/data-seed.json
• Invoke graphql-spec/bin/graphql-spec.exe -endpoint https://localhost:3000/graphql

And then graphql-spec.exe would handle all the testing, provided the endpoint of the test graphql server.

This would place the responsibility of doing the actual testing on graphql-spec and the responsibility of standing up a graphql compatible endpoint on the graphql framework that is being tested.

[pranshuchittora]

@sungam3r I think @BenjaBobs and I are on the same page. To handle all the events, we must require some way to communicate b/w the client and the server to update schema/data.

The idea you proposed i.e. to run just a graphql instance, in this case there is very less control to change/update the data on the server-side. And what happens when a mutation fails in that case, all the tests ran after that will fail as well.

Can you please state, how your idea deviates from what @BenjaBobs proposed? :confused:

[BenjaBobs]

in this case there is very less control to change/update the data on the server-side

Well that is why instead of comparing the actual data in the database, there should be a graphql query verifying the result.
This is because the infrastructure for doing the graphql query verification is already there, and is implementation agnostic. Doing an actual test on the database would require an extra implementation for each database implementation and data format.

And what happens when a mutation fails in that case, all the tests ran after that will fail as well.

Well, that is why I proposed that the data is reset after each test, and no test is dependent on any other test.
Example:

1) Start test server
2) Load test schema from graphql-spec/test-schema.gql
3) Load seed test data into test database from graphql-spec/data-seed.json
4) Invoke graphql-spec/bin/graphql-spec.exe -graphqlEndpoint https://localhost:3000/graphql -resetEndpoint https://localhost:3000/reset
5) foreach test
5.1 request resetEndpoint to make sure the state is clean
5.2 request graphqlEndpoint with each query/mutation defined in test

And a test could then be something like

// a test consisting of an array of requests and expected results
[
    {
        // first a request, a mutation creating person John Doe
        "request": "mutation {
            createPerson (id: \"2\", name: \"John Doe\") {
                id,
                name
            }
        }",
        // the expected result based on the test schema
        "expectedResult": {
            "id": "2",
            "name": "John Doe",
        }
    },
    {
        // afterwards a query to verify that the person was indeed created
        "request": "query {
            people {
                id,
                name
            }
        }",
        // expect the new person to appear along side existing people
        "expectedResult": [
            {
                "id": "1",
                "name": "John Smith",
            },
            {
                "id": "2",
                "name": "John Doe",
            }
        ]
    }
]

Now I realize that whether John Doe would actually appear in the people query is 100% dependent on what the query is actually semantically supposed to display, I think that using an agreed upon schema with agreed upon semantics one can simply make the test server implement this schema. A side effect of this is also that now every graphql framework that tests against graphql-spec has an update example application that readers can read in order to see how to implement x or y in that framework.

[BenjaBobs]

In case of graphql-dotnet the test server could implement EF Core with an InMemory provider for simplicity, which will serve both as test server, and as a guide for people on how to set up graphql-dotnet with EF Core.

[sungam3r]

Now it’s worth asking about the testing method in https://github.com/graphql/graphql-spec/issues/698