Graphiql: Incremental introspection

[ShishkinDmitriy]

Hello,

We are working on database with 10k+ tables. Each table maps into several graphql types and fields.
We have a problem with introspecting graphiql, which fails on serializing such huge schema at once.
Can you provide incremental introspection loading when each documentation screen loads their data separately?
Maybe it can be startup option --inctremental-introspection?

Best regards,
Dmitriy.

[asiandrummer]

@ShishkinDmitriy Hi there! Incremental introspection is an interesting space - a large schema is problematic for not only the documentation explorer but also the language service features (and also maybe a prettify at one point), basically anything that utilizes the schema instance. It's also a very difficult problem since the entire schema should be validated prior to be used anywhere and is needed to properly power GraphiQL, albeit this doesn't really apply to documentation explorer as it just looks up the information about the schema.

For above reasons I'm not sure what the elegant solution would be. We can definitely try solving just for the documentation explorer, but that'd just be working around a bigger problem we have at hand.

I'm not sure why your serialization is failing though - it might be a signal for a different problem that you have within your server code.

[lunicon]

I use pre-builded schema.json from introspection query for service with 10k+ types.
And init application with it.

fetch('graphql/schema.json', {
  method: 'get',
  headers: { 'Accept': 'application/json'},
  credentials: 'include',
}).then(function (response) {
    return JSON.parse(response.text());
}).then(function (schema) {
  ReactDOM.render(
    React.createElement(GraphiQL, {
      schema: buildClientSchema(schema),
      ...
    }),
    document.getElementById('graphiql')
  );      
});

Also need to add function buildClientSchema to globals. Find it in graphiql.js and add
window.buildClientSchema= export.buildClientSchema = ..

And finally use compression (br, gzip) for faster schema.json loading
Have fun

[complanboy2]

I use pre-builded schema.json from introspection query for service with 10k+ types.
And init application with it.

fetch('graphql/schema.json', {
  method: 'get',
  headers: { 'Accept': 'application/json'},
  credentials: 'include',
}).then(function (response) {
    return JSON.parse(response.text());
}).then(function (schema) {
  ReactDOM.render(
    React.createElement(GraphiQL, {
      schema: buildClientSchema(schema),
      ...
    }),
    document.getElementById('graphiql')
  );      
});

Also need to add function buildClientSchema to globals. Find it in graphiql.js and add
window.buildClientSchema= export.buildClientSchema = ..

And finally use compression (br, gzip) for faster schema.json loading
Have fun

This is good although I'm curious that it is only at init. How does it capture the database changes, in this case table and it's fields (it has 10k+ tables).

[benjie]

I think there's some features that just cannot work with incremental introspection; one such thing would be search. GraphQL does not have "search" in it's introspection facilities, so we couldn't search fields/types without a full catalog of them locally. We can solve this by just pulling down the types and their fields ahead of time, and then only loading more data about each type/field when it's necessary, but I don't know if this is a sufficiently large saving to justify the added complexity (and load times!)

I think supplying the GraphQL schema as @lunicon advises would work best. This is what we do in PostGraphile, and to answer @complanboy2's question when the database changes we capture that event, pull down the latest schema, and just* update the schema prop to point to the new schema instance.

* Well, it's not _just_ update the prop, we also fiddle in GraphiQL's internal state to maintain the documentation browser's position, but I'm hoping to make this unnecessary in a future GraphiQL release.

[complanboy2]

Thank you @benjie , fetching the updated schema from the server to client on each database change is a good option. Just worry about the performance since the database has 10k+ tables. I've tried with a moderate database of same size and tried the pojo to schema generation and the size is ~ 50MB.

Now to fetch the data of this size at each database change (since there are more tables, changes probability is more) especially when your customer (who are end users for the server app) can make the database changes.

[benjie]

@complanboy2 If you go with the external schema solution (where you pass the constructed schema to GraphiQL) then you can do something fancier outside of GraphiQL, for example you could track versions of the GraphQL schema and then supply "JSON diffs" between the different versions - so you only need the 50MB the first time, after that you apply a patch using something like http://jsonpatch.com/

[acao]

facebook has a 100mb schema, and found the best option was to load the schema from a json file on the local filesystem the introspection query and the http payload are otherwise very slow. hoping to support this in graphiql 1.1.0

[complanboy2]

@complanboy2 If you go with the external schema solution (where you pass the constructed schema to GraphiQL) then you can do something fancier outside of GraphiQL, for example you could track versions of the GraphQL schema and then supply "JSON diffs" between the different versions - so you only need the 50MB the first time, after that you apply a patch using something like http://jsonpatch.com/

Thanks @benjie , i heard the graphql guys are working on to support incremental introspection queries which support the schema fetch at batches or increment levels. Hope they will address the dynamic changes as well.

@acao thanks. facebook update this when the database changes happen or is this loading when the init happen. And I think first time, they have to fetch from remote server and read the http payload before they write that to json.

[acao]

We don't know exactly how they're doing it but I think they are streaming the schema introspection JSON. http is only one vehicle for introspection! they just wanted to make it pluggable so they they could bypass the HTTP introspection query thats baked in by default. we will be exposing this from a plugin level and to graphql-config options that folks are used to for this