Postgraphile: Feature Request: Add support for Apollo Federation `@key` in generated schema

[purplemana]

I'm submitting a ...

• [ ] bug report
• [x] feature request
• [ ] question

Apollo Federation introduces a cleaner way for micro-service oriented schema stitching. The auto-generated Postgraphile schemas can include the newly introduced @key directive for the generated types, which will help consuming them in federated schemas.

E.g. Given this table:

CREATE TABLE app_public.users (
  id serial PRIMARY KEY,
  username citext NOT NULL unique,
);

The generated type would be:

type User @keys(fields: "nodeID") @keys(fields: "id") @keys(fields: "userName") {
  nodeID: ID!
  id: Int!
  userName: String!
}

[purplemana]

Was going through the documentation and examples of plugins to see how this can be implemented. This looks to be better fit for a raw plugin. Or would https://github.com/graphile/graphile-engine/blob/master/packages/graphile-build-pg/src/plugins/PgTablesPlugin.js#L106 be a better place to inject the directives?

Also, along with the directives, __resolveReference also needs to be added to the type resolvers.

[benjie]

From my reading of federation a couple days ago, we have to expose a stringified schema representation under { _service { sdl } } that shows those schema directives. GraphQL does not currently have a way of exposing directive applications through introspection, so we need to create our own GraphQL schema printer that inserts these directives manually - this can be done entirely in one field via makeExtendSchemaPlugin; though I'm not sure _how_ we'd do it yet.

Further reading:

https://github.com/graphql/graphql-spec/issues/300
https://www.apollographql.com/docs/apollo-server/federation/federation-spec/

[purplemana]

Hmmm, would you check out this sandbox created by the federation team?

https://codesandbox.io/s/x7jn4y20pp

It does produce the schema with directives. Here's the query I used:

{
  _service{
    sdl
  }
}

And here's the result:

{
  "data": {
    "_service": {
      "sdl": "type Product @key(fields: \"upc\") {\n  upc: String!\n  name: String\n  price: Int\n  weight: Int\n}\n\nextend type Query {\n  topProducts(first: Int = 5): [Product]\n}\n"
    }
  }
}

[benjie]

I think you’ve misunderstood what I wrote; your example agrees with what I said. We’d need to add the _service.sdl field via makeExtendSchemaPlugin and then its resolver has to do custom stuff (not standard introspection) to add the directives to the output.

[purplemana]

Ok, so we need to provide the sdl containing the @key directive. I would try to see how this can be done using makeExtendSchemaPlugin, if you have examples where directives are being added, please point me to them.

[benjie]

I don't; that's the part I don't know how to address. I've not researched it, but my first attempt would probably be something like this:

1. const printedSchema = printSchema(postgraphileSchema) printSchema the PostGraphile schema
2. const ast = parse(printedSchema) parse that into an AST
3. Walk the AST, adding directives in the relevant places
4. return print(ast) return the printed AST
5. Memoize all of this based on the schema object (so it still works under watch mode, but only has to be computed once per schema)

[purplemana]

This way we will have to map the AST fields with the SQL schema, augmenting table types with directive. Looking at the arguments for makeExtendSchemaPlugin, this looks feasible. However, would this be easier while the schema is being built with GraphQL types, in PgTablesPlugin? We can add directives to the table types there itself when we are building it.

[benjie]

Sure, we may need to do something like that to determine what data to expose in step 3 (we can do it with external plugins, there's no need to mod core I think), but that doesn't avoid us having to do what I outlined because adding directives to the table types will _not_ be exposed via printSchema, so we still have to implement a custom print for our schema that can output these directives. Ref: graphql/graphql-spec#300

build.scopeByType will be very helpful in implementing this, I suspect.

[purplemana]

My guess is that the new buildFederatedSchema would add these additional fields, so long as it has @key directives. I don't see any additional code used in the sandbox example.
As every Table type has a nodeId field, I will first try to add @key(fields="nodeId") directive to them, and then add __resolveReference method as well. Then I will try to use the buildFederatedSchema API and return that schema.

[benjie]

Oh, I see - I thought you were looking for PostGraphile to support this natively, but instead you're trying to mount PostGraphile's schema inside of Apollo Server.

The code you've linked to uses the GraphQL SDL:

  modulesOrSDL: (GraphQLSchemaModule | DocumentNode)[] | DocumentNode,

so you can't pass a pre-constructed schema to it, I think? Hopefully you can find a way to make it work 👍

[purplemana]

Actually the plan is to only import @apollo/federation package within the plugin code OR PostGraphile to use buildFederatedSchema. From this example, this API takes the same inputs which makeExtendSchemaPlugin returns.

[purplemana]

I don't have a clear picture of how to solve this at this point in PostGraphile. I haven't looked deeply into makeExtendSchemaPlugin. Please look at my suggestions with n00b goggles. But the overview (with a lot of guesswork) is that as PostGraphile makes it's schema through inflection, the inflection portion is where I think it'd be easy to insert the directives. Then at the very last moment where the complete schema is done, run it through buildFederatedSchema and return that SDL.
Maybe makeExtendSchemaPlugin is not the right fit for this, if it doesn't receive the SDL itself. Maybe makeProcessSchemaPlugin is better suited to do this as it is run after the schema is built and it gets the sdl.

[benjie]

inflection portion is where I think it'd be easy to insert the directives

There is no "insertion of the directives" - a built GraphQLSchema object (using the graphql-js library) does not have directives associated with its built GraphQLObjectTypes, fields, etc. The only directives it has is the consumer-facing directives that GraphQL clients can _add to queries_, such as @skip and @include.

buildFederatedSchema works by parsing the GraphQL SDL which includes these directives in the text representation. We don't use user-written SDL as the schema source in PostGraphile, instead we use the database as the schema source and generate the underlying GraphQL objects directly (which is the standard way of constructing a GraphQL schema for any automated GraphQL tool). Once you turn SDL text into a functioning GraphQL schema, all directive information is lost (see graphql/graphql-spec#300), which is why buildFederatedSchema works with the _source_ textual schema representation rather than the resulting physical GraphQL schema object.

In PostGraphile I believe that we're going to have to do the work of buildFederatedSchema ourselves - namely we have to implement a custom schema printer that works with our schema and adds in the textual representation of these directives (which _do not physically exist in our schema_ but are just added to the printer to augment the description so Apollo can understand it). So we need to add a field that exposes this _special_ textual representation. The easiest way to do that is with makeExtendSchemaPlugin.

Since the sdl field executes at run time, we don't need to use makeProcessSchemaPlugin since we don't need the GraphQL schema at build time to add it. We can just construct the results the first time the field is called (hence me suggesting memoization).

Hopefully the above has helped you understand a little more, if you go back through and re-read my previous comments they may make some more sense now. To really solidify your understanding, I suggest you create a schema with the graphql module directly using new GraphQLSchema(...) (not with graphql-tools/Apollo/etc that use SDL) and add schema federation functionalities to it. A good place to start is the official swapi Star Wars API which works in this way: https://github.com/graphql/swapi-graphql/blob/eb15f31b9e7a360d430eb419bc8723e03dd6059e/src/schema/index.js#L129

[purplemana]

Thanks for your inputs and all the patience :), let me come back with better understanding (and questions).

[pyramation]

this could be a great way to use apollo server with postgraphile, I'll be paying attention to this thread 👍

[purplemana]

I am not actively working on it. Being a non-trivial implementation requiring a good understanding of existing PostGraphile implementation and the new Apollo Federation specs, my progress has been slow. @pyramation please add your ideas and thoughts on the topic.

[ebekebe]

I played around with how federation can be enabled with minimal effort, because we really need it. Our use case is limited to adding the key directive to all types that implement Relay's Node interface with the --classic-ids option enabled. It is hacked together and working for our use case. We did this with the hope that there will be native support for federation eventually.

Also I hope that it helps @benjie in the process of finding the proper way to integrate federation natively. So please have a look: https://github.com/ebekebe/postgraphile-relay-federation

[benjie]

Oh cool, I wasn't aware of the astNode interface - it's not in the GraphQL.js documentation: https://graphql.org/graphql-js/type/#graphqlobjecttype

It looks like all of this can be achieved with standard Graphile plugins at schema build time (rather than applying modifications to the schema after the fact); are you able to create some acceptance tests for the schema so I can take a punt at refactoring it without breaking anything? Just a simple node script will do that runs a few assertions, it doesn't have to be jest tests unless you want to go that route.

[ebekebe]

I can write some tests, but I need to think about what to test and how. It will probably involve a postgres db with static data from an sql file, the apollo gateway and a second service that extends the postgraphile schema. This is how I tested it manually.
Putting this in a test setup will take some time and I probably won't be able to do it this week. Nevertheless, I am happy that you want to build on this and I try to support as much as I can.

It is true, that the documentation around graphql-js and apollo-graphql is sparse when you want to do advanced stuff. I read a lot of apollo and graphql-js code in the process. So it is important to keep in mind that my draft is very implementation specific and might break with future releases of the used libraries.

[benjie]

I think just testing that the interfaces exist and function would be best; but if you need a full stack to test it that's fine 👍 Any tests is better than no tests.

[JeffJankowski]

@ebekebe @benjie Any idea why using transformSchema within a makeProcessSchemaPlugin callback would break my resolvers? Even a trivial example will cause query errors:

makeProcessSchemaPlugin(schema => transformSchema(schema, type => type))

or

makeProcessSchemaPlugin(schema => transformSchema(schema, type => undefined)), which is supposed produce the old type when returning undefined.

I have a feeling all the custom helper methods to "recreate" the type inside transformSchema is stripping out something critical to Postgraphile. According to some github discussion, only the properties defined in the GraphQL spec are preserved during that recreation, so that would be in line with my thinking.

[benjie]

Not sure; I’d have to dig into it. But the whole point of the Graphile Engine system is so you don’t have to tweak the schema after its built, instead you tweak it as it’s being built.

[benjie]

Here's a version that uses the Graphile Engine system:

https://github.com/graphile/federation/blob/master/src/index.ts

It required PostGraphile v4.4.2-rc.0+ because I decided rather than writing things out using the manual code I'd give makeExtendSchemaPlugin support for defining scalars, unions and directives.

I've not tested it with Apollo Federation, I just scanned through the spec and implemented things using the Node interface.

[benjie]

Oh, try it with postgraphile@next since rc-0 is not officially @latest yet.

[JeffJankowski]

@benjie Thank you for whipping up that plugin; I was struggling to attach aggregated values like the _Entity union type and printed sdl with the Engine hooks.

Everything looks good except we need to hide the new federation fields for Query (_entities and _service) from printSchema. If those fields are included in the printed sdl, then the federation server will complain with something like:

(node:77124) UnhandledPromiseRejectionWarning: Error: Unknown type: "_Entity".

Not sure why the apollo's print function doesn't strip those fields out considering they're explicitly named and required for federation.

[JeffJankowski]

Forking apollo's printSchema function and stripping out the appropriate fields resolved that error for me.

const excludedFields = ['_entities', '_service'];

function printFields(
  type: GraphQLInterfaceType | GraphQLObjectType | GraphQLInputObjectType
) {
  const fields = Object.values(type.getFields())
    .filter(f => !excludedFields.includes(f.name))
    .map(
      f =>
        printDescription(f, '  ') +
        '  ' +
        f.name +
        printArgs(f.args, '  ') +
        ': ' +
        String(f.type) +
        printDeprecated(f) +
        // Federation change: print usages of federation directives.
        printFederationDirectives(f)
    );
  return printBlock(fields);
}

The federation gateway is now throwing the following:

(node:81116) UnhandledPromiseRejectionWarning: GraphQLSchemaValidationError: [postgraphile] Query -> A @key directive specifies the nodeId field which has no matching @external field.

I think we simply need to exclude the @key directive on the Query type:

builder.hook('GraphQLObjectType:interfaces', (interfaces, build, context) => {
    const { getTypeByName, inflection, nodeIdFieldName } = build;
    const { GraphQLObjectType: spec, Self, scope } = context;
    const NodeInterface = getTypeByName(inflection.builtin('Node'));
    if (
      scope.isRootQuery ||
      !NodeInterface ||
      !interfaces.includes(NodeInterface)
    ) {
      return interfaces;
    }
    ...

At this point I am able to spin up an Apollo gateway pointed at the Postgraphile instance without schema validation errors, and query as I normally would from the gateway endpoint.

I'll update this thread with any problems I encounter while I start to compose a federated schema with other services.

[benjie]

Try v0.0.2 I just pushed; it has the changes you've suggested. I've also done some refactoring, added comments, etc. If you felt like writing some tests for this and raising a PR that'd be awesome.

[JeffJankowski]

Things looks good so far in v0.02, just from my small functionality test.

If you're going to rely on @apollo/federation for printSchema (which seems reasonable), you can also pull in the federation types/directives:

https://github.com/apollographql/apollo-server/blob/master/packages/apollo-federation/src/types.ts
https://github.com/apollographql/apollo-server/blob/master/packages/apollo-federation/src/directives.ts

I know you prefer to use the graphql module passed to the plugin build object, so up to you. Also, I think printSchema will remove the federation _types_ for you, just not the fields on Query. So you shouldn't need the filter:

new FilterTypes(type => !FEDERATION_TYPE_NAMES.includes(type.name)),

[benjie]

Things looks good so far in v0.02, just from my small functionality test.

Awesome, thanks for testing. (I still haven't 😳 )

If you're going to rely on @apollo/federation for printSchema

We already have too many dependencies, I'll be trying to thin them out in future. For a POC I don't mind it, but I think having a clean implementation of the spec is a good thing. I'm also not sure if it'd work with how we load nodes.

Looking through their code I think I might be processing the entities incorrectly - looks like that _Any is going to be {__typename: "Whatever", id: "WGQafp=="} or similar, rather than just "WGQafp==" as I incorrectly assumed. Should only need a line of code or two to fix.

I think printSchema will remove the federation types [...] so you shouldn't need the filter

🤷‍♂ I'm less likely to get issues filed in future if I guarantee they're not present. Always optimising to deal gracefully with future breakage (to minimise my support load!)

[JeffJankowski]

Looking through their code I think I might be processing the entities incorrectly - looks like that _Any is going to be {__typename: "Whatever", id: "WGQafp=="} or similar, rather than just "WGQafp==" as I incorrectly assumed. Should only need a line of code or two to fix.

Yeah, I think you're right. I see the typename being added in Apollo's code as well as @ebekebe's POC. That introspection information must only be used when another federated service tries to access an @external across the service boundary, since I haven't hit any errors yet.
Edit: confirmed re-reading the federation spec

I assume these types of bugs will reveal themselves more easily as I implement some other services that leverage Postgraphile's schema.

We already have too many dependencies, I'll be trying to thin them out in future. For a POC I don't mind it, but I think having a clean implementation of the spec is a good thing. I'm also not sure if it'd work with how we load nodes.

Apollo apparently forked their schema printer from graphql-js anyway, so you could just do the same. I think that's the only @apollo/federation dependency you're you using.

[benjie]

v0.0.3 just released, parses the representations more correctly. Tested via:

query ($ids:[_Any!]!) {
  _entities(representations:$ids) {
    __typename
    ... on A{
      id
      nodeId
    }
  }
}

{
  "ids":[
    {"__typename": "A", "nodeId": "WyJhcyIsMV0="},
    {"__typename": "A", "nodeId": "WyJhcyIsMl0="}
  ]
}

(Also supports classicIds better, in theory.)

I think this might actually be legit now. Appreciate you taking it for a spin for me.

[ebekebe]

I just tried v0.0.3 and it starts up together with another federated service and both are exposed via @apollo/gateway like so:

const getSDL = async (schema) => {
    // Based on
    // https://github.com/apollographql/apollo-server/blob/master/packages/apollo-federation/src/service/__tests__/buildFederatedSchema.test.ts

    const query = '{ _service { sdl } }'
    const { data, errors } = await graphql(schema, query)
    if (errors) throw new AggregateError(errors)

    return data._service.sdl
}

// Returns a promise
const getPostgraphileSchema = config => {
    const postgraphileOptions = {
        appendPlugins: [FederationPlugin],
        classicIds: true,

        // Set false (recommended) to exclude filters, orderBy, and relations that would be expensive to access due to
        // missing indexes. Changing this from true to false is a breaking change, but false to true is not, so we recommend
        // you start with it set to false. The default is  true, in v5 the default may change to false.
        noIgnoreIndexes: false,

        // Extends the error response with additional details from the Postgres error.
        extendedErrors: ['hint', 'detail', 'errcode'],

        // Should we use relay pagination, or simple collections?
        // "omit" (default) - relay connections only,
        // "only" (not recommended) - simple collections only (no Relay connections),
        // "both" - both.
        simpleCollections: 'only'

    }

    return createPostGraphileSchema(
        config.database,
        [config.databaseSchema],
        postgraphileOptions
    )
}


const postgraphileSchema = await getPostgraphileSchema(config)

const gateway = new ApolloGateway({
    localServiceList: [
        {
            name: 'postgraphile',
            typeDefs: gql(await getSDL(postgraphileSchema)),
            schema: postgraphileSchema
        },
        {
            name: 'extensions',
            typeDefs: gql(typeDefs),
            schema
        }
    ],
    buildService ({ schema }) {
        return new LocalGraphQLDataSource(schema)
    }
})

const { schema: mergedSchema, executor } = await gateway.load()
const apollo = new ApolloServer({ schema: mergedSchema, executor })

The good news is that startup and introspection work as expected. However, I tried to run a query and get a weird error that looks like that pgClient is not passed correctly in the resolver context. I didn't look into it too deeply, since @benjie might have a better guess at what is wrong here:

TypeError: Cannot read property 'query' of undefined
    at query ([...]/node_modules/graphile-build-pg/src/plugins/PgMutationCreatePlugin.js:234:40)
    at field.resolve ([...]/node_modules/@apollo/gateway/node_modules/graphql-extensions/src/index.ts:277:18)
    at resolveFieldValueOrError ([...]/node_modules/graphql/execution/execute.js:486:18)
   [...]

Since introspection works I am pretty sure, that my database config must valid and somehow be used by graphile already.

Another guess: This query is executed from the gateway as it tries to resolve a postgraphile type that is used by the other federated service. So maybe the gateway passes a different context in that case.

[benjie]

Use PostGraphile as a remote schema, not a local schema. If you use it as a local schema you have to use withPostGraphileContext to generate the context (as documented in the schema-only usage docs) and I haven't looked into how to merge that with Apollo Federation (i.e. how to ensure the context _gets released_ when the request is complete).

[ebekebe]

I worked on it again today and federation works for me now, when I don't use schema-only. Thanks for all the help. 😀

[benjie]

Can everyone who's using federation chime in and let me know how it's going, what the limitations are in the Graphile Federation plugin, etc? I have a window of time coming up...

[phryneas]

Right now, I'm using a version I manually patched with PR-2. That seems to be working quite well.

I even have a plugin that pushes additional @key annotations. Might be nice to have a smart annotation for that.

But so far, I'm happy :)

[ebekebe]

Currently, we are more limited by apollo gateway than postgraphile. In particular, we need to share non-entity types like Location {lat, lon}, which is not fully supported yet. So I cannot report a lot of insights.

Nevertheless, we noticed that working with nodeId is a split experience: On the one hand it is what makes postgraphile work with federation, on the other hand it results in a complex GraphQL interface that works with (in our case) UUID and ID. Foreign keys are always UUID, but ID is used for the id (nodeId), which is the federation @key.
This leads to "ugly" types as demonstrated here (with classicIds: true):

Service P (postgraphile):

type FavoritePlace implements Node @key(fields: "id") {
    id: ID!
    rowId: UUID!

    name: String!

    pointOfInterestId: ID!
    addressId: UUID!
    address: Address!
}

type Address implements Node @key(fields: "id") {
    id: ID!
    rowId: UUID!
    ...
}

Service Q:

extend type FavoritePlace @key(fields: "id") {
    id: ID! @external

    pointOfInterestId: ID! @external
    pointOfInterest: PointOfInterest! @requires(fields: "pointOfInterestId")
}

type PointOfInterest @key(fields: "id") {
    id: ID!
    ...
}

Merged schema:

type FavoritePlace implements Node {
    id: ID!
    rowId: UUID!

    name: String!

    pointOfInterestId: ID!
    pointOfInterest: PointOfInterest!

    addressId: UUID!
    address: Address!
}

type Address implements Node @key(fields: "id") {
    id: ID!
    rowId: UUID!
    ...
}

type PointOfInterest @key(fields: "id") {
    id: ID!
    ...
}

Here we have the weird case that FavoritePlace.addressId and FavoritePlace.pointOfInterestId have different types.
Primary keys, visible to API consumers, are of type ID, but foreign keys inside of postgraphile are UUID. However, foreign keys that are resolved by another service via federation use ID.

Ideally I would be able to hide the UUID type completely from my API consumers, because it is an implementation detail leaking out.

The really annoying thing is, however, that even if I commit completely to UUID and try to use it everywhere as key, graphile federation will always use id: ID! or nodeId: ID! as federation @key. So I am stuck with both types.

[JeffJankowski]

As @ebekebe said, I'm more limited by Apollo's federation gateway/tools than anything on the postgraphile side. I think everyone is still working out the kinks for federation in non-trivial applications.

I'll keep my eye on this thread, and make sure to post any issues I run into with postgraphile while the tech evolves.

[benjie]

Since it basically seems to be working I'm going to close this issue; please file further issues against:

https://github.com/graphile/federation

(Also, do star it/watch it.)

@ebekebe The primary key / nodeId duality is something that frustrates me too. We have it due to legacy (v3 had it) but I'm planning to make it optional (default off) in v5 so you always use the Relay object IDs and the PKs aren't present.

Hopefully Graphile federation won't always be dependent on the node interface.