Apollo-tooling: Proposal: Defining a simpler IR for simpler targets

[rricard]

Introduction

apollo-codegen's shared base has been designed to offer a fully-fledged IR for the Apollo iOS/Swift Target. This target has a complex output and rely on the complex IR.
However, some targets such as Flow and TS are way simpler in their final form. This proposal introduces a simple and declarative way to describe what we should output in those targets.

Principle

Today we're sending a very complex structure that carries all of the information provided by GraphQL directly into every target, this makes it very hard for any newcomer to implement any simple language:

This information is important and should be kept, that's why we're only going to create a new parallel representation that should make it super dumb to create a language output:

This new abstract IR does not share anything with the original GraphQL representation. It just describes type definitions that should be converted to an another language.

Old proposal

Type specification

Here is a typed representation of this new IR:

// Defines the new IR
type AbstractIR = {
  // A list of types to export
  types: TypeDeclaration[],
};

// Defines a type to export, for instance, in flow:
//   export type HeroQuery = (...);
type TypeDeclaration = {
  // Name of the type such as:
  //   HeroQuery, HeroQuery_hero, HeroQuery_hero_name
  name: string,
  // Contents of the type
  definition: TypeDefinition[],
  // A comment you can add that will go before the declaration
  comment?: string,
};

// One of the ways to describe contents of a type
type TypeDefinition =
  RecordDefinition |
  UnionDefinition |
  IntersectionDefinition |
  ReferenceDefinition;

// Defines a record-shaped content, for instance:
//   {
//     (...)
//   }
type RecordDefinition = {
  defining: 'record',
  // List of all fields in the record
  fields: Field[],
};

// Defines a field in a record, for instance, in flow:
//   hero: ?HeroQuery_hero,
type Field = {
  // The name of the field
  name: string,
  // Define this one if you reference a GraphQL scalar,
  // we will need to convert the scalar into something that corresponds in the language
  // For instance: String => string, Int => number
  graphqlScalarTypeName: ?string,
  // Defines a reference to an another type generated via this IR
  // For instance: HeroQuery_hero, HeroQuery_hero_name
  referencedTypeName: ?string,
  // Defines a name as a string litteral in the target language, for instance:
  //   'Hero', 'Droid'
  stringLitteralType?: string,
  // Defines if the type is optional, for instance: ?HeroQuery_hero
  optional?: boolean,
  // Defines if the type is in a list: HeroQuery_hero[]
  inList?: boolean,
  // Defines if the type is in a list is optional: (?HeroQuery_hero)[]
  optionalInList?: boolean,
  // A comment you can add that will go alongside the field
  comment?: string,
};

// Defines recursively a type that represents an union, for instance:
//   ((...)) | ((...)) | ((...))
type UnionDefinition = {
  defining: 'union',
  // List of unioned types
  definitions: TypeDefinition[],
};

// Defines recursively a type that represents an intersection, for instance:
//   ((...)) & ((...)) & ((...))
type IntersectionDefinition = {
  defining: 'intersection',
  // List of intersected types
  definitions: TypeDefinition[],
};

// Defines a reference to an another type generated via this IR:
//   HeroQuery_hero
type ReferenceDefinition = {
  defining: 'reference',
  // The reference name
  referencedTypeName: string,
};

This should cover most of the type generation needs and at least make Flow & TS share more logic.

Example

Here is a sample query we would like to represent in this IR, and show how the IR would translate to Flow:

query Hero {
  hero {
    friends {
      name
    }
    ... on Droid {
      primaryFunction {
        friends {
          appearsIn
        }
      }
    }
  }
}

Would output this AbstractIR:

{
  types: [
    {
      name: 'Hero',
      definition: {
        defining: 'record',
        fields: [
          { name: '__typename', stringLitteralType: 'Query' },
          { name: 'hero', referencedTypeName: 'Hero_hero' },
        ],
      },
    },
    {
      name: 'Hero_hero',
      definition: {
        defining: 'intersection',
        definitions: [
          {
            defining: 'record',
            fields: [
              { name: '__typename', stringLitteralType: 'Hero' },
              { name: 'friends', referencedTypeName: 'Hero_hero_friends', inList: true },
            ],
          },
          {
            defining: 'union',
            definitions: [
              { defining: 'record', fields: [] },
              {
                defining: 'record',
                fields: [
                  { name: '__typename', stringLitteralType: 'Droid' },
                  { name: 'primaryFunction', referencedTypeName: 'Hero_hero_primaryFunction' },
                ],
              },
            ],
          },
        ],
      },
    },
    {
      name: 'Hero_hero_friends',
      definition: {
        defining: 'record',
        fields: [
          { name: '__typename', stringLitteralType: 'Hero' },
          { name: 'name', graphqlScalarTypeName: 'String' },
        ],
      },
    },
    {
      name: 'Hero_hero_primaryFunction',
      definition: {
        defining: 'record',
        fields: [
          { name: '__typename', stringLitteralType: 'DroidFunction' },
          { name: 'friends', referencedTypeName: 'Hero_hero_primaryFunction_friends' },
        ],
      },
    },
    {
      name: 'Hero_hero_primaryFunction_friends',
      definition: {
        defining: 'record',
        fields: [
          { name: '__typename', stringLitteralType: 'Hero' },
          { name: 'appearsIn', graphqlScalarTypeName: 'String', optional: true },
        ],
      },
    },
  ],
}

This IR would directly translate, in Flow syntax:

export type Hero = {
  __typename: 'Query',
  hero: Hero_hero,
};

export type Hero_hero = ({
  __typename: 'Hero',
  friends: Hero_hero_friends[],
}&({
}|{
  __typename: 'Droid',
  primaryFunction: Hero_hero_primaryFunction,
}));

export type Hero_hero_friends = {
  __typename: 'Hero',
  name: string,
};

export type Hero_hero_primaryFunction = {
  __typename: 'DroidFunction',
  friends: Hero_hero_primaryFunction_friends,
};

export type Hero_hero_primaryFunction_friends = {
  __typename: 'Hero',
  appearsIn?: string,
};

As you can see, most of the work happens when translating the GraphQL-based IR to the AbstractIR. Translating from AbstractIR to an actual language does not need much work and do not require to manipulate things like indentation.

New Proposal

Related Issues/PR

192 inspired this new IR design, many thanks to @lewisf

206 directly linked, this is an internal library

202 is definitely something doable easily after doing something like this

201 will be easier since the API surface with this will be greatly reduced

190 will not be needed if this is solved

183 would likely be fixed with this pr

192 actually does something very similar to that but as an after step

I may forget some so feel free to add them in the comments

[martijnwalraven]

I like the idea of having a lower level IR (or part of the IR) that is closer to the semantics of the target languages.

We also need different levels of representation for the more complex targets like Swift and Android, so it's my hope some of that will also be useful for Flow and TypeScript. But you may want to go further and make it even closer to the code you want to generate for those languages.

My concern with some of this is that it's very closely tied to the current output of the Flow and TypeScript targets, and it doesn't leave much room for alternatives. For example, the fact that the __typename is the discriminant is implicit in your representation, which works for discriminated unions in Flow and TypeScript but may not work for case classes in Scala or variants in Reason. Whether to nest types or not, or how to generate compound names, is also something other targets may want to make different choices about.

Another point of caution is with the representation of fields. I think there is a lot of benefit of sticking to graphql-js types for this, and use utility methods to convert them to target-specific types. At a minimum, you'll need a compositional representation of types, because the combination of optional, inList and optionalInList is not enough to represent all types. Lists may be nested, so [[Int]]! is a valid GraphQL type that you won't be able to represent otherwise.

As an example of what I'm thinking of for the target independent IR, let's take a query that's similar to the one you included above (but with primaryFunction as a leaf field):

query Hero {
  hero {
    name
    friends {
      name
    }
    ... on Droid {
      primaryFunction
      friends {
        appearsIn
      }
    }
  }
}

What I'd want out of this is a list of fields (a record) for every possible type. One way of doing that would be to generate something like a type case (the name 'type case' comes from Modula and LISP, and it seemed to fit :)) for every selection set:

interface TypeCase {
  default: Record;
  typeMap: Map<GraphQLObjectType, Record>;
}

interface Record {
  fields: Field[];
}

The reason for including a default case here is that we don't want to generate structs for every possible type in Swift if the selection set doesn't contain specific conditions for those types.

For the query above, you can imagine there being dozens of possible Characters for example, and the only one we need to differentiate is Droid.

{
  kind: 'CompositeField',
  name: 'hero',
  type: 'Character',
  typeCase: {
    default: {
      fields: [
        {
          kind: 'LeafField',
          name: 'name',
          type: 'String'
        },
        {
          kind: 'CompositeField',
          name: 'friends',
          type: '[Character]',
          typeCase: {
            default: {
              fields: [
                {
                  kind: 'LeafField',
                  name: 'name',
                  type: 'String'
                }
              ]
            }
          }
        }
      ]
    },
    typeMap: {
      Droid: {
        fields: [
          {
            kind: 'LeafField',
            name: 'name',
            type: 'String'
          },
          {
            kind: 'LeafField',
            name: 'primaryFunction',
            type: 'String'
          },
          {
            kind: 'CompositeField',
            name: 'friends',
            type: '[Character]',
            typeCase: {
              default: {
                fields: [
                  {
                    kind: 'LeafField',
                    name: 'name',
                    type: 'String'
                  },
                  {
                    kind: 'LeafField',
                    name: 'appearsIn',
                    type: '[Episode]'
                  }
                ]
              }
            }
          }
        ]
      }
    }
  }
}

It seems a representation like this would be a good starting point for the Flow and TypeScript targets as well. You could process it first into something similar to the IR you proposed, or maybe it's close enough to the generated code to not require further transformations.

Not sure if this makes sense, but feedback is very welcome!

[rricard]

@martijnwalraven You're completely right! The __typename is very language-bound, I did not think much about it yesterday. So that info should be moved in a different location indeed...
optional, inList and optionalInList are also solutions that I poorly thought of as well, I think we can have a solution without relying on something too close to graphql though...
I like the idea of the typeMap though, either a target would choose to directly try to resolve in the typemap and insert directly its contents in place or just output the whole typemap without thinking much about it... I'll try to do a revised proposal tonight.

[rricard]

New proposal

Type specification

Here is a typed representation of this new IR:

// Defines the new IR
type AbstractIR = {
  // A list of the primary types to export (operations and fragments)
  primaryTypeDeclarations: TypeDeclaration[],
  // A map of dependended on types (nested types)
  typeMap: { [referencedTypeName: string]: TypeDeclaration }
};

// Defines a type to export, for instance, in flow:
//   export type HeroQuery = (...);
type TypeDeclaration = {
  // Name of the type such as:
  //   HeroQuery, HeroQuery_hero, HeroQuery_hero_name
  name: string,
  // Contents of the type
  definition: TypeDefinition[],
  // A comment you can add that will go before the declaration
  comment?: string,
};

// One of the ways to describe contents of a type
type TypeDefinition =
  RecordDefinition |
  UnionDefinition |
  IntersectionDefinition |
  ReferenceDefinition;

// Defines a record-shaped content, for instance:
//   {
//     (...)
//   }
type RecordDefinition = {
  defining: 'record',
  // List of all fields in the record
  fields: Field[],
  // Original typename  around those fields in the schema
  schemaType: string,
};

// Defines a field in a record, for instance, in flow:
//   hero: ?HeroQuery_hero,
type Field = {
  // The name of the field
  name: string,
  // Define this one if you reference a GraphQL scalar,
  // we will need to convert the scalar into something that corresponds in the language
  // For instance: String => string, Int => number
  graphqlScalarTypeName: ?string,
  // Defines a reference to an another type in the typeMap
  // For instance: HeroQuery_hero, HeroQuery_hero_name
  referencedTypeName: ?string,
  // Defines types wrapping the actual type, for instance, we could wrap it
  // in an array and/or a nullable and/or an another array
  // The outermost wrapper is the last in the list
  typeWrappers?: TypeWrapper[],
  // A comment you can add that will go alongside the field
  comment?: string,
};

// Defines a decoration around a type such as an array or a nullable
type TypeWrapper = 'Array'|'Nullable';

// Defines recursively a type that represents an union, for instance:
//   ((...)) | ((...)) | ((...))
type UnionDefinition = {
  defining: 'union',
  // List of unioned types
  definitions: TypeDefinition[],
};

// Defines recursively a type that represents an intersection, for instance:
//   ((...)) & ((...)) & ((...))
type IntersectionDefinition = {
  defining: 'intersection',
  // List of intersected types
  definitions: TypeDefinition[],
};

// Defines a reference to an another type generated via this IR:
//   HeroQuery_hero
type ReferenceDefinition = {
  defining: 'reference',
  // The reference name in the typeMap
  referencedTypeName: string,
};

Example

Here is a sample query we would like to represent in this IR, and show how the IR would translate to Flow:

query Hero {
  hero {
    friends {
      name
    }
    ... on Droid {
      primaryFunction {
        friends {
          appearsIn
        }
      }
    }
  }
}

Would output this AbstractIR:

{
  primaryTypeDeclarations: [
    {
      name: 'Hero',
      definition: {
        defining: 'record',
        schemaType: 'Query',
        fields: [
          { name: 'hero', referencedTypeName: 'Hero_hero' },
        ],
      },
    },
  ],
  typeMap: {
    Hero_hero: {
      name: 'Hero_hero',
      definition: {
        defining: 'intersection',
        definitions: [
          {
            defining: 'record',
            schemaType: 'Hero',
            fields: [
              {
                name: 'friends',
                referencedTypeName: 'Hero_hero_friends',
                typeWrappers: ['Array'],
              },
            ],
          },
          {
            defining: 'union',
            definitions: [
              { defining: 'record', schemaType: 'Hero', fields: [] },
              {
                defining: 'record',
                schemaType: 'Droid',
                fields: [
                  { name: 'primaryFunction', referencedTypeName: 'Hero_hero_primaryFunction' },
                ],
              },
            ],
          },
        ],
      },
    },
    Hero_hero_friends: {
      name: 'Hero_hero_friends',
      definition: {
        defining: 'record',
        schemaType: 'Hero',
        fields: [
          { name: 'name', graphqlScalarTypeName: 'String' },
        ],
      },
    },
    Hero_hero_primaryFunction: {
      name: 'Hero_hero_primaryFunction',
      definition: {
        defining: 'record',
        schemaType: 'DroidFunction',
        fields: [
          { name: 'friends', referencedTypeName: 'Hero_hero_primaryFunction_friends' },
        ],
      },
    },
    Hero_hero_primaryFunction_friends: {
      name: 'Hero_hero_primaryFunction_friends',
      definition: {
        defining: 'record',
        schemaType: 'Hero',
        fields: [
          { name: 'appearsIn', graphqlScalarTypeName: 'String', typeWrappers: ['Optional'] },
        ],
      },
    },
  ],
}

This IR would directly translate, in Flow syntax:

export type Hero = {
  __typename: 'Query',
  hero: Hero_hero,
};

export type Hero_hero = ({
  __typename: 'Hero',
  friends: Hero_hero_friends[],
}&({
}|{
  __typename: 'Droid',
  primaryFunction: Hero_hero_primaryFunction,
}));

export type Hero_hero_friends = {
  __typename: 'Hero',
  name: string,
};

export type Hero_hero_primaryFunction = {
  __typename: 'DroidFunction',
  friends: Hero_hero_primaryFunction_friends,
};

export type Hero_hero_primaryFunction_friends = {
  __typename: 'Hero',
  appearsIn?: string,
};

[rricard]

@martijnwalraven I iterated on my proposal with this latest comment. I tried to add things from your comment. I know that IR is still not sufficient to cover everything and that we still need something close to graphql. I just want to see how far I can push this...

[pspeter3]

Do we also imagine the IR allowing people to generate custom files? This project is excellent but highly coupled with the Apollo ecosystem right now. I'd like to be able to generate the IR and then hydrate custom templates from the IR.

[martijnwalraven]

@pspeter3: Sure, it would be great if people could write their own targets if they have specific needs. For the Swift target, and to a lesser extent Flow and TypeScript, templates do not seem like a good match because the processing can get pretty complicated. But if you have simpler needs (or a better templating system!) you should be able to hook that up to the IR.

[pspeter3]

I guess what I'm imaging is splitting apollo-codegen into several packages. The first is responsible for converting queries to a stable and documented IR. Then there N packages for converting the IR to a specific output (eg TypeScript, Swift, Flow, Scala). The apollo-codegen convenience package would then include all of those and handle the pipeline for you.

[martijnwalraven]

Yes, I think that is exactly where we'd like to end up. There is an existing issue for moving to lerna.