Gqlgen: [Proposal] Split generated stubs into multiple files

[kevinmbeaulieu]

Problem

377 introduced the problem that gqlgen's output file can get quite large (in our case, it's over 4MB / 100k lines and we've only been building on GraphQL for ~2 years- it will likely grow further as our schema continues to mature). Such large file sizes can still be problematic for a few reasons:

• Merely opening the file can take seconds to load.
• It's near impossible to find what you're looking for in a file that big.
• It's hard to get a high-level understanding of the generated code's structure because each semantic section might be thousands/tens of thousands of lines long.
• Some code review systems can be configured with a file size limit, which the current output file may exceed, forcing teams like ours to write custom post-processing scripts to break up the output of gqlgen into smaller files.

Based on the upvotes/comments on that original issue, it seems to be a relatively common problem in the community, so I would like to propose an actionable solution (which I can tentatively plan to implement if it's accepted).

Proposed solution

gqlgen already splits up the templates for the generated code into args.gotpl, directives.gotpl, field.gotpl, generated!.gotpl, input.gotpl, interface.gotpl, object.gotpl, and type.gotpl. The code generation process essentially generates code based on each of these templates, adds a header, and concatenates the results into the file specified by templates.Options.Filename.

I propose that templates.Options be updated to support passing a directory into Filename. If Filename is a single file, behavior would be unchanged. If Filename is a directory, it would output files for each template as separate files.

For instance, the following gqlgen.yaml

...
exec:
    package: schema
    filename: ./generated/
...

would yield the file structure:

generated/
    args.generated.go
    directives.generated.go
    field.generated.go
    generated!.generated.go
    input.generated.go
    interface.generated.go
    object.generated.go
    type.generated.go

The header for each *.generated.go would follow the same format as the single output file's header, so each output file would look like:

[If cfg.GeneratedHeader] // Code generated by github.com/99designs/gqlgen, DO NOT EDIT. 

[cfg.PackageDoc]

package [cfg.PackageName]

[cfg.FileNotice]

import (
    ...
)

[Code generated by `*.gotpl` template for this region]

How

In templates.Render(), update the loop over roots to write to its own file, rather than a single buffer. Also move the header lines into the loop so they get added to each file.

Open questions

1. What should the behavior be if templates.Options.Template is specified and templates.Options.Filename is a directory?

The most straightforward approach would probably be to create:

generated/
    <template name>.generated.go

But open to any other suggestions here.

1. What should the behavior of templates.Options.RegionTags be if templates.Options.Filename is a directory?

Currently, when templates.Options.Template is set, RegionTags is still used, and just encloses that single template's generated code in region tags. To mimic this behavior, it'd probably make sense to enclose each *.generated.go file contents in region tags. However this doesn't seem to be particularly useful, so the alternative would be to simply ignore RegionTags when templates.Options.Filename is set to a directory.

1. Is this division of generated files sufficient, or should we split things up further?

If splitting up the output based on the current .gotpl templates still yields files which are too large, we could take this one step further and reorganize the templates to be based on operation type rather than declaration type.

For example, given the following schema:

extend type Query {
    foo(id: ID): Foo
}
extend type Mutation {
    bar(id: ID): Bar
}

it could generate

generated/
    foo.generated.go
    bar.generated.go
    common.generated.go

where each generated file would contain all the declarations relating to that operation in the schema (plus some common.generated.go with any base declarations that might be needed which aren't specific to an individual operation). I'd be interested to gauge interest in this, but would probably leave implementation of it to a follow-up proposal.

[kevinmbeaulieu]

Quick update here: I discussed this proposal with @vektah last week and we revised the proposed output from following the region tags in the existing generated.go to following the schema (similar to the layout: follow-schema supported for resolvergen). So if your schema looks like

schema/
    common.graphql
    user.graphql
    settings.graphql
    onboarding_flow.graphql

then the output would be

generated/
    common.generated.go
    user.generated.go
    settings.generated.go
    onboarding_flow.generated.go

This would have a few benefits over the original proposal:

• Under the original proposal, field.generated.go would still be quite large. With this new proposal, if any generated file in your project gets too big, you can simply split up the corresponding schema into multiple files.
• It's easier to find the generated code corresponding to a given type/feature since it matches your schema file structure.

Hoping to have a draft of this implemented sometime in the next few weeks, but feel free to comment here with any thoughts on this revision.

[ibantoo]

@kevinmbeaulieu Hey Kevin, I follow your instruction for split the file,
here are my structure and config

- graph/
  - document.graphql
  - schema.graphql
- pkg/
  - resolver/
     - resolver.go

in document.graphql :

extend type Mutation {
    changeAccessLevelDocument(documentID: String!): Boolean! @isAuthenticated(for: ADMIN)
}

and my config like this:

schema:
  - graph/*.graphql

exec:
  filename: pkg/resolver/generated.go
  package: resolver

resolver:
  layout: follow-schema
  dir: pkg/resolver
  package: resolver
  filename_template: "{name}.resolvers.go"

and i have custom generate also for mutating hook with a filename called generate_hook.go:

package main

import (
    "fmt"
    "github.com/99designs/gqlgen/api"
    "github.com/99designs/gqlgen/codegen/config"
    "github.com/99designs/gqlgen/plugin/modelgen"
    "os"
    "strings"
)

func mutateHook(b *modelgen.ModelBuild) *modelgen.ModelBuild {
    for _, model := range b.Models {
        for _, field := range model.Fields {
            name := field.Name
            if name == "id" {
                name = "_id,omitempty"
                omit := strings.TrimSuffix(field.Tag, `"`)
                field.Tag = fmt.Sprintf(`%v,omitempty"`, omit)
            }
            field.Tag += ` bson:"` + name + `"`
        }
    }
    return b
}

func main() {
    cfg, err := config.LoadConfigFromDefaultLocations()
    if err != nil {
        _, _ = fmt.Fprintln(os.Stderr, "failed to load config", err.Error())
        os.Exit(2)
    }

    p := modelgen.Plugin{
        MutateHook: mutateHook,
    }

    err = api.Generate(cfg,
        api.NoPlugins(),
        api.AddPlugin(&p),
    )
    if err != nil {
        _, _ = fmt.Fprintln(os.Stderr, err.Error())
        os.Exit(3)
    }
}

after I run go run generate_hook.go
it doesn't generate files like what u said, and it just updated the code in pkg/resolver/generated.go in my project
am i wrong to use that??

[kevinmbeaulieu]

@ibantoo The follow-schema layout for resolver should already exist, this issue is about extending that functionality to exec as well. So not sure what's going wrong with your experience, but might be worth filing a separate ticket. From a quick look your config looks right to me, but I'm somewhat new to this so might be missing something (only sanity checks I can think of would be to make sure you're on the newest version of gqlgen, because iirc follow-schema was added semi-recently