Graphback: [Discussion] Production ready client side generation

[wtrocki]

For every single model we going to create at least 10 client side queries as separate file. This gives 100 files for small projects containing just 10 models.
Files are really hard to manage - Some of them are not that useful in the project and may not even be imported. When removing model file will not be removed.

This brings me to the conclusion that we should also aggregate generated queries into single files for typescript and js targets. The investigation is needed how we can do that for gql files

[wtrocki]

CC @craicoverflow

[wtrocki]

CC @Eunovo

[craicoverflow]

Yeah there really is no need for multiple files since they are not to be edited by user.

[wtrocki]

Challenge is that graphql documents cannot be defined as a single files so we will need to have a compiled version which forces them as js or ts files.

Then having js/ts files will prevent this format from being used by apollo-codegen and graphql-code-generator forcing us to build competitive feature for both very advanced generators that have large community.

I always though that TS files were an problem as they kinda missing the types.
We missing currently js format as well.

Challenge here is to prove value of the automatically generated client side queries vs giving people control over the what files are created - since crud is global to the server and it is being used to client we might use different crud for client only. Another challenge is nestability mentioned in separate issue.

I think to approach that properly we will need to do following investigations:

1) Can we settle with using thrid party client generator (like graphql-code-generator) or we want to use our own

2) If we use graphql-code-generator to build types do we still need graphback codegenerator for client or maybe we can simply have graphql-codegen-client that will produce the same results (I have tried it an they are actually ackward to work with multiple files that is why we created our generator in first place.

3) Our generator allows creating multiple files but it is actually hard to tweak what is generated or manage manual edits (rule - do not edit anything generated (we can add comments to the top of the file)

Once we cover this we have basic plan for the client side generator.
Follow up will be to see how we can extend graphql-code-generator or anything else to expose offlineMutations from offix (as alternative to the normal mutations) and datasynchronization (so cache updates will be embedded into the client calls so people will not need to make them.

I have brief conversation with @darahayes about this while ago but we did not made any progress on that front.

[Eunovo]

Challenge is that graphql documents cannot be defined as a single files so we will need to have a >compiled version which forces them as js or ts files.

Why can't graphql documents be defined as a single file?

[wtrocki]

Because the way they are compiled into queries - each file is query. If we have all queries and mutations in single file how we can split them.

They are string basically so even if we do gqlrequire("./largefile").UserQuery most of the code generators will expect or use single files to build their types etc.

Edit:
Actually you are right - there could be way to make this work with GraphQL-Code-Generator. Requires investigation.

[Eunovo]

I put all the fragments and queries into a single file like this

fragment CommentFields on Comment {
   id
   text
   description

} 

fragment CommentExpandedFields on Comment {
   id
   text
   description
   note {
      id
      title
      description
   }
} 
// .. note fragments here

query findAllComments($limit: Int, $offset: Int) {
    findAllComments(limit: $limit, offset: $offset) {
      ...CommentExpandedFields
    }
  }

query findComments($fields: CommentInput!, $limit: Int, $offset: Int) {
    findComments(fields: $fields, limit: $limit, offset: $offset) {
      ...CommentExpandedFields
    }
  }
// ...note queries here

I ran the codegen tool. I show an abbreviated version of the output below

/**  @model  */
export type Comment = {
   __typename?: 'Comment';
  id: Scalars['ID'];
  text?: Maybe<Scalars['String']>;
  description?: Maybe<Scalars['String']>;
  /** @manyToOne field: 'comments', key: 'noteId' */
  note?: Maybe<Note>;
};

export const CommentFieldsFragmentDoc = gql`
    fragment CommentFields on Comment {
  id
  text
  description
}
    `;
export const CommentExpandedFieldsFragmentDoc = gql`
    fragment CommentExpandedFields on Comment {
  id
  text
  description
  note {
    id
    title
    description
  }
}
    `;

export function useFindAllCommentsQuery(baseOptions?: ApolloReactHooks.QueryHookOptions<FindAllCommentsQuery, FindAllCommentsQueryVariables>) {
        return ApolloReactHooks.useQuery<FindAllCommentsQuery, FindAllCommentsQueryVariables>(FindAllCommentsDocument, baseOptions);
      }
export function useFindAllCommentsLazyQuery(baseOptions?: ApolloReactHooks.LazyQueryHookOptions<FindAllCommentsQuery, FindAllCommentsQueryVariables>) {
          return ApolloReactHooks.useLazyQuery<FindAllCommentsQuery, FindAllCommentsQueryVariables>(FindAllCommentsDocument, baseOptions);
        }
export type FindAllCommentsQueryHookResult = ReturnType<typeof useFindAllCommentsQuery>;
export type FindAllCommentsLazyQueryHookResult = ReturnType<typeof useFindAllCommentsLazyQuery>;
export type FindAllCommentsQueryResult = ApolloReactCommon.QueryResult<FindAllCommentsQuery, FindAllCommentsQueryVariables>;
export const FindCommentsDocument = gql`
    query findComments($fields: CommentInput!, $limit: Int, $offset: Int) {
  findComments(fields: $fields, limit: $limit, offset: $offset) {
    ...CommentExpandedFields
  }
}
    ${CommentExpandedFieldsFragmentDoc}`;
// ...continues

It appears that the types have been generated as expected.

[wtrocki]

Then.. Can we generate types from TS files? We have couple output formats in graphback for client want to validate if we need them

[Eunovo]

Then.. Can we generate types from TS files?

Take a look at https://graphql-code-generator.com/docs/getting-started/documents-field/#available-formats

Graphql Code generator can detect gql strings in code files and generate types from them

[wtrocki]

Ok.. So challenge here is that if we go solid about graphql-code-generator it will be that people will rarely interact with our generated code - they will use wrapper. We also have a business to contribute to that wrapper by building additional plugin with our use cases from offline/delta sync/cache updates

So would it make sense to use two generators (Ours and GraphQLCodeGen) to exist even if the queries will not be consumed directly? Technically this composes nicely - we generate non generic stuff based on our strongly opiniated model and codegen wraps that into hooks or hoc, but is that actually simpler to use?

Can we depend on graphql-code-generator as a whole to generate client side queries (format here is not important (it can be gql etc.) and then use existing plugins from graphql-code-generator.

Currently, you need to setup graphback generator generate client-side queries, then use graphql-code-generator to generate wrappers in ts or js

Example config that shows complexity of this:
https://github.com/aerogear/OpenVolunteerPlatform/blob/master/platform/client/.graphqlrc.yml#L21-L37

This is a more convoluted approach. If we port Graphback Client Plugin to the GraphQL-Code-Generator plugin - this can be possibly simpler and it might invalidate entire file generator use case (as schema will be pointless to generate as well)

Do we see any drawbacks on this use case?
If we later come with delta sync schema queries would it be more beneficial to still keep client generator

Any feedback?
@craicoverflow
@machi1990
@darahayes
@Eunovo
@kingsleyzissou

Note: This is not to kill plugins - they are core of the extensibility feature - this is more if we need createResources method in the plugins itself or we will keep all inmemory and delegate some of the codegen generation to apollo-codegen or graphql-code-generator

[machi1990]

Example config that shows complexity of this:
https://github.com/aerogear/OpenVolunteerPlatform/blob/master/platform/client/.graphqlrc.yml#L21-L37

I agree with you, without a starter templates, setting up a config like that can be hard (trail and error kind of).

I have a question:

From my experience of OVP client side as a user what interests me is the generated /src/dataFacade.tsx, I always wondered why do we need this graphql folder? Is it purely for integrations with other codegen tools?

Can we have a js variant of the generated src/dataFacade.tsx file?

[wtrocki]

dataFacade.tsx is generated by GraphQL-code-generator from the config. GraphQL code generator can generate facade but it needs source queries - those are usually written by developers - in our case we can generate some of them for convenience. In this issue we are trying to decide how we:

• generate those - single file/multiple files.
• How we connect those with user-written ones
• How to remove add queries that you might not need (In OVP we have a separate config for that which is kinda hard to document
• Is there value of generic queries and how to handle multiple cases (expanding results sets to relationships)

[kingsleyzissou]

I don't have particularly strong opinions on this to be honest. I understand the concern of having too many files. My two cents though, with a single file for each query, it's easier to know what is in that file without having to open it and I can import it straight away. With a large file with multiple queries, I would have to go into the file, find the query name and then import that into whatever client-side query I'm trying to make.

[machi1990]

dataFacade.tsx is generated by GraphQL-code-generator from the config. GraphQL code generator can generate facade but it needs source queries - those are usually written by developers - in our case we can generate some of them for convenience.

Thanks for the workflow.

In this issue we are trying to decide how we:

• generate those - single file/multiple files.

Agreed. I'd agree with @craicoverflow above that generating a single file is the best way. A user will not look to edit them.

To expand my question above, I was curious to see if there is a way to make graphback-client to not generate a gql file but instead hook up with GraphQL-code-generator and generate a single facade file?

Sorry if I might be sidetracking from the original issue.

[wtrocki]

@machi1990 Your question was my question. This is possible, but ...

So would it make sense to use two generators (Ours and GraphQLCodeGen) to exist even if the queries will not be consumed directly? Technically this composes nicely - we generate non generic stuff based on our strongly opinionated model and codegen wraps that into hooks or hoc, but is that actually simpler to use?

> Can we depend on graphql-code-generator only to generate client side queries (format here is not important (it can be gql etc.) and then use existing plugins from graphql-code-generator.

To expand my question above, I was curious to see if there is a way to make graphback-client to not generate a gql file but instead hook up with GraphQL-code-generator and generate a single facade file?

• when in single file we would not be able to use fragments in their own queries
• People need to have source of this generated typings where they write their own queries anyway.
• queries are usually considered a userspace - typings and hoc are just derivatives. Mixing derivatives with user logic might by really confusing

[machi1990]

Files are really hard to manage - Some of them are not that useful in the project and may not even be imported. When removing model file will not be removed.

Strongly agree and was bitten by dandling generated gql files in this commit https://github.com/aerogear/OpenVolunteerPlatform/pull/61/commits/132438baf714735b623b5ca313ecd983410a09e9 and I had to manually remove them for the generation command to work again.

[wtrocki]

Plan for phase 1 ◀️

• I'm working on finalizing client-side extended CRUD capabilities - couple issues there
• We going to test client-side codegen in the OVP with @machi1990
• We going to explore options how we can use GraphQL-Code-Generator instead of Graphback client for client-side queries

For Phase 2

• We can explore options on how we can integrate clients with GraphQL-Code-Generator. Would queries will be in single file etc.
• Put on the table disabling code generation for schema/client-side as a whole in graphback (generation at runtime for schema only)

[wtrocki]

@Eunovo #1268 contains some small fixes for the client side queries - still some things failing

[craicoverflow]

Put on the table disabling code generation for schema/client-side as a whole in graphback (generation at runtime for schema only)

• 1 on this. The whole idea of having an independent CRUD specification makes it possible to build completely independent tooling that references the spec.

[Eunovo]

when in single file we would not be able to use fragments in their own queries

Why? All the generated query files do not explicitly import any fragment file

[wtrocki]

Why? All the generated query files do not explicitly import any fragment file

Actually you are right.
As long fragment will be there graphql-codegen will be able to produce wrapper

[Eunovo]

If we are talking about completely removing generated documents, it might be possible to allow users still use graphback's fragments (even though they are not in any files) if we generate the fragments at runtime. I don't fully understand all of graphql-code-generator's workings yet but it might be possible to do this with a custom loader or plugin.

[wtrocki]

@Eunovo That is amazing. GraphQL code generator has nicely documented plugin docs. Maybe we can explore that and see if there will be any limitation to move our client-side generator to graphql-codegen plugin and what possible and most developer-friendly ways we could use to do so.

[Eunovo]

Currently, GraphQL code generator plugins cannot modify documents. The only way to add our custom documents is to use a custom loader. This custom loader must be a JS/TS file so we would still have a generated file.

I have also discovered that we can keep our client generator and use codegen inside it.
See graphql-code-generator, programmatic-usage. This seems like the best option. The only output would be the generated types.

[Eunovo]

Also, using the codegen module, users will not need a codegen.yml file anymore.
But we would need to pass codegen config like plugins and loader to graphback-client

[wtrocki]

@Eunovo I forgot to mention that both graphback and codegen are available as GraphQL CLI so magic can happen here.

As for not being able to edit documents - that is not limitation for us. Our plugin could create an new file crud.graphql or something based on existence of our crud stuff in schema. There is no need to edit user based documents - we just want generated ones to coexist with user ones

Looks like we might need to hack our way for codegen plugin.

@ardatan FYI We are looking into building client side graphql codegen plugin.

[Eunovo]

@wtrocki I wasn't referring to "editing user documents". We can't add to the documents being processed by codegen from the plugin. The plugin can't extend the documents the way it can extend the schema.
If we generate a crud.graphql in the plugin, how is it going to get processed? If a user constructs a query with non-existent fragments(to be generated by the plugin), codegen will reject it.

[Eunovo]

@Eunovo I forgot to mention that both graphback and codegen are available as GraphQL CLI so magic can happen here.

What magic? I don't understand what you mean

[wtrocki]

@wtrocki I wasn't referring to "editing user documents". We can't add to the documents being processed by codegen from the plugin. The plugin can't extend the documents the way it can extend the schema.

Sorry for not getting back to you early. I needed to try out this myself first.
That is fine. I did not mean to add to the existing documents. I was more about creating a new document. For that, we only need schema as input

schema: ./server/src/schema/schema.graphql
documents: ./client/src/graphql/**/**.graphql
generates:
  ./client/src/graphql/crud.graphql
    plugins:
      - graphback-client
    config:
      create: false
      delete: false

  ./client/src/generated-types.tsx:
    plugins:
      - add: '/* tslint:disable */'
      - typescript
      - typescript-operations
      - typescript-react-apollo
    config:
      withComponent: false
      withHOC: false
      withHooks: true
      skipDocumentsValidation: false

There are a couple problems here:

• Client plugin is not working properly without schema plugin embedded as schema plugin adds the relationship fields etc. so in this case I needed to ignore model and use full schema

• I could not make this work as documents are read once. So if another document will be created by the plugin it will not be appended - as you mentioned.

What magic? I don't understand what you mean

Integration between graphql-codegenerator and graphback client - whatever it will be

[wtrocki]

Conclusion - it will not be easy to merge all together so we might need two cmds and figure out the best workflow/patterns for generating clients first.

[wtrocki]

Fixed by @Eunovo
Amazing work that I have tested in OVP. Really like how simple this was to resolve.

There are still some of the issues around flexibility of the generated queries that we can address in separate issues.