Amplify-cli: RFC: Amplify api pull command

[UnleashedMind]

Overview

Currently, if a GraphQL API is updated outside of the Amplify CLI, such as in the AppSync console or service API calls, there is no convenient way for the customers to pull down those updates to their local development environments.
Based on customer feedbacks, such as these feature requests for a “pull” command,
https://github.com/aws-amplify/amplify-cli/issues/174
https://github.com/aws-amplify/amplify-cli/issues/293
https://github.com/aws-amplify/amplify-cli/issues/1190
we are evaluating the “amplify api pull” command that aims to bridge this disconnection between the local development and the AppSync’s web console.
This RFC defines the use case, analyzes the problems and options, and asks the community to give us feedback and suggestions on this command.

The use cases

The following are the steps of the most common workflow for the amplify api pull command:

1. Customer adds a new GraphQL API to his project (or updates an existing API) in the local development environment and pushes local development to the cloud
   $ amplify api add / update
   $ amplify push

2. Customer goes to the AppSync console and makes modifications on the GraphQL API, such as adding a new type, adding/removing a field from a type, or changing some of the resolver mappings.
   $ amplify api console

3. Customer comes back to the local development environment, pulls down the developments that happened in the AppSync console, and continues his local development.
   $ amplify api pull

There’s another use case that is associated with a “pull” command, team member collaborations using the Amplify CLI: Developer A pushes his local development to the cloud, and developer B needs to pull down the changes to his local development environment. This use case is addressed by the amplify env pull (https://aws-amplify.github.io/docs/cli/multienv#team-members-sharing-the-same-dev-backend) command, which is already implemented, and out of the scope of this document.

The amplify api pull command focuses on giving the customer the ability to combine local developments using Amplify CLI with the developments in the AppSync’s web console or any other methods outside of the Amplify CLI management.

Conversion between schemas

In developing a GraphQL API, the developer’s input to the Amplify CLI is not a regular graphql schema, but a schema annotated with graphql transformer directives. The annotated schema is called the “input schema document”.
The Graphql Transformer first builds the annotated schema locally, transforms it into the API’s regular graphql schema, resolver mapping templates, and CloudFormation templates, etc.. Those build artifacts are the actual items used by the amplilfy push command to construct and update the GraphQL API in the cloud.
When retrieving the latest updates from the he AppSync’s web console, the AppSync's public API methods, such as 'getIntrospectionSchema', ‘listTypes’ and ‘listResolvers’ etc., return the regular schema, the resolver mapping templates and other resources. They are equivalent to the above mentioned build artifacts. And currently there is no tool to reverse engineer, and re-construct the annotated “input schema document” from the build artifacts.

To summarize, the AppSync console and the local development are represented by two different formats:

• Local: annotated schema
• AppSync console: regular schema and other resources:

The hard problem is how to diff them and represent the diffs in the local development’s input format: the annotated schema.

The options

1. Reverse GraphQL Transformer’s build operation:
   Implement a full-fledged tool that reverses the GraphQL Transformer’s build operation; use it to construct the annotated schema from items retrieved from the AppSync service; and then use the constructed annotated schema to diff, resolve conflict and merge with the developer’s local “input schema document”.
   This would be the ideal solution _if it is possible_. However, it does not seem to be a trivial problem upon examining the current operations, and we would first need to refactor the GraphQL Transformer to make sure all its operations have reverse operations.

2. Best effort approach:
   Set the goal a little differently, from providing the pull command's functionalities in full, to that the cli will use its best effort to pull from the AppSync console. And if the cli cannot handle the situation, it will guide the customer to restore the project locally and in the cloud, so that she/he can continue to use the CLI in the project’s development. The ability of the “best effort” can be iteratively improved to include more and more use cases.
   _Pro_: Although the scope is limited, the partial implementation can be useful in certain scenarios, and it preserves the input format: annotated schema.
   _Con_: Can be confusing because “sometimes it works, sometimes it does not”; it can also be complicated to use when the cli tries to restore the project if it can not handle the discrepancies between the AppSync console and the local development.

3. Use regular schema
   The pull command will replace the annotated schema with the regular schema. Developments after the pull command will be directly on the regular schema, the resolver mapping templates and the CloudFormation templates, etc.
   New resources can still be set up by the GraphQL Transformer, for example, adding a new @model annotated type into the regular schema will set up the DynamoDB table etc.. Of course, the schema is now an annotated schema again.
   _Pro_: Easier to implement than the other options. Can pull down all the developments happened in the AppSync console, it is a full implementation of the pull functionalities.
   _Con_: If the customer starts with annotated schema, the execution of the pull command will change it to regular schema, it can be confusing. Development will involve both annotated schema and direct resource specifications, such as datasources and resolve mappings etc.. And there can be discrepancies and conflicts between themselves as the combined input documents, which introduces more complexities in the development.

Please comment with your thoughts and suggestions.

[hisham]

Option 2 seems best. It may be worth also to allow us to pass parameters to the pull command to target specific resolvers. Often we're just iterating on a custom resolver code and nothing else. That might simplify things.

I think when creating models and developing connections, using the annotated schema and doing a push will be faster than doing this stuff manually in the console and doing a pull.

Using the console however is useful when customizing resolver code and adding custom data sources that the graphql transformer does not yet support. So I think the pull command is most useful for custom resolvers and data sources that would need a custom nested stack anyways rather than trying to recreate an annotated graphql schema.

[jiajern]

Hi all, so what will be the work around currently? If I add an API using amplify add api and then later on when I want to change the schema, do I have to modify the schema and the graphql query that were generated at the beginning? Thanks in advance.

[tracycaruso]

1+

[YoanGJ]

First I'm very happy to know that this feature is seriously considered! I'm not using the CLI anymore because I can keep my work in sync.

I think that the first option is of course the best and if you're currently investigating this way it would be awesome to have an ETA of your work so we can have an idea to know if there's some hope to see this solution arriving between our hands anytime soon.

But personally I'm ready to wait longer without the CLI instead of having kind of a partial solution with option 2 & 3.

[phil-andrews]

This is the one big drawback I've been seeing when trying to run AppSync as a multi-client backend. There's no seamless way to modify in one client and then pull it into another. Seems to be a no-brainer that if the source API has changed that any client configured to it should be able to pull in the changes. Almost identical to git but without having to go through the process of setting up and maintaining a separate AppSync repo.

Vote = Option 1

[Genkilabs]

+1 Bump.
This would be really great for me right now. ^_^

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because of inactivity. Please open a new issue if you are still encountering problems.

[Genkilabs]

@kaustavghosh06 Thanks for keeping this request alive.

[stephanrb3]

+1

[jesse-cape]

+1

[kenchoong]

+1 this should be done

[araker]

+1 for option 1. Or make it very clear when the discrepancies can't be handled in case of 2 or schema changes (annotated <--> regular) in case of 3.