Prisma1: Graphcool 1.0: New `graphcool.yml` format

[schickling]

Graphcool 1.0 introduces many changes which requires some adjustments of the graphcool.yml format.

What's new

• Local .graphcoolrc file is now part of graphcool.yml (stages instead of targets)
• stages are now pointing to the cluster the service should be deployed to
• Added apikey in favour of root tokens
• datamodel can be split up in multiple files #774 (formerly types)
• Added migrations which references YAML migration files #1263
• Renamed functions to subscriptions and restructured layout (removed handler level)
• Removed permissions & authTokens

Full Example

service: my-demo-app

stages:
  default: dev
  dev: local
  staging: shared-eu-west-1
  prod: shared-eu-west-1

apikey: my-secret-apikey

datamodel:
- database/models.graphql
- database/enums.graphql

schema: schemas/database.graphql

migrations: database/migrations/

custom:
  serverlessEndpoint: 'https://bcdeaxokbj.execute-api.eu-west-1.amazonaws.com/dev'

subscriptions:
  userSignupEmail:
    query: database/subscriptions/user-signup.graphql
    webhook:
      url: ${self:custom.serverlessEndpoint}/user-signup
      headers:
        Authorization: ${env:MY_ENDPOINT_SECRET}

Minimal example

service: my-demo-app

stages:
  dev: local

datamodel: database/types.graphql

In this example the following defaults are used:

• Missing apikey: No API key required to authenticate to the GraphQL API (not recommended)
• Missing migrations: By default looks for files matching this pattern migrations/*.yml
• Missing schema: See #1361
• Missing subscriptions: Optional
• Missing custom: Optional

Notes

Merging .graphcoolrc and graphcool.yml

Unifying both file reduces the "file footprint" and therefore simplifies the getting started experience.

You might argue that moving the targets section from the .graphcoolrc to the graphcool.yml is violating the 12 factor app principles. While this is technically the case, it's very simple to get back the full separation if needed like in the following examples:

Using environment variables

targets:
  default: dev
  dev: ${env:GRAPHCOOL_TARGET_LOCAL}
  staging: ${env:GRAPHCOOL_TARGET_STAGING}
  prod: ${env:GRAPHCOOL_TARGET_PROD}

Using source file

targets: ${file(.graphcoolrc.yml):targets}

Open questions

• [x] Should migrations point to a directory or to *.yml files via a globbing pattern?
  
  
  
  • decision: directory
  
  
• [x] This depends on #1329
• [x] Should we already introduce a version: 1 field (see docker-compose)
  
  
  
  • decision: no, we can introduce version: 2 field when we need it
  
  

[kbrandwijk]

• Putting targets in graphcool.yml forces you to manually reference an external file. I don't see any benefits of having targets in the same file, based on a normal workflow. You would normally not want to have these targets under source control. It simply doesn't work for local targets, and is inconvenient for shared cluster targets
• Using different environment variables for different targets is still hard. I don't understand why this can't be made easier (using dotenv for example)
• The serverless endpoint for example, also has a stage (target) in it. So with multiple targets, this will introduce another set of environment variables, or the use of ${opt:stage}, which is not necessarily a 1-on-1 match
• Migrations are linked to a deployment, not to the service definition itself. It's the migration from one version to the next, but there's no version concept. I don't think migrations belong in the service definition
• Disconnecting the service name from the stage (e.g. allowing the same service name for multiple stages) should make configuration a lot easier. Like serverless, you would have just one service name, and it can be deployed to multiple stages and clusters. This would be far more straightforward
• If rootTokens are removed, then how do you set/retrieve the rootToken to connect to the endpoint?
• If targets are pointing to service names, and endpoints are still using service id's, that means that it's no longer possible to know the endpoint address by just looking at .graphcoolrc, you would need to run gc info. This is a very common task, that should be as easy as possible. Unless you are also going to use service names for API endpoints, then this is not an issue.

[kbrandwijk]

The proposal in #1329 might affect the graphcool.yml structure, as it proposes to add service name to this file, as well as defaults for stage, cluster and region.

[schickling]

Edit: I've adjusted the initial proposal to incorporate these changes.

---

Thanks a lot for your great suggestions @kbrandwijk. 👍

Following your proposal here is what the graphcool.yml file would look like:

service: my-demo-app

stages:
  default: dev
  dev: local
  staging: shared-eu-west-1
  prod: shared-eu-west-1

types:
- database/models.graphql
- database/enums.graphql

migrations: database/migrations/*.yml 

custom:
  serverlessEndpoint: 'https://bcdeaxokbj.execute-api.eu-west-1.amazonaws.com/dev'

subscriptions:
  userSignupEmail:
    query: database/subscriptions/user-signup.graphql
    webhook:
      url: ${self:custom.serverlessEndpoint}/user-signup
      headers:
        Authorization: ${env:MY_ENDPOINT_SECRET}

[kbrandwijk]

Maybe cluster and regions (shared-eu-west-1) should be split, but that depends on what happens with #1329. This is more like Serverless:

service: my-demo-app
defaults:
  stage: dev
  cluster: shared
  region: eu-west-1

stages:
  dev: local #shorthand notation
  test:
    region: us-east-2

types:
- database/models.graphql
- database/enums.graphql

migrations: database/migrations/*.yml 

custom:
  serverlessEndpoint: 'https://bcdeaxokbj.execute-api.eu-west-1.amazonaws.com/dev'

subscriptions:
  userSignupEmail:
    query: database/subscriptions/user-signup.graphql
    webhook:
      url: ${self:custom.serverlessEndpoint}/user-signup
      headers:
        Authorization: ${env:MY_ENDPOINT_SECRET}

You only specify the defaults once, and only specify the overrides for the stages.

[sorenbs]

@schickling - Is the final decision that default is a special key in the stages map?

stages:
  default: dev
  dev: local
  staging: shared-eu-west-1
  prod: shared-eu-west-1

[sorenbs]

Two considerations:

• It should be possible to specify a different apiKey for different stages.
• It should be possible to rotate apiKey without any applications using it loosing access. This requires support for multiple active apiKeys
• The apiKey should not be in transit between application and graphcool. Instead it should be used to sign a jwt. Hence I suggest we rename it to secret.
• It should be highly discouraged to write secrets directly in graphcool.yml as this will be checked into git.

I suggest the following format:

service: my-demo-app

stages:
  default: dev
  dev: 
    cluster: local
    secrets:
      - ${env:GRAPHCOOL_SECRET}
  staging: 
    cluster: shared-eu-west-1
    secrets:
      - ${loaded from .env file}
  prod: 
    cluster: shared-eu-west-1
    secrets:
      - ${loaded from .env file}

Additionally I suggest that graphcool init creates a .env file with a cryptographically secure secret.

[kbrandwijk]

I like the secret approach. That will also close https://github.com/graphcool/framework/issues/362 as part of this proposal.

[sorenbs]

Thanks @kbrandwijk!

Do you see any issue in allowing the secret to be set directly in graphcool.yml?
I'm concerned that people might accidentally commit secrets to source control, but I think we can alleviate that issue somehow by always referencing .env in examples and files generated by the cli.

[kbrandwijk]

@sorenbs I don't think you need to enforce that. Educate people to follow best practices, after that, it's up to them.

[sorenbs]

@schickling new Minimal example:

service: my-demo-app

stages:
  dev:
    cluster: local
    secrets: mySecret

datamodel: database/types.graphql

[sorenbs]

After much deliberation with @schickling we came to a new proposal. The main points driving this proposal are:

• We prefer to keep the stages field simple

service: my-demo-app

disableAuth: ${env:DISABLE_GRAPHCOOL_AUTH}
secret: "my-secret"

stages:
  dev: local

datamodel: database/types.graphql

Environment Variables

The Graphcool CLI will load environment variables from three places in the following order:

• the local environment
• a .env file specified with the --dotenv parameter
• a file called .env in the same directory, if no --dotenv parameter is specified

Auth

A Graphcool service can have authentication enabled or disabled. When enabled, requests to the service must include an authorisation header:

Authorisation: "Bearer ${jwt}"

Where ${jwt} is a token signed with one of the secrets provided in graphcool.yml.

Setting secrets

It is possible to set one or more secrets for a service. Secrets can be set in two ways:

the secret field

secret: my-secret

```
secret: my-secret, my-secret-2

> The format of secrets has to be a utf8 string without spaces less than 256 characters

**GRAPHCOOL_SECRET environment variable**

if no `secret` field is present, the cli will try to read the `GRAPHCOOL_SECRET` environment variable.

GRAPHCOOL_SECRET=my-secret

GRAPHCOOL_SECRET="my-secret, my-secret-2"

> Note that in both cases it is possible to provide multiple secrets as a comma separated list. All whitespace is removed, so you can include a space after the comma.

To disable authentication, set `disableAuth: true`. If no secrets are set and `disableAuth` is not set to true, the CLI will return an error.

### Simple Example

In a simple case where the developer want to run the service locally without auth and be able to deploy to production from their local machine, the setup could look like this:

#### graphcool.yml
```yml
service: my-demo-app

disableAuth: ${env:DISABLE_GRAPHCOOL_AUTH}

stages:
  default: dev
  dev: local
  prod: london-cluster

datamodel: database/types.graphql

.env

This is the env configuration for local development

GRAPHCOOL_DISABLE_AUTH=true

.env.prod

This is the env configuration for production

GRAPHCOOL_SECRET=secret-1, secret-2

Deploying the service

Deploying to local dev:

graphcool deploy

Deploying to production:

graphcool deploy --stage prod --dotenv .env.prod

Open Questions

• [ ] Should we automatically load .env file if present? now requires explicit loading by using the argument --dotenv=true
• [ ] Should we support passing env arguments directly like this -e GRAPHCOOL_SECRET="secret-1, secret-2"?
• [ ] Should we also automatically read GRAPHCOOL_DISABLE_AUTH from the environment?

Known Tradeoffs

• Secrets are included on every deploy operation as the CLI has no way to know when secrets have changed. This increases the likelihood of accidentally changing a secret. This issue is somewhat mitigated by the fact that you cannot remove a secret without also changing the value of the disableAuth field.

[marktani]

A couple of questions:

• what exactly is the effect of disabling auth? is this about the client API, deployment, ...?
• it looks like secrets is of type String in the Json schema, and only _encodes_ a String list (comma separated list of values). Wouldn't it be cleaner if secrets is a String array instead:

secrets:
  - mysupersecret
  - ${env:PROD_SECRET}

[sorenbs]

@marktani - Good questions. I have updated my comment to make this more clear.

what exactly is the effect of disabling auth?

You can query the service without an Authorization header.

secrets is a string instead of a list

The reason to support string is that it allows us to support a simpler syntax in the case where only one is required. Additionally, we need to support a comma separated list in a string to support environment variables. We could consider also supporting a list in the yml definition.
Please note that we renamed it to secret per Johannes request

[FredyC]

Um, I feel kinda confused here. Where can I find more information about why has been permissions removed? Is there some replacement I am failing to find?

[marktani]

Great question @FredyC 🙂
Permissions are moving towards the GraphQL Server in Graphcool 1.0. You can find out more about the changes in 1.0 in the announcement blog post and in the discussion thread in the Forum.

The forum is also the best place to discuss further questions 🙂

[FredyC]

Ok, my opinion regarding secrets. I do have a different proposal. Why not allow to manage secrets with CLI tool? In my opinion, it's way more secure and less awkward than having such security measure in plain text. It could be even write-only operation. Then you can either add another secret or "deprecate" existing ones. This approach would prevent accidental commits of secrets to source control. Furthermore, this allows having a different set of secrets per stage more naturally since CLI is already working with stages.

Another pointer is regarding disableAuth. What about simply disabling auth if you don't have any secret specified? It's kinda seamless way of handling this instead of a global flag for all stages.

[FredyC]

Hm, so no opinion on my opinion? It feels super awkward right now to have secrets in a plain text :/

[marktani]

You'll typically use env vars to provide the secrets:

secret: ${env:GRAPHCOOL_SECRET}

Does that answer your question? 🙂

[FredyC]

Well, I feel it's a bit awkward way of handling that. I have to either keep those secrets in .env files (plain text again) and transport those manually in some secure way to other developers whenever it's needed/changed. Handling environment variables on Windows is generally the pain (you know where) so I am not even considering that a valid solution.

I am wondering what part you don't like about handling this with CLI. For example, Heroku has it like that and I totally love it. I understand it's extra labor to develop this, but in my opinion, it's totally worth it for security concerns and ease of use.

[marktani]

Ah now I see your point! I created a new feature request: #1553.
For now, we'll stick to the current proposal, but will closely assess your suggestion and other alternatives in the mentioned issue.

Thanks! 🙂

[jcheroske]

Is the dotenv stuff a sure thing? If so, when is that expected to drop?

Thanks for a your hard work!

[kbrandwijk]

@jcheroske afaik, dotenv has already dropped in the latest 1.0RC. The graphcool cli will read values from the .env file if found in the project folder.

[marktani]

See https://www.graph.cool/forum/t/1-0-different-environments-different-graphcool-yml-files/2105/2?u=nilan as well 🙂