Graphql-engine: Auto-apply Migration problems

[dbhardwa]

I am facing two issues both related to applying migrations to the database. I have initialized Hasura using docker to a pre-existing project and Postgres database. I used the Hasura CLI to initialize the migrations and metadata files within a sub-directory of my project. Here are the two problems I am facing:

(1) Auto-applying migration scripts is not working

• I am following this guide: https://hasura.io/docs/1.0/graphql/core/migrations/advanced/auto-apply-migrations.html#auto-apply-migrations to set up auto-applying upon starting the Docker container.
• I have followed the steps in the 'Applying migrations' section where I have tried both setting the migration and metadata env variables as well as trying -v.
• I am also using the hasura/graphql-engine:v1.3.2.cli-migrations-v2 image.
• I try setting the directory for the env variable in the script to something like this: -e HASURA_GRAPHQL_MIGRATIONS_DIR=/Users/<name>/Desktop/<app>/db/hasura-migrations/migrations.
• I even tried putting the bash script next to the migrations folder and just using ./migrations as the path.
• In both cases, I get the following error in my Docker log: {"kind":"migrations-apply", "info":"directory /Users/<name>/Desktop/<app>/db/hasura-migrations/migrations does not exist, skipping migrations"}}
• I have repeatedly verified that the directory is accurate, but I continue to get the same error saying the directory does not exist, and subsequently skips the migration apply. I am also unsure if this is a Docker or Hasura related error I am making, so I apologize if this error is not caused by Hasura.

(2) hasura migrate status is misleading

• I have been testing the migration scripts to see if they work by regressing my database such that some metadata inconsistency exists when I run hasura console.
• The console correctly gives the inconsistency error, but when I go to verify this using hasura migrate status, the DATABASE STATUS column reads PRESENT for all migration files which should be in theory not present, since they are not in the database.
• As a result of this, running hasura apply simply returns nothing to apply.
• I have actually tried manually running all of the up.sql migration files against the database and everything works, hasura metadata apply then works, and everything is all good, it's just that it won't work automatically through the cli.

Overall, I am facing such challenges in automating the migration/metadata application process and would appreciate any help. Apologies once more if this is not the right place to post these, I had tried reaching out to a representative over the Hasura website, but have gotten little to no response.

[GavinRay97]

I try setting the directory for the env variable in the script to something like this: -e HASURA_GRAPHQL_MIGRATIONS_DIR=/Users/<name>/Desktop/<app>/db/hasura-migrations/migrations

So, the way that Docker containers work is that they are self-contained. The filesystem on a Docker image is not connected to your host filesystem (unless you use a bind-mount).

Inside of your Docker image, /Users/<name>/Desktop/<app>/db/hasura-migrations/migrations does not exist, unless in your Dockerfile you have something like this:

RUN mkdir -p /Users/<name>/Desktop/<app>/db/hasura-migrations/migrations

So what you need to do is put the metadata and migrations folder into the built container image. You can do this with:

FROM hasura/graphql-engine:v1.3.2.cli-migrations-v2
COPY ./metadata /hasura-metadata
COPY ./migrations /hasura-migrations

Assuming that this Dockerfile exists in the same directory as your metadata and migrations folder.

The other thing you can do in local development if you don't want to write a custom Dockerfile, is to mount your migrations and metadata images from your host into the container directories:

version: "3.8"

services:
  postgres:
    image: postgres:12
    restart: always
    ports:
      - 5432:5432
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: postgrespassword

  graphql-engine:
    image: hasura/graphql-engine:v1.3.1.cli-migrations-v2
    ports:
      - 8080:8080
    depends_on:
      - "postgres"
    restart: always
    volumes:
      - ./migrations:/hasura-migrations
      - ./metadata:/hasura-metadata
    environment:
      # your environment variables here

volumes:
  db_data:

The reason why this works is because of this:

    volumes:
      - ./migrations:/hasura-migrations
      - ./metadata:/hasura-metadata

You are mounting the local directory ./migrations into the container directory /hasura-migrations.

Hope that answers your questions and clears things up :+1:

(2) hasura migrate status is misleading

When you run a migration against a Hasura instance, it makes an entry in the DB it's connected to that the migration has been successfully applied on it.

hasura migrate status essentially just checks the list of migrations against what the database has had run on it before.

How are you "regressing the database" to break this? Are you deleting tables without tracking the deletions with migrations?

If you don't track all changes ever made, Hasura's migrations and metadata management become useless because there's a "break" in the history. You need to manually fix it if this happens.

[dbhardwa]

@GavinRay97 Thank you very much for the detailed answer, this definitely helps clear some things up. I have got the auto-migration working, but it looks like there is an issue with the auto-metadata. The container fails to initialize due to this error:

time="2020-10-05T19:46:25Z" level=fatal msg="failed to apply metadata: cannot apply metadata on the database: [parse-failed] key \"tables\" not found ($[1])"

Is there a way I can get more detail as to what this error is suggesting? I can't seem to find a "tables" key in any of the metadata files of my hasura project. I am a little surprised since the hasura metadata apply command works fine, and it was my understanding that the auto-application does the same thing. Am I mistaken?

[GavinRay97]

The metadata/tables.yaml file gets turned into a top-level JSON/YAML key in the combined metadata object called tables that contains the array of objects there.

I would try to run a shell inside of the Hasura container and do ls /hasura-metadata and check that the metadata is actually getting mounted in there.

If it's applying properly from your local system my only guess is that the metadata isn't actually fully present in the container.

Can you post a reproduction repo containing your docker-compose.yaml and hasura folder with migrations + metadata?

[dbhardwa]

It's all good, I managed to figure out the problem (one of the paths was was missing). Thank you very much for your help I have managed to get it working! I am closing this issue.