Graphql-engine: cli: running cli migrations image as non root user will not have cli-ext installed

[jchonde]

Hello Hasura team,

This is what happen on a fresh dockerized Heroku application with image hasura/graphql-engine:latest.cli-migrations-v2

Here is what I am trying to deploy:

repo

https://github.com/jchonde/obscure-hamlet-63320

env vars:

DATABASE_URL: "postgres://..."
HASURA_GRAPHQL_DATABASE_URL: "postgres://..."
HASURA_GRAPHQL_METADATA_DIR: "/metada"
HASURA_GRAPHQL_MIGRATIONS_DIR: "/migrations"

logs

2020-05-03T13:43:44.171393+00:00 app[web.1]: {"timestamp":"2020-05-03T13:43:44.000+0000","level":"info","type":"startup","detail":{"kind":"migration-apply","info":"applying metadata from /metadata"}}

...

2020-05-03T13:43:44.249753+00:00 app[web.1]: panic: runtime error: invalid memory address or nil pointer dereference
2020-05-03T13:43:44.249758+00:00 app[web.1]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xa7628f]
2020-05-03T13:43:44.249758+00:00 app[web.1]:
2020-05-03T13:43:44.249767+00:00 app[web.1]: goroutine 1 [running]:
2020-05-03T13:43:44.249851+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/plugins.(*Config).findPluginManifestFiles.func1(0xc0003989e0, 0x1e, 0x0, 0x0, 0x1272fa0, 0xc000031bc0, 0x1272fa0, 0xc000031bc0)
2020-05-03T13:43:44.249870+00:00 app[web.1]: /root/graphql-engine/cli/plugins/scanner.go:27 +0x2f
2020-05-03T13:43:44.249915+00:00 app[web.1]: github.com/spf13/afero.Walk(0x129cdc0, 0x1b70d18, 0xc0003989e0, 0x1e, 0xc0005ec440, 0xc0003989e0, 0x1e)
2020-05-03T13:43:44.249926+00:00 app[web.1]: /root/workspace/go/pkg/mod/github.com/spf13/[email protected]/path.go:103 +0x8b
2020-05-03T13:43:44.249931+00:00 app[web.1]: github.com/spf13/afero.Afero.Walk(...)
2020-05-03T13:43:44.249939+00:00 app[web.1]: /root/workspace/go/pkg/mod/github.com/spf13/[email protected]/path.go:97
2020-05-03T13:43:44.250000+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/plugins.(*Config).findPluginManifestFiles(0xc000302c90, 0xc0003989e0, 0x1e, 0xc0003989e0, 0x1e, 0x0, 0x0, 0x0)
2020-05-03T13:43:44.250019+00:00 app[web.1]: /root/graphql-engine/cli/plugins/scanner.go:26 +0x8b
2020-05-03T13:43:44.250059+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/plugins.(*Config).LoadPluginByName(0xc000302c90, 0x1078cfd, 0x7, 0x2, 0xc0005ec5e0, 0x40c2d4)
2020-05-03T13:43:44.250077+00:00 app[web.1]: /root/graphql-engine/cli/plugins/scanner.go:57 +0xfe
2020-05-03T13:43:44.250120+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/plugins.(*Config).GetPlugin(0xc000302c90, 0x1078cfd, 0x7, 0x0, 0x0, 0xc000522d20, 0x0, 0x0, 0x0, 0x0, ...)
2020-05-03T13:43:44.250140+00:00 app[web.1]: /root/graphql-engine/cli/plugins/plugins.go:104 +0xc7
2020-05-03T13:43:44.250257+00:00 app[web.1]: github.com/hasura/graphql-engine/cli.ExecutionContext.InstallPlugin(0x7fffcacdad0f, 0xa, 0xc0003b73b0, 0x24, 0xc0003b7c80, 0x24, 0xc000233360, 0xc000113180, 0xc000038024, 0x13, ...)
2020-05-03T13:43:44.250270+00:00 app[web.1]: /root/graphql-engine/cli/cli.go:663 +0x84
2020-05-03T13:43:44.250296+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/metadata/actions.(*ActionConfig).Build(0xc000040300, 0xc00000c5a0, 0x0, 0x0)
2020-05-03T13:43:44.250315+00:00 app[web.1]: /root/graphql-engine/cli/metadata/actions/actions.go:267 +0x6a
2020-05-03T13:43:44.250369+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/migrate/database/hasuradb.(*HasuraDB).BuildMetadata(0xc000113d50, 0x203000, 0x203000, 0x203000, 0xc0002a0270, 0xe)
2020-05-03T13:43:44.250390+00:00 app[web.1]: /root/graphql-engine/cli/migrate/database/hasuradb/metadata.go:152 +0xa8
2020-05-03T13:43:44.250410+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/migrate/database/hasuradb.(*HasuraDB).ApplyMetadata(0xc000113d50, 0x5, 0x10763db)
2020-05-03T13:43:44.250428+00:00 app[web.1]: /root/graphql-engine/cli/migrate/database/hasuradb/metadata.go:161 +0x43
2020-05-03T13:43:44.250446+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/migrate.(*Migrate).ApplyMetadata(...)
2020-05-03T13:43:44.250449+00:00 app[web.1]: /root/graphql-engine/cli/migrate/migrate.go:360
2020-05-03T13:43:44.250473+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/commands.executeMetadata(0x10762c8, 0x5, 0xc0001213f0, 0xc0000ee780, 0x0, 0x1)
2020-05-03T13:43:44.250553+00:00 app[web.1]: /root/graphql-engine/cli/commands/metadata.go:75 +0xfa
2020-05-03T13:43:44.250591+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/commands.(*MetadataApplyOptions).Run(0xc0000bfae0, 0x0, 0x0)
2020-05-03T13:43:44.250592+00:00 app[web.1]: /root/graphql-engine/cli/commands/metadata_apply.go:81 +0x109
2020-05-03T13:43:44.250609+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/commands.newMetadataApplyCmd.func1(0xc000179180, 0x1b70d18, 0x0, 0x0, 0x0, 0x0)
2020-05-03T13:43:44.250620+00:00 app[web.1]: /root/graphql-engine/cli/commands/metadata_apply.go:41 +0x75
2020-05-03T13:43:44.250654+00:00 app[web.1]: github.com/spf13/cobra.(*Command).execute(0xc000179180, 0x1b70d18, 0x0, 0x0, 0xc000179180, 0x1b70d18)
2020-05-03T13:43:44.250665+00:00 app[web.1]: /root/workspace/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x460
2020-05-03T13:43:44.250719+00:00 app[web.1]: github.com/spf13/cobra.(*Command).ExecuteC(0x19c1640, 0xc0001704a0, 0xc0000a8150, 0x3)
2020-05-03T13:43:44.250725+00:00 app[web.1]: /root/workspace/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
2020-05-03T13:43:44.250751+00:00 app[web.1]: github.com/hasura/graphql-engine/cli/commands.Execute(0xc0000a4058, 0x0)
2020-05-03T13:43:44.250763+00:00 app[web.1]: /root/graphql-engine/cli/commands/root.go:118 +0x13a
2020-05-03T13:43:44.250769+00:00 app[web.1]: main.main()
2020-05-03T13:43:44.250782+00:00 app[web.1]: /root/graphql-engine/cli/cmd/hasura/hasura.go:11 +0x22
2020-05-03T13:43:44.321024+00:00 heroku[web.1]: State changed from starting to crashed

Thanks guys. Keep up the awesome work.

[scriptonist]

Hey @jchonde, this was caused because a plugin,cli-ext which is required to read metadata files was not getting installed properly.

This happened because,
The Dockerfile which builts the cli-migrations-v2 image has a step which will install the cli-ext plugin.

https://github.com/hasura/graphql-engine/blob/8a58ef8b0cca5e872385ddc20830bcfe9b809b4c/scripts/cli-migrations/v2/Dockerfile#L19

The catch here is that It's installed in /home/root/.hasura/plugins

Heroku, however, runs the container as a non root user, so the CLI when trying to find the plugin will not find it under /home/{heroku_user}/.hasura/plugins.

The CLI is intelligent enough to download plugins which are not present, But this is not enabled in cli-migrations-v2 image by default.

As a workaround, what you can do is set the following environment variable.

HASURA_GRAPHQL_CLI_ENVIRONMENT=default

This will enable downloading plugins which are not present and everything should work just fine 😄

As a side note, the contents of metadata/actions.yaml in the repository seems to be outdated. it should have the following contents as generated by the latest version of CLI

actions: []
custom_types:
  enums: []
  input_objects: []
  objects: []
  scalars: []

[jchonde]

Thanks @scriptonist really appreciated!

[adav]

The workaround sorted me out as well! Many thanks indeed :)

[adav]

Is this related?:

2020-05-07T22:32:12.388633+00:00 app[web.1]: {"level":"info","msg":"Installing plugin cli-ext...","time":"2020-05-07T22:32:12Z"}
2020-05-07T22:32:13.263349+00:00 app[web.1]: {"level":"info","msg":"unable to install cli-ext plugin. execute the following commands to continue:\n\n  hasura plugins install cli-ext\n","time":"2020-05-07T22:32:13Z"}
2020-05-07T22:32:13.263540+00:00 app[web.1]: time="2020-05-07T22:32:13Z" level=fatal msg="failed to apply metadata: cannot apply metadata on the database: cannot build actions from metadata: cannot install plugin cli-ext: install failed: failed to unpack into staging dir: failed to unpack the plugin archive: checksum does not match, want: 25df7059911ff3e939f2161c0b1667526eba3a3171ba29abe959b86a8a237504, got 446327911f5c0688ba558ed3b979eebf3c13cf82fa73988cab891b90ed81d701"

[hoeggi]

I'm also running into a similar issue after trying the workaround from above:

{"level":"info","msg":"plugin installed","name":"cli-ext","time":"2020-05-13T10:47:57Z"}
time="2020-05-13T10:47:57Z" level=fatal msg="failed to apply metadata: cannot apply metadata on the database: cannot build actions from metadata: error in converting sdl to metadata: exit status 127: /root/.hasura/plugins/bin/hasura-cli_ext: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory\n"

[scriptonist]

@adav @hoeggi Just to confirm, this is happening when running on Heroku right?

[hoeggi]

I'm having the error in openshift. By default container in openshift are also not permitted to run as root but get assigned a random user (id).

[scriptonist]

Got it @hoeggi, this is very helpful. Will look into it.

[hoeggi]

I also had issues with the old v1 migrations in openshift but i managed to work around them by updaing the entrypoint script. In case this helps, see: https://github.com/hasura/graphql-engine/issues/3824

[adav]

@adav @hoeggi Just to confirm, this is happening when running on Heroku right?

@scriptonist Yes, it was on Heroku for me. The following cli-ext error seems not to be an issue now though. Once I had double checked all the settings and restarted the dyno a couple times, it seemed to correct itself.

[scriptonist]

Thanks for the input @adav, this will really help.

[scriptonist]

@hoeggi we have some pointers regarding this in the following comment https://github.com/hasura/graphql-engine/issues/4105#issuecomment-609444008

[hoeggi]

Not sure if it is directly related. In https://github.com/hasura/graphql-engine/issues/4105#issuecomment-609444008 it seems to be the missing libstdc++ that's causing the problem (and using and old or no hasura image). Since v2 libsdc++ is already installed at image build time (https://github.com/hasura/graphql-engine/blob/master/scripts/cli-migrations/v2/Dockerfile#L9).

Further down it is mentioned that it might be some missing glibc stuff, but if that was the case it would also fail if run as root.

[hoeggi]

This is interesting, dpkg-deb -x libstdc++6*.deb should install the binary to /usr/lib/x86_64-linux-gnu as far as i can tell, but it's actually missing in my image.

[joshuarobs]

Anyone manage to make migrations work with Heroku?

I have my server set to HASURA_GRAPHQL_CLI_ENVIRONMENT=default and it copied the migrations and metadata folder properly but when I go to the server, that was freshly made, there are no tables imported from the migration.

Edit:

Managed to make it work with this repo that I forked from OP's test repo: https://github.com/joshuarobs/obscure-hamlet-63320

[loiclegoff]

Hello @scriptonist .
I think I have the same error on windows when I set up this project : https://github.com/hasura/hasura-actions-examples/tree/master/auth

Is it normal?
Is your image up to date? hasura/graphql-engine:v1.2

[SurgeS]

Hi guys, is this something that's currently being worked on or should I try to make it work by forking the repo mentioned above?

Although, my error is slightly different:

{"level":"info","msg":"Checking for update... ","time":"2020-05-28T09:34:02Z"}
{"level":"info","msg":"hasura cli is up to date","time":"2020-05-28T09:34:03Z","version":"1.2.1"}
{"level":"info","msg":"Applying migrations...","time":"2020-05-28T09:34:06Z"}
{"level":"info","msg":"nothing to apply","time":"2020-05-28T09:34:08Z"}
{"level":"info","msg":"Applying metadata...","time":"2020-05-28T09:34:09Z"}
{"level":"info","msg":"Installing plugin cli-ext...","time":"2020-05-28T09:34:10Z"}
{"level":"info","msg":"plugin installed","name":"cli-ext","time":"2020-05-28T09:34:12Z"}
time="2020-05-28T09:34:12Z" level=fatal msg="failed to apply metadata: cannot apply metadata on the database: cannot build actions from metadata: error in converting sdl to metadata: exit status 4: "

Looks like the cli-ext was installed just fine but then I got an exit status 4 error.

[scriptonist]

Hey @SurgeS, this is currently being worked on, will update here once a fix lands.

[hoeggi]

In case someone needs this, here is how I got it working in openshift without running as root:

Dockerfile: https://gist.github.com/hoeggi/9c771249fd9d22f1421c0ddaa473d7c0#file-dockerfile
docker-entrypoint-sh: https://gist.github.com/hoeggi/9c771249fd9d22f1421c0ddaa473d7c0#file-docker-entrypoint-sh

Since openshift by default assigns a random user/uid when creating the container RUN chmod -R 777 was the only way I could find to make the cli-ext reliably execute.
The other problem was in docker-entrypoint-sh with calling cp -a (see here for an explanation: https://github.com/hasura/graphql-engine/issues/3824 )

[scriptonist]

Hey, this was a similar issue https://github.com/hasura/graphql-engine/issues/4953, can you folks see if this is related? @SurgeS @loiclegoff

[SurgeS]

@scriptonist that error certainly looks the same but the command there works fine for me and I don't have any NODE_OPTIONS env variable setup either :confused:

[oliverpool]

@hoeggi I had the same issue.

FATA[0000] failed to export metadata: cannot export metadata from server: cannot export actions from metadata: error in converting metadata to sdl: exit status 127: /root/.hasura/plugins/bin/hasura-cli_ext: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory 

The Dockerfile you pointed to is in the v2 folder, which is building the *-v2 Docker image (using v1.2.2.cli-migrations-v2 in my case solved the issue !)

[ighormartins]

The workaround works fine.
However, because it's downloading plugins, it now uses a lot of memory.
I'm running Hasura on Cloud Run, and I actually had to upgrade the memory to 512Mb :(

[jflambert]

@scriptonist I understand an official solution is being worked on, but I have no idea how to use the workaround in the meantime. I'm trying to add hasura-cli to an all-purpose docker image which I use in gitlab for build/test scripts. As you can see, I download the cli directly from gitlab, then try to install the plugin. Seems like it works at that point.

# use rabbitmq as a base image, since the apk is not available in alpine as of 3.12
FROM rabbitmq:3-alpine as rabbitmq

ENV HASURA_GRAPHQL_CLI_ENVIRONMENT=default

# Fetch additional tools
RUN apk --update add go postgresql-client curl gcc libc-dev
RUN curl -L# -f -o /bin/hasura-cli https://github.com/hasura/graphql-engine/releases/download/v1.3.0/cli-hasura-linux-amd64 && chmod +x /bin/hasura-cli

# test various tools
RUN go version && rabbitmq-diagnostics version && pg_isready -V && hasura-cli plugins install cli-ext --skip-update-check && hasura-cli plugins list --skip-update-check

output of docker build . --tag test_cli

{"level":"info","msg":"Installing plugin \"cli-ext\"...","time":"2020-08-10T19:24:15Z"}
{"level":"info","msg":"plugin installed","name":"cli-ext","time":"2020-08-10T19:24:17Z"}
{"level":"info","msg":"Updating plugin index...","time":"2020-08-10T19:24:17Z"}
{"level":"info","msg":"Fetching plugins list...","time":"2020-08-10T19:24:18Z"}
NAME     DESCRIPTION                VERSION       INSTALLED
cli-ext  Hasura CLI extension       v1.3.0        yes
pro      CLI plugin for Hasura Pro  v1.2.1-pro.1  no

but then when I try to run hasura-cli inside the image: docker run --net=host --rm -it test_cli /bin/bash

# hasura-cli plugins list --skip-update-check
NAME     DESCRIPTION                VERSION       INSTALLED
cli-ext  Hasura CLI extension       v1.3.0        no
pro      CLI plugin for Hasura Pro  v1.2.1-pro.1  no

# hasura-cli md apply
INFO plugin installed                              name=cli-ext
FATA[0002] failed to apply metadata: cannot apply metadata on the database: cannot build actions from metadata: error in converting sdl to metadata: fork/exec /var/lib/rabbitmq/.hasura/plugins/bin/hasura-cli_ext: no such file or directory:

# ls -al /var/lib/rabbitmq/.hasura/plugins/bin/hasura-cli_ext
lrwxrwxrwx    1 root     root            75 Aug 10 19:27 /var/lib/rabbitmq/.hasura/plugins/bin/hasura-cli_ext -> /var/lib/rabbitmq/.hasura/plugins/store/cli-ext/v1.3.0/cli-ext-hasura-linux
# ls -al /var/lib/rabbitmq/.hasura/plugins/store/cli-ext/v1.3.0/cli-ext-hasura-linux
-rwxr-xr-x    1 root     root      64458048 Aug 10 19:27 /var/lib/rabbitmq/.hasura/plugins/store/cli-ext/v1.3.0/cli-ext-hasura-linux

[collinbachi]

The catch here is that It's installed in /home/root/.hasura/plugins

Heroku, however, runs the container as a non root user, so the CLI when trying to find the plugin will not find it under /home/{heroku_user}/.hasura/plugins.

The CLI is intelligent enough to download plugins which are not present, But this is not enabled in cli-migrations-v2 image by default.

As a workaround, what you can do is set the following environment variable.

HASURA_GRAPHQL_CLI_ENVIRONMENT=default
This will enable downloading plugins which are not present and everything should work just fine 😄

But then this will require an internet connection, right? It's common to see airgapped OpenShift installations, in which case the cli-ext needs to be downloaded at image build time.