Compose: Docker Compose fails to pull from private registry when using credential helper

[ghost]

Docker seems to authenticate and pull images fine, but docker-compose fails. The docker container URLs below have been redacted. This issue seems to be present in versions 1.13.0 and 1.14.0rc1.

$ docker pull us.gcr.io//mysql:latest
latest: Pulling from /mysql
10a267c67f42: Already exists
c2dcc7bb2a88: Pull complete
17e7a0445698: Pull complete
9a61839a176f: Pull complete
a1033d2f1825: Pull complete
0d6792140dcc: Pull complete
cd3adf03d6e6: Pull complete
d79d216fd92b: Pull complete
b3c25bdeb4f4: Pull complete
02556e8f331f: Pull complete
4bed508a9e77: Pull complete
47cc4a3f7009: Pull complete
Digest: sha256:b899cae2e17184411adde639bd61ba808ada6d01514213c5569862c88a8d5649
Status: Downloaded newer image for us.gcr.io//mysql:latest

$ docker-compose pull
Pulling mysql (us.gcr.io//mysql:latest)...
ERROR: repository us.gcr.io//mysql not found: does not exist or no pull access



md5-13cab4dce5a08c2d24115b2873a49815



$ docker-compose --verbose pull
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('us.gcr.io//mysql:latest')
compose.service.pull: Pulling mysql (us.gcr.io//mysql:latest)...
compose.cli.verbose_proxy.proxy_callable: docker pull <- ('us.gcr.io//mysql', tag='latest', stream=True)
docker.auth.get_config_header: Looking for auth config
docker.auth.resolve_authconfig: Looking for auth entry for 'us.gcr.io'
docker.auth.resolve_authconfig: No entry found
docker.auth.get_config_header: No auth config found
ERROR: compose.cli.errors.log_api_error: repository us.gcr.io//mysql not found: does not exist or no pull access



md5-13cab4dce5a08c2d24115b2873a49815



$ cat ~/.docker/config.json
{
    "auths": {},
    "credHelpers": {
        "appengine.gcr.io": "gcr",
        "asia.gcr.io": "gcr",
        "b.gcr.io": "gcr",
        "beta.gcr.io": "gcr",
        "bucket.gcr.io": "gcr",
        "eu.gcr.io": "gcr",
        "gcr.io": "gcr",
        "gcr.kubernetes.io": "gcr",
        "us.gcr.io": "gcr",
        "mirror.gcr.io": "gcr"
    }
}

[shin-]

What does GCR stand for? It seems we might be missing support for their auth method.

[ghost]

GCR is Google Cloud Registry. I authenticate using this credential helper: https://github.com/GoogleCloudPlatform/docker-credential-gcr

[shin-]

Thanks! We do have an implementation for credential helpers support, but it seems it is outdated. Can you follow the instructions at the end of the README and check that it works?

[ghost]

I've updated my config file to resemble the old style:

$ cat ~/.docker/config.json
{
    "auths": {
        "https://gcr.io": {},
        "https://us.gcr.io": {}
    },
    "credsStore": "gcr"
}

After doing that docker-compose pull and docker pull work, so this functions as a workaround and confirms that only the old credential helper method is supported.

[shin-]

This will be fixed in the upcoming 1.15 release.

[suvash]

@shin- @ilyaBV was this fixed in the 1.15 release ? I can confirm that it's still broken in 1.16 (and has to be fixed using the hack @ilyaBV mentioned above). would like to re-open this.

Docker version 17.06.2-ce, build cec0b72
docker-compose version 1.16.1, build 6d1ac21

[yachi]

works for me on docker-compose 1.16.1 after running

 docker-credential-gcr configure-docker

[suvash]

@yachi can you paste your config ?
$ cat ~/.docker/config.json

[jak]

The issue I've had is Docker for Mac re-installing compose 0.14.0 in /usr/local/bin, so I've had to download the latest docker-compose and put it in a different path so it doesn't re-occur.

[yachi]

@suvash
gcloud sdk 170.0.1
config.json: https://gist.github.com/yachi/02a9ca4adf576f546b4188fb859507b6

[ernsheong]

I've ran docker-credential-gcr configure-docker with docker version 17.03.2-ce but I am still getting the error after running docker-compose 1.16.1:

repository gcr.io/$PROJECT_ID/$IMAGE_NAME not found: does not exist or no pull access

Running docker pull gcr.io/$PROJECT_ID/$IMAGE_NAME works just fine.

Contents of ~/.docker/config.json (default, not edited):

}
        "auths": {},
        "credHelpers": {
                "appengine.gcr.io": "gcr",
                "asia.gcr.io": "gcr",
                "eu.gcr.io": "gcr",
                "gcr.io": "gcr",
                "gcr.kubernetes.io": "gcr",
                "us.gcr.io": "gcr"
        }
}         

I have to add that this docker-compose runs as an alias with the following command as found in https://cloud.google.com/community/tutorials/docker-compose-on-container-optimized-os:

echo alias docker-compose="'"'docker run \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v "$PWD:/rootfs/$PWD" \
    -w="/rootfs/$PWD" \
    docker/compose:1.16.1'"'" >> ~/.bashrc

Any idea what is missing here would be greatly appreciated.

[ghost]

I managed to have it working by adding the following volume:

-v ~/.docker:/root/.docker

I believe there is an issue for docker-compose not seeing the config files from the host.

docker login
docker run -v /var/run/docker.sock:/var/run/docker.sock -v $PWD:/rootfs/$PWD -w=/rootfs
/$PWD -v ~/.docker:/root/.docker docker/compose:1.17.1 pull

[russell]

When running with

$ docker --version
Docker version 17.12.0-ce, build c97c6d6
$ docker-compose --version
docker-compose version 1.18.0, build 8dd22a9

I had to run docker-credential-gcr configure-docker then add

  "credsStore": "gcr",

to ~/.docker/config.json so that docker-compose build would work if the base image needed to be fetched from GCR

[gastlich]

gcloud docker --authorize-only

will create short-lived access for ['gcr.io', 'us.gcr.io', 'eu.gcr.io', 'asia.gcr.io', 'staging-k8s.gcr.io', 'k8s.gcr.io']

and will make your docker-compose working

It's quite funny, because this command is run if you try to pull docker image through gcloud docker -- pull, and makes your docker-compose working magically 😄 Spent some time on figuring this out...

[yanc0]

With the newest gcloud, gcloud docker --authorize-only is deprecated.

Use gcloud auth configure-docker instead.

WARNING: `gcloud docker` will not be supported for Docker client versions above 18.03.
Please use `gcloud auth configure-docker` to configure `docker` to use `gcloud` as a 
credential helper

[jak]

That method doesn’t work if docker-compose is invoked by another process, such as docker-sync. When the short lived access is removed there will be no workaround.

[theist]

I've been using docker compose with GCR pulls for a while in CI nodes. What I had to do to get it working was:

At image creation ensure docker-credential-gcr is installed and the project is set, Activate a service account and perform a docker-credential-gcr configure-docker I needed to use a generated service account json because I was not able to complete the procedure with the default credentials.

Once al configured I had to do a docker login and then docker-compose was able to pull from a private GCR repo

gcloud components install docker-credential-gcr --quiet
gcloud config set project my_project
gcloud auth activate-service-account --key-file=/path/to/custom/service-account-key.json
docker-credential-gcr configure-docker

# Once configured and before starting any operation involving pulls

/usr/bin/docker login -u _json_key -p \"$(cat /path/to/custom/service-account-key.json)\" https://eu.gcr.io

The last step must be repeated before attempting any interaction with GCR, once logged in, it will work until the server is restarted.

This worked for me, not sure if it is the best option.

[kilianc]

I can confirm this is still an issue today with 18.05.0-ce and 1.21.2

[shin-]

@kilianc you are probably running into #5930 which is getting fixed in 1.22. You can test the release candidate to confirm.

[kilianc]

@shin- it doesn't look like I am. I am simply getting a permission denied exactly like described here and I am on 1.22.0

[shin-]

Create a new ticket with all the deets please!