Compose: Not able to use gcloud credential helpers with docker-compose build --pull

[Stono]

Hey,
We are seeing strange behaviour with compose and default cred helpers when pulling as part of a build.

Both our image, and our base image are in a private gcloud repo.

We do gcloud auth configure-docker to configure docker to give us this config:

{
  "credHelpers": {
    "gcr.io": "gcloud",
    "us.gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud"
  }
}

Which allows us to pull private images, eg:

bash-4.2$ docker pull eu.gcr.io/at-artefacts/platform-base-nodejs
Using default tag: latest
latest: Pulling from at-artefacts/platform-base-nodejs
Digest: sha256:ca558cb49a6fd2e0f178e4016c30638ed722328608bfb87fdc389c76a9c9af03
Status: Image is up to date for eu.gcr.io/at-artefacts/platform-base-nodejs:latest

We have this compose file:

version: '3.2'

services:
  app:
    image: "eu.gcr.io/at-artefacts/platform-base-nodejs:${IMAGE_TAG:-latest}"
    restart: always
    build:
      context: .
      dockerfile: Dockerfile
      args:
        GO_DEPENDENCY_LABEL_BASE_CENTOS: "${GO_DEPENDENCY_LABEL_BASE_CENTOS:-latest}"

And a docker-compose pull doesn't give any output, but it does pull the image.

bash-4.2$ docker-compose pull
Pulling app ... done

But, a build with --pull in it fails:

bash-4.2$ docker-compose build --pull --compress app
Building app
Step 1/8 : FROM eu.gcr.io/at-artefacts/platform-base-centos:latest
ERROR: Service 'app' failed to build: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Any ideas?

[Stono]

Ahhh cross linking https://github.com/docker/compose/issues/5930 although i think it's actually a slightly different problem?

[Stono]

Hello, further confirming the issue, as you can see the docker-compose pull uses the credHelpers:

bash-4.2$ docker-compose --verbose pull
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.utils.config.find_config_file: Trying paths: ['/var/go/.docker/config.json', '/var/go/.dockercfg']
docker.utils.config.find_config_file: Found file at path: /var/go/.docker/config.json
docker.auth.load_config: Found 'credHelpers' section
urllib3.connectionpool._new_conn: Starting new HTTP connection (1): 127.0.0.1
urllib3.connectionpool._make_request: http://127.0.0.1:2375 "GET /v1.25/version HTTP/1.1" 200 557
compose.cli.command.get_client: docker-compose version 1.21.2, build a133471
docker-py version: 3.3.0
CPython version: 3.6.5
OpenSSL version: OpenSSL 1.0.1t  3 May 2016
compose.cli.command.get_client: Docker base_url: http://127.0.0.1:2375
compose.cli.command.get_client: Docker version: Platform={'Name': ''}, Components=[{'Name': 'Engine', 'Version': '18.03.1-ce', 'Details': {'ApiVersion': '1.37', 'Arch': 'amd64', 'BuildTime': '2018-04-26T07:23:58.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9ee9f40', 'GoVersion': 'go1.9.5', 'KernelVersion': '3.10.0-693.5.2.el7.x86_64', 'MinAPIVersion': '1.12', 'Os': 'linux'}}], Version=18.03.1-ce, ApiVersion=1.37, MinAPIVersion=1.12, GitCommit=9ee9f40, GoVersion=go1.9.5, Os=linux, Arch=amd64, KernelVersion=3.10.0-693.5.2.el7.x86_64, BuildTime=2018-04-26T07:23:58.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('dockerplatformbasenodejs_default')
urllib3.connectionpool._make_request: http://127.0.0.1:2375 "GET /v1.25/networks/dockerplatformbasenodejs_default HTTP/1.1" 404 65
Pulling app ...
compose.parallel.feed_queue: Pending: {<Service: app>}
compose.parallel.feed_queue: Starting producer thread for <Service: app>
compose.cli.verbose_proxy.proxy_callable: docker pull <- ('eu.gcr.io/at-artefacts/platform-base-nodejs', tag='latest', stream=True, platform=None)
docker.auth.get_config_header: Looking for auth config
docker.auth.resolve_authconfig: Using credentials store "gcloud"
docker.auth._resolve_authconfig_credstore: Looking for auth entry for 'eu.gcr.io'
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
docker.auth.get_config_header: Found auth config
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://127.0.0.1:2375 "POST /v1.25/images/create?tag=latest&fromImage=eu.gcr.io%2Fat-artefacts%2Fplatform-base-nodejs HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker pull -> <generator object APIClient._stream_helper at 0x7fed5a08e7d8>
Pulling app ... done
compose.parallel.feed_queue: Pending: set()

But build --pull does not:

bash-4.2$ docker-compose --verbose build
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.utils.config.find_config_file: Trying paths: ['/var/go/.docker/config.json', '/var/go/.dockercfg']
docker.utils.config.find_config_file: Found file at path: /var/go/.docker/config.json
docker.auth.load_config: Found 'credHelpers' section
urllib3.connectionpool._new_conn: Starting new HTTP connection (1): 127.0.0.1
urllib3.connectionpool._make_request: http://127.0.0.1:2375 "GET /v1.25/version HTTP/1.1" 200 557
compose.cli.command.get_client: docker-compose version 1.21.2, build a133471
docker-py version: 3.3.0
CPython version: 3.6.5
OpenSSL version: OpenSSL 1.0.1t  3 May 2016
compose.cli.command.get_client: Docker base_url: http://127.0.0.1:2375
compose.cli.command.get_client: Docker version: Platform={'Name': ''}, Components=[{'Name': 'Engine', 'Version': '18.03.1-ce', 'Details': {'ApiVersion': '1.37', 'Arch': 'amd64', 'BuildTime': '2018-04-26T07:23:58.000000000+00:00', 'Experimental': 'false', 'GitCommit': '9ee9f40', 'GoVersion': 'go1.9.5', 'KernelVersion': '3.10.0-693.5.2.el7.x86_64', 'MinAPIVersion': '1.12', 'Os': 'linux'}}], Version=18.03.1-ce, ApiVersion=1.37, MinAPIVersion=1.12, GitCommit=9ee9f40, GoVersion=go1.9.5, Os=linux, Arch=amd64, KernelVersion=3.10.0-693.5.2.el7.x86_64, BuildTime=2018-04-26T07:23:58.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('dockerplatformbasenodejs_default')
urllib3.connectionpool._make_request: http://127.0.0.1:2375 "GET /v1.25/networks/dockerplatformbasenodejs_default HTTP/1.1" 404 65
compose.service.build: Building app
compose.cli.verbose_proxy.proxy_callable: docker build <- (path='/ciagents/pipelines/docker.platform-base-nodejs', tag='eu.gcr.io/at-artefacts/platform-base-nodejs:latest', rm=True, forcerm=False, pull=False, nocache=False, dockerfile='Dockerfile', cache_from=['eu.gcr.io/at-artefacts/platform-base-nodejs:latest'], labels=None, buildargs={'GO_DEPENDENCY_LABEL_BASE_CENTOS': 'latest'}, network_mode=None, target=None, shmsize=None, extra_hosts=None, container_limits={'memory': None}, gzip=False, isolation=None, platform=None)
docker.api.build._set_auth_headers: Looking for auth config
docker.api.build._set_auth_headers: Sending auth config ()
urllib3.connectionpool._make_request: http://127.0.0.1:2375 "POST /v1.25/build?t=eu.gcr.io%2Fat-artefacts%2Fplatform-base-nodejs%3Alatest&q=False&nocache=False&rm=True&forcerm=False&pull=False&dockerfile=Dockerfile&buildargs=%7B%22GO_DEPENDENCY_LABEL_BASE_CENTOS%22%3A+%22latest%22%7D&cachefrom=%5B%22eu.gcr.io%2Fat-artefacts%2Fplatform-base-nodejs%3Alatest%22%5D HTTP/1.1" 200 None
compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object APIClient._stream_helper at 0x7f96c0ec1258>
Step 1/9 : ARG GO_DEPENDENCY_LABEL_BASE_CENTOS
Step 2/9 : FROM eu.gcr.io/at-artefacts/platform-base-centos:$GO_DEPENDENCY_LABEL_BASE_CENTOS
ERROR: compose.cli.main.main: Service 'app' failed to build: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

[Stono]

This did work with docker directly:

bash-4.2$ docker build --build-arg GO_DEPENDENCY_LABEL_BASE_CENTOS=latest  .
Sending build context to Docker daemon   7.68kB
Step 1/9 : ARG GO_DEPENDENCY_LABEL_BASE_CENTOS
Step 2/9 : FROM eu.gcr.io/at-artefacts/platform-base-centos:$GO_DEPENDENCY_LABEL_BASE_CENTOS
latest: Pulling from at-artefacts/platform-base-centos
469cfcc7a4b3: Already exists
c52b01e8ce8d: Already exists
b469a2e5f7af: Already exists
e00b285f48a2: Already exists
a65d2640c291: Already exists
98234839be6c: Already exists
933c944834d9: Already exists
c7ebdce4f576: Already exists
1093f185d600: Already exists
62e1170cfe74: Already exists
Digest: sha256:606a158e08b21b8fd9909a5a9464fd8360973d7921b52df38346f1a9de2d40e7
Status: Downloaded newer image for eu.gcr.io/at-artefacts/platform-base-centos:latest
 ---> 5cd6727c3220
Step 3/9 : USER root
 ---> Running in 42391038dece
Removing intermediate container 42391038dece
 ---> 62c6447fe516
Step 4/9 : LABEL at.nodejs_last_updated=2018-5-4
 ---> Running in 65fd9566cb72
Removing intermediate container 65fd9566cb72
 ---> 3a8bd3c5b3fa
Step 5/9 : RUN curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -
 ---> Running in a116a4c8066f

[Stono]

This does work with docker login -u oauth2accesstoken -p "$(gcloud auth application-default print-access-token)" https://eu.gcr.io

So the problem appears to be entirely isolated to credentials helpers when the FROM image is from a private repository.

Versions:

bash-4.2$ docker -v
Docker version 18.03.1-ce, build 9ee9f40
bash-4.2$ docker-compose -v
docker-compose version 1.21.2, build a133471
bash-4.2$ gcloud -v
Google Cloud SDK 200.0.0
alpha 2018.04.30
beta 2018.04.30
bq 2.0.33
core 2018.04.30
gsutil 4.31

[enanablancaynumeros]

I have the same issue since I upgraded from docker compose 1.19.0 to 1.21.2, but it it failing also for any version over 1.20.

Ubuntu 16.04:

docker -v
Docker version 17.12.1-ce, build 7390fc6

docker-compose -v
docker-compose version 1.20.0, build ca8d3c6

gcloud -v
Google Cloud SDK 176.0.0

[falnyr]

Same issue for:

$ docker -v
Docker version 18.03.1-ce, build 9ee9f40
$ docker-compose -v
docker-compose version 1.19.0, build 9e633ef
$ gcloud -v
Google Cloud SDK 200.0.0
alpha 2018.04.30
app-engine-go 
app-engine-java 1.9.63
app-engine-python 1.9.69
app-engine-python-extras 1.9.69
beta 2018.04.30
bigtable 
bq 2.0.33
cbt 
cloud-datastore-emulator 1.4.1
core 2018.04.30
datalab 20180412
gsutil 4.31
pubsub-emulator 2018.04.30

[sebglon]

work for me for docker-compose 1.21.2 with docker 18.03.1-ce on ubuntu
And with a private gcloud registry

[vladfr]

This is not working for me using the latest 1.23.rc1, getting the same behavior as described by OP:

$ docker -v
Docker version 18.06.1-ce, build e68fc7a
$ docker-compose -v
docker-compose version 1.23.0-rc1, build 320e4819
$ gcloud -v
Google Cloud SDK 219.0.1
bq 2.0.34
core 2018.09.28
gsutil 4.34
kubectl 2018.09.17

[mklinke]

Also same as OP:

$ docker -v
Docker version 18.09.0-ce-beta1, build 78a6bdb
$ docker compose -v
Docker version 18.09.0-ce-beta1, build 78a6bdb
$ gcloud -v
Google Cloud SDK 219.0.1
beta 2018.07.16
bq 2.0.34
core 2018.09.28
docker-credential-gcr
gsutil 4.34

[cbuchacher]

Fixed for me in 1.23.0-rc3 on Linux, but still broken with the same version on Windows.

[lmapii]

same issue on docker-compose version 1.23.1, build b02f130 macos, still seeing it on ubuntu (1.21)

[carlosgeos]

1.23.2 works fine for me (both macOS and Debian)

[cbuchacher]

@lmapii A related fix #6451 is in 1.24.0-rc1.

[lmapii]

@cbuchacher thanks for the mention, i've updated to 1.23.2, seems to be working fine now