Compose: docker-compose fails to pull image from private registry

[cnkuyan]

Hi,
docker-compose fails to pull the image, whereas plain `docker pull o_auth:${O_AUTH_VERSION}' is successful. ( Upon successfully logging in to the registry , with 'docker login')

I'm using a vagrant plugin (https://github.com/leighmcculloch/vagrant-docker-compose ) to do this.

Host OS: ubuntu 16.04

~/dev/docker$ docker-compose --version
docker-compose version 1.11.2, build dfed245
~/dev/docker$ docker --version
Docker version 17.04.0-ce, build 4845c56
~/dev/docker$ vagrant --version
Vagrant 1.9.4

~/dev/docker$ cat Vagrantfile
....
 config.vm.box = "ubuntu/trusty64"
  config.vm.provision :docker
  config.vm.provision :docker_compose , yml: "/vagrant/mongo_replica/docker-compose.yml", run: "always"
  config.vm.provision :docker_compose , yml: "/vagrant/docker-compose.yml",
  options: " --verbose",
  run: "always",
..

~/dev/docker$ cat docker-compose.yml
version: '2'
services:
  o_redis:
    hostname: o_redis
    image: redis:latest
    ports:
    - "6379:6379"
    networks:
      mongoreplica_app_net:
        ipv4_address:  172.18.0.9

  o_auth:
    hostname: o_auth
    image: MYPRIVATE_REG:5011/o_auth:${O_AUTH_VERSION}
    environment:
    - HOST_IP=172.17.0.1
    - ETCD2_PROT=http
    - O_AUTH_PORT_6379_TCP_ADDR=172.18.0.9
    - VERSION=${O_AUTH_VERSION}
    links:
        - o_redis
    ports:
      - "8081:8081"
    networks:
      mongoreplica_app_net:
          ipv4_address: 172.18.0.10

networks:
  mongoreplica_app_net:
      external: true

---

The output of the docker-compose

==> default: compose.config.config.find: Using configuration files: /vagrant/docker-compose.yml
==> default: docker.auth.find_config_file: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
==> default: docker.auth.find_config_file: No config file found
==> default: compose.cli.command.get_client: docker-compose version 1.11.2, build dfed245
==> default: docker-py version: 2.1.0
==> default: CPython version: 2.7.13
==> default: OpenSSL version: OpenSSL 1.0.1t  3 May 2016
==> default: compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
==> default: compose.cli.command.get_client: Docker version: KernelVersion=3.13.0-117-generic, Arch=amd64, BuildTime=2017-05-04T22:06:06.693142599+00:00, ApiVersion=1.29, Versi
on=17.05.0-ce, MinAPIVersion=1.12, GitCommit=89658be, Os=linux, GoVersion=go1.7.5
==> default: compose.cli.verbose_proxy.proxy_callable: docker info <- ()
==> default: compose.cli.verbose_proxy.proxy_callable: docker info -> {u'Architecture': u'x86_64',
==> default:  u'BridgeNfIp6tables': True,
==> default:  u'BridgeNfIptables': True,
==> default:  u'CPUSet': True,
==> default:  u'CPUShares': True,
==> default:  u'CgroupDriver': u'cgroupfs',
==> default:  u'ClusterAdvertise': u'',
==> default:  u'ClusterStore': u'',
==> default:  u'ContainerdCommit': {u'Expected': u'9048e5e50717ea4497b757314bad98ea3763c145',
==> default:                        u'ID': u'9048e5e50717ea4497b757314bad98ea3763c145'},
==> default: ...
==> default: compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('mongoreplica_app_net')
==> default: compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {u'Attachable': False,
==> default:  u'Containers': {u'25734b8f2d6160709935b0957cca981a41f924474bd6541c9080a69184b12923': {u'EndpointID': u'f654c0c7301a97c30e7b04b4c6b4b2403e89dd6dbc729ed329ddb006bb8
e9cb8',
==> default:                                                                                        u'IPv4Address': u'172.18.0.3/24',
==> default:                                                                                        u'IPv6Address': u'',
==> default:                                                                                        u'MacAddress': u'02:42:ac:12:00:03',
==> default:                                                                                        u'Name': u'mongoreplica_mongo2_1'},
==> default:                  u'9c7b4f568eace9605186d4b4a4d535681eae0f105f73ac3840d8dd792fb10a13': {u'EndpointID': u'80abf1b8035e6bb982598e2786fa7e24e6a940c762353d3908f946b4d2b
54ac7',
==> default:                                                                                        u'IPv4Address': u'172.18.0.2/24',
==> default:                                                                                        u'IPv6Address': u'',
==> default:                                                                                        u'MacAddress': u'02:42:ac:12:00:02',
==> default: ...
==> default: compose.network.ensure: Network mongoreplica_app_net declared as external. No new network will be created.
==> default: compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={u'label': [u'com.docker.compose.project=vagrant', u'com.docker.compose.oneoff=F
alse']})
==> default: compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
==> default: compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=vagrant', u'com.docker.compose.service=o
_redis', u'com.docker.compose.oneoff=False']})
==> default: compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
==> default: compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=vagrant', u'com.docker.compose.service=o
_auth', u'com.docker.compose.oneoff=False']})
==> default: compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
==> default: compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('redis:latest')
==> default: compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {u'Architecture': u'amd64',
==> default:  u'Author': u'',
==> default:  u'Comment': u'',
==> default:  u'Config': {u'ArgsEscaped': True,
==> default:              u'AttachStderr': False,
==> default:              u'AttachStdin': False,
==> default:              u'AttachStdout': False,
==> default:              u'Cmd': [u'redis-server'],
==> default:              u'Domainname': u'',
==> default:              u'Entrypoint': [u'docker-entrypoint.sh'],
==> default: ...
==> default: compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('MYPRIVATE_REG:5011/o_auth:0.6.0-5-g29276c8')
==> default: compose.service.pull: Pulling o_auth (MYPRIVATE_REG:5011/o_auth:0.6.0-5-g29276c8)...
==> default: compose.cli.verbose_proxy.proxy_callable: docker pull <- (u'MYPRIVATE_REG:5011/o_auth', tag=u'0.6.0-5-g29276c8', stream=True)
==> default: docker.auth.get_config_header: Looking for auth config
==> default: docker.auth.get_config_header: No auth config in memory - loading from filesystem
==> default: docker.auth.find_config_file: Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
==> default: docker.auth.find_config_file: No config file found
==> default: docker.auth.resolve_authconfig: Looking for auth entry for u'MYPRIVATE_REG:5021'
==> default: docker.auth.resolve_authconfig: No entry found
==> default: docker.auth.get_config_header: No auth config found
==> default: compose.cli.errors.log_api_error: Get https://MYPRIVATE_REG:5011/v1/_ping: x5MYPRIVATE_REG09: certificate signed by unknown authority

docker.auth.resolve_authconfig: No entry found
docker.auth.get_config_header: No auth config found
compose.cli.errors.log_api_error: Get https://MYPRIVATE_REG:5011/v1/_ping: x509: certificate signed by unknown authority

[shin-]

You need to run docker login in your VM first I imagine. Otherwise docker-compose won't have access to your registry credentials, and it obviously can't pull.

[daveisfera]

I'm seeing this in Fedora 26 (it worked in Fedora 25) and I have definitely logged in:
https://bugzilla.redhat.com/show_bug.cgi?id=1471319

[TomasTomecek]

Can this one be related? https://github.com/docker/docker-py/issues/1683

[shin-]

@TomasTomecek I don't think so - that docker-py issue is limited to pulls that occur during a build, where the auth payload is different from doing a straight-up pull.

[daveisfera]

@shin- Has this been fixed? If not, is there another issue I can follow to get things working on Fedora 26?

[shin-]

@daveisfera Given the context, my understanding is that this is a Fedora issue or user issue, since no other report of a similar bug has surfaced. If you'd like to share your Compose file and the output of docker-compose --verbose pull, I'm happy to do a quick spot check with you for possible configuration issues.

[daveisfera]

I opened a bugzilla for Fedora but the response was to get the issue fixed upstream. Every .yml file I've tried has had the problem and it can be something as simple as this (NOTE: I replaced the actual repo and container names to protect any potential sensitivities with what I was working):

version: '2'

services:
    my_container:
        container_name: my_container
        image: my_repo/my_container:dev

And here's the example output:

docker-compose --verbose -f test.yml pull
compose.config.config.find: Using configuration files: ./test.yml
docker.auth.find_config_file: Trying paths: ['/home/dave-nm/.docker/config.json', '/home/dave-nm/.dockercfg']
docker.auth.find_config_file: Found file at path: /home/dave-nm/.docker/config.json
docker.auth.load_config: Found 'auths' section
docker.auth.parse_auth: Found entry (registry='https://index.docker.io/v1/', username='davenm')
docker.auth.parse_auth: Found entry (registry='https://registry.fedoraproject.org/v1/', username='davenm')
compose.cli.command.get_client: docker-compose version 1.14.0, build c7bdf9e
docker-py version: 2.4.2
CPython version: 3.6.2
OpenSSL version: OpenSSL 1.1.0f-fips  25 May 2017
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: Version=1.13.1, ApiVersion=1.26, MinAPIVersion=1.12, GitCommit=27e468e/1.13.1, GoVersion=go1.8.1, Os=linux, Arch=amd64, KernelV ersion=4.11.11-300.fc26.x86_64, BuildTime=2017-06-19T19:26:21.482459655+00:00, PkgVersion=<unknown>
compose.service.pull: Pulling my_container (my_repo/my_container:dev)...
compose.cli.verbose_proxy.proxy_callable: docker pull <- ('my_repo/my_container', tag='dev', stream=True)
docker.auth.get_config_header: Looking for auth config
docker.auth.resolve_authconfig: Looking for auth entry for 'docker.io'
docker.auth.resolve_authconfig: Found 'https://index.docker.io/v1/'
docker.auth.get_config_header: Found auth config
compose.cli.verbose_proxy.proxy_callable: docker pull -> <generator object APIClient._stream_helper at 0x7f2f366f2830>
Trying to pull repository registry.fedoraproject.org/my_repo/my_container ...
Trying to pull repository registry.access.redhat.com/my_repo/my_container ...
Trying to pull repository docker.io/my_repo/my_container ...
ERROR: compose.cli.main.main: repository docker.io/my_repo/my_container not found: does not exist or no pull access

[TomasTomecek]

I'm sorry but I cannot reproduce. I created a private repo at Docker Hub:

$ docker logout
Removing login credentials for https://index.docker.io/v1/

$ cat docker-compoes.yml
version: '2'
services:
  x:
    image: tomastomecek/secret-repo

$ docker-compose pull
Pulling x (tomastomecek/secret-repo:latest)...
Trying to pull repository docker.io/tomastomecek/secret-repo ...
ERROR: repository docker.io/tomastomecek/secret-repo not found: does not exist or no pull access

Correct, no credentials provided yet.

$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: tomastomecek
Password:
Login Succeeded

$ docker-compose pull
Pulling x (tomastomecek/secret-repo:latest)...
Trying to pull repository docker.io/tomastomecek/secret-repo ...
sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac: Pulling from docker.io/tomastomecek/secret-repo
Digest: sha256:8573b4a813d7b90ef3876c6bec33db1272c02f0f90c406b25a5f9729169548ac
Status: Downloaded newer image for docker.io/tomastomecek/secret-repo:latest

These are the packages I'm using:

docker-1.13.1-20.git27e468e.fc27.x86_64
docker-compose-1.15.0-1.fc27.noarch

Since docker-compose is using docker-py, feel free to try if the pull works with docker-py:

$ python
In [1]: import docker

In [2]: c = docker.APIClient()

In [3]: c.pull("namespace/image-name")

[CJPoll]

Hmmm... I'm getting the same issue on my Mac. Sad to see noone else has reproduced.

[robinbryce]

We have 4 engineers on macs. 3 see this problem. 1 does not. The difference appears to be the shell. 1 engineer is running bash 5 on his mac and does not see this problem.

[asolopovas]

same problem here

[s-geiger-si]

I am seeing this problem on Ubuntu 20.04 with docker-compose version 1.25.0. docker-compose pull fails fast with errors for all services. If I run a docker login <registry>, and then docker-compose pull then images are pulled as expected.