Compose: `docker-compose build --pull` on Mac fails with no basic auth credentials

[rca]

Description of the issue

When installing docker and docker-compose in a virtual environment, the current versions that are installed are:

docker==3.7.2
docker-compose==1.24.0

When building a project that is pulling the FROM image from a private repo I get the following error:

Step 1/16 : FROM registry.mydomain.net/image:1.2.3
ERROR: Service 'app' failed to build: Get https://registry.mydomain.net/v2/image/manifests/1.2.3: no basic auth credentials

Context information (for bug reports)

Output of docker-compose version

[berto:env@MacBook-Pro]$ docker-compose version
docker-compose version 1.24.0, build 0aa5906
docker-py version: 3.7.2
CPython version: 3.6.6
OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018

Output of docker version

[berto:env@MacBook-Pro]$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     true

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

[berto:env@MacBook-Pro]$ docker-compose -f docker-compose.yml config
version: '3.4'
services:
  app:
    build:
      context: /Users/berto/Projects/openslate/service
    image: registry.mydomain.net/service:local

Steps to reproduce the issue

1. pipenv shell
2. pip install docker-compose
3. build Dockerfile whose FROM image is private

Observed result

[berto:env@MacBook-Pro]$ docker-compose -f docker-compose.yml build --pull
WARNING: The DOCKER_IMAGE variable is not set. Defaulting to a blank string.
Building app
Step 1/16 : FROM registry.mydomain.net/image:1.2.3
ERROR: Service 'app' failed to build: Get https://registry.mydomain.net/v2/image/manifests/1.2.3: no basic auth credentials

Expected result

When configuring the virtual environment with the following, build succeeds:

pip install docker==3.6.0 docker-compose==1.23.2

Stacktrace / full error message

[berto:env@MacBook-Pro]$ DOCKER_IMAGE=test docker-compose -f docker-compose.yml build --pull
Building app
Step 1/16 : FROM registry.mydomain.net/image:1.2.3
Service 'app' failed to build: Get https://registry.mydomain.net/v2/image/manifests/1.2.3: no basic auth credentials

Additional information

• Mac OS X 10.14.4
• docker-compose install via pip install in a pipenv-created virtual environment

[ijc]

@rca did you do a docker login at any point?

[rca]

Indeed. Simply downgrading the packages works, upgrading them back to the new versions breaks.

[ijc]

@rca, thanks for the answer.

Does a simple docker pull registry.mydomain.net/image:1.2.3 work? (if it does then this may hide the issue until you do a docker image rm registry.mydomain.net/image:1.2.3 in case you try to repro afterwards)

Are you using any credential helpers? Please could you post the content of ~/.docker/config.json (you will want to anonymise at least some of the values, in particular of the auths key).

[rca]

I have tried removing the image locally and pulling. The docker pull command succeeds. The docker-compose command fails.

My Docker config:

[berto:env@MacBook-Pro]$ cat ~/.docker/config.json
{
  "HttpHeaders" : {
    "User-Agent" : "Docker-Client/18.06.1-ce (darwin)"
  },
  "credsStore" : "osxkeychain",
  "auths" : {
    "registry.mydomain.net" : {

    },
    "https://registry.mydomain.net" : {

    }
  },
  "stackOrchestrator" : "swarm"
}

[rca]

I noticed the docker-py 4.0.0 release notes, under bug fixes notes:

APIClient.inspect_distribution now correctly sets the authentication
headers on the request, allowing it to be used with private repositories
This change also applies to DockerClient.get_registry_data

The docker-compose setup.py, however, pins to a version of docker-py prior to 4.0:

 'docker[ssh] >= 3.7.0, < 4.0',

Per my initial post, this does work with docker==3.6.0 and its corresponding compose version.

[ijc]

Thanks @rca. You are using the oxskeychain cred helper, so I was wondering if this is the same issue as #6688, but the versions where you see/don't see the issue do not line up with that theory, so I think we can scrub it.

Your theory about docker-py is worth pursuing, I think the only reason for the current constraint is that docker-py 4.0 is ~2 days old and compose simply hasn't been updated yet.

@ulyssessouza do you know what the situation is wrt bumping to docker-py? Given there are breaking changes in 4.0 I suppose it is not as simple as just updating the constraints.

[ulyssessouza]

We should be able to bump docker-py without problems. On local tests docker-compose behaved very well with docker-py 4.0.0
By now, the actual newest version is 4.0.1. The next release of docker-compose (in a few days) is intended to include this bump.

[rca]

Awesome, thank you @ulyssessouza! 🙌🏽

[davidPazmino]

I had the same issue in IOS. For uninstall i used AppCleaner.
I uninstalled and restarted and works.

[fong-kinduct]

We are experiencing this issue with the latest version of (2.1.0.0) Build 36874

Specifically we can successfully pull via docker pull, but when a private image is used as a FROM
ie:
FROM <account>.dkr.ecr.<region>.amazonaws.com/<repo>/<image>:<tag>
in a docker-compose build statement it fails to pull with

ERROR: Service '<service>' failed to build: Get https://<account>.dkr.ecr.<region>.amazonaws.com/v2/<repo>/manifests/<tag>: no basic auth credentials

I am logged in via docker login, as i can successfully pull with:

docker pull <account>.dkr.ecr.<region>.amazonaws.com/<repo>/<image>:<tag>

For more context, this error has existed in docker edge for months.

Funnily enough after upgrading to the most recent version of docker edge, it is not affected by this bug.

[quaelin]

We're experiencing this, too, using the same versions as @fonged. I'm wondering why this issue is tagged as a question... seems to be a bug. Possibly related to #6832.

My team is having to downgrade to v2.0.0.3 to work around this :(

[Giaco9]

Hi all, I'm experiencing the same issue. I'm not able to pull image from private repositories (AWS ECR in my case) using docker-compose but I'm able to pull same images using docker.
Here I found that running sudo docker login and sudo docker-compose everything is working fine. I think running those commands using sudo isn't acceptable so I'm looking for a different solution but this may help investigate the problem.

Thanks

[ulyssessouza]

Looks like the last reproductions are related to https://github.com/docker/compose/issues/6517#issuecomment-518646263
It does not look like a docker-compose issue. Could you please confirm that?

[fluetm]

@ulyssessouza this is not a docker-compose problem. I've isolated it to a commit in docker-py. Opened an issue there with a detailed explanation of the root cause: https://github.com/docker/docker-py/issues/2402.

Also have a PR open with a candidate fix which works for me locally: https://github.com/docker/docker-py/pull/2401. Not sure if it's the correct fix, but at least should hopefully shed light in the right area.

[roynasser]

Experiencing the same...

[jasonmevans]

Just found that we are experiencing this as well. My forehead is sore from banging it on my desk.

[jondkelley]

Bump! My organization has this issue as well.

[ndeloof]

https://github.com/docker/docker-py/pull/2401 has been merged, so we just need a release of docker-py to get this fixed

[CpuID]

https://github.com/docker/docker-py/releases/tag/4.1.0 was released 11 days ago