Docker-py: Cannot pull images from AWS ECR (login does not seem to work properly)

[mobileben]

I'm currently using docker-py 3.7.0 on an Ubuntu VM running Docker version 17.09.0-ce.

I'm having difficulty in what appears to be properly logging into docker. I've tried to get the AWS ECR credentials one of two ways: via boto3 and calling a subprocess for aws ecr get-login.

What happens is that when I try and pull an image, I get the dreaded

repository does not exist or may require 'docker login'

message.

I invoke this script with sudo (eg. sudo ./myscript.py). If, prior to running the script, I run

aws ecr get-login --no-include-email --region us-west2

and then run the results with sudo, the script will properly run.

I've tried variations and even used reauth during login. When I do that, I get the response

http://localhost:None "POST /v1.35/auth HTTP/1.1" 200 48
login_results {'IdentityToken': '', 'Status': 'Login Succeeded'}

I've even deleted the ~/.docker/config.json file but this doesn't help (a new file isn't even written).

here is a code snippet of what I'm doing for the login. It's a little messy right now since I've been trying permutations

    command = "aws ecr get-login --no-include-email --region us-west-2"
    p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    (out, err) = p.communicate()
    outstr = out.decode("utf-8")
    errstr = err.decode("utf-8")
    if p.returncode == 0:
        # Remove prefix
        outstr = outstr.lstrip('docker login ')
        parts = outstr.split(' ')
        print(parts)
        # -u
        username = parts[1].strip()
        # -p
        password = parts[3].strip()

        registry_url = parts[len(parts)-1].strip()
    else:
        print(p.returncode)
    '''
    token = ecr_client.get_authorization_token()
    username, password = base64.b64decode(token['authorizationData'][0]['authorizationToken']).decode('utf-8').split(":")
    registry_url = token['authorizationData'][0]['proxyEndpoint']
    '''
    print('username {}'.format(username))
    print('password {}'.format(password))
    print('registry_url {}'.format(registry_url))
    docker_client = docker.from_env()
    treg = registry_url + '/'
    login_results = docker_client.login(username=username, password=password, reauth=True, registry=treg)
    print('login_results {}'.format(login_results))

[mobileben]

One other thing to note here. I mentioned that if I get login and then log into docker prior to running the script it works. I noticed it also recreates the config.json file, whereas calling it from python does not. If config.json does exist, the entry is not updated when run from python.

If I run login as a subprocess, config.json does get modified. This behavior seems rather strange.

[mobileben]

I need to do a little more testing here, but if I run docker login as a subprocess. (which defeats the purpose of using docker-py), it works.

It seems that the problem comes from when config.json contains an entry for the registry with a stale auth token.

If config.json does not exist or if config.json exists but doesn't have an entry for the registry, it will work properly. Note this is independent of whether or not the reauth parameter is passed in with login.

[mobileben]

Okay, I ran some experiments the last few days. Had to deal with the 12 hour AWS ECR ticket so it took a little longer to do.

It does seem that there is an issue with docker-py.

Based on my findings, I can either use boto3 or run a sub-process calling the command line to aws ecr. However the only permutation that seems to work with the following steps.

1. use a sub-process to perform the docker login. This will result in the config.json file being updated (not sure if this has any relevance at all or not).
2. Create the docker client via docker_client = docker.from_env(). I have found doing this prior to the sub-process results in it not working properly (unless you have an already valid config.json
3. Then call docker_client.login(username=username, password=password, registry=registry_url)

Whether or not this is expected or not or if I'm doing something wrong, I don't know. This is what I've come up with as steps that work.

[mikeage]

I'm not sure if this is related to your issue, but I'm having the same problem, and I was able to resolve it by calling docker_client.login with registry equal to registry_url = token['authorizationData'][0]['proxyEndpoint'].replace("https://", ""). Note that I'm removing the leading https://

[jonapich]

the docker.login() response usually has a Status: Login Succeeded when it works, but when the user has a ~/.docker/config.json then the reply will be a totally different payload, containing the username/password (AWS and the super long b64 string use for docker login) instead of actually performing the login.

2 of my collegues had that issue, and after deleting the config.json the response was appropriate again.

[delwaterman]

Bump on this issue. I can't just delete the ~/.docker/config.json since I am using a combination of GCR and ECR and this will run on developer machines

[norton120]

Looks like this is still an issue? My workaround has been to wrap the whole thing in a context manager that clears out the config.json first, then does the work, and finally returns it to the original state. It's not pretty but it does the job.
(updated to use the decorator)

from contextlib import contextmanager

    @contextmanager
    def _flush_existing_login(registry: str) -> None:
        """ handles the known bug 
            where existing stale creds cause login
            to fail.
            https://github.com/docker/docker-py/issues/2256
        """
        config = Path(Path.home() / ".docker" / "config.json") 
        original = config.read_text()
        as_json = json.loads(original)
        as_json['auths'].pop(registry, None)
        config.write_text(json.dumps(as_json))
        try:
            yield
        finally:
            config.write_text(original)

[tehsven]

I also just ran into this issue today. The workaround provided by @norton120 is working for me.

[dugarsumit]

You can also set reauth=True from here https://docker-py.readthedocs.io/en/stable/client.html#docker.client.DockerClient.login

[durka]

This seems like a significant bug that should at least be documented! Ideally, login would throw an exception instead of silently doing nothing. The combination of removing https:// (@mikeage) and setting reauth=True (@dugarsumit) solved the issue for me. Either of those on its own didn't change the behavior at all.