Testcontainers-java: Authentication for Docker Hub private registry without credential helpers

[nefilim]

Following up here on a question about authentication for a private repo hosted at docker hub that I posted on Slack.
For some context, credentials are stored in ~/.docker/config.json in the auth field (this is actually base64(username:password)) eg:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "abcdef=",
            "email": "[email protected]"
        }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/18.03.1-ce (darwin)"
    }
}

We supply image names such acme/microservice:0.1 to testcontainers. There are also public images from Docker Hub (redis, postgres etc) and implicitly from quay.io (ryuk). Now, looking at the following code:

https://github.com/testcontainers/testcontainers-java/blob/master/core/src/main/java/org/testcontainers/utility/RegistryAuthLocator.java#L80

reposName ends up being "" (empty) and it falls back to defaultAuthConfig which I _believe_ is controlled by ~/.docker-java.properties (ala docker-java project), which does not exist and hence it fails trying to download the image:

2018-08-05 14:22:22,061 ERROR [testcontainers-netty-1-13] c.g.d.c.a.ResultCallbackTemplate: Error during callback
com.github.dockerjava.api.exception.NotFoundException: {"message":"pull access denied for acme/microservice, repository does not exist or may require 'docker login'"}

    at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:103)
    at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
    at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

In contrast, the default behaviour of docker pull in the absence of a registry host, is to pull from docker hub and as such matches the URL https://index.docker.io/v1/ in ~/.docker/config.json - this does not appear to be the default behaviour of testcontainers.

I would like to see testcontainers choosing the auth from ~/.docker/config.json in the https://index.docker.io/v1/ stanza within findExistingAuthConfig(config, ""). To pursue this behaviour I put together this PR: https://github.com/testcontainers/testcontainers-java/pull/819 - I ran into a number of dead ends:

1. Just replacing the reposName with index.docker.io in case it's empty does not work, docker-java explicitly checks for that hostname and errors out (why???? it's completely valid using the docker command line docker pull index.docker.io/acme/microservice:0.1)
2. Modifying the auth lookup to find the stanza for index.docker.io in config.json (as in this PR: https://github.com/testcontainers/testcontainers-java/pull/819) does not work either, from what I can tell, docker-java does not decompose the auth field (into username/password) before creating the JSON authentication header as per https://docs.docker.com/engine/api/v1.37/#section/Authentication so credentials are not being sent properly:

2018-08-05 14:33:08,776 INFO  [pool-6-thread-3] ?.0.112]: effective auth config [AuthConfig[username=<null>,password=<null>,email=<null>,registryAddress=https://index.docker.io/v1/,auth=abcdef=,registrytoken=<null>]]
2018-08-05 14:33:09,917 ERROR [testcontainers-netty-1-15] c.g.d.c.a.ResultCallbackTemplate: Error during callback
com.github.dockerjava.api.exception.NotFoundException: {"message":"pull access denied for acme/microservice, repository does not exist or may require 'docker login'"}

    at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:103)
    at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)

The problems with using docker-java.properties are:

1. Only supports a single registry URL, it seems these credentials are aimed at pushing to a single private registry (not really for pulling from multiple (private) registries)
2. If credentials are set here, testcontainers fails to pull ryuk:0.2.2 from quay.io - I have not confirmed but I wonder if it's trying to the use the credentials configured in docker-java.properties (for index.docker.io) for quay.io - a bug?

2018-08-05 14:43:34,343 ERROR [testcontainers-netty-1-5] c.g.d.c.a.ResultCallbackTemplate: Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://quay.io/v2/testcontainers/ryuk/manifests/0.2.2: unauthorized: Invalid Username or Password"}

    at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:109)
    at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
    at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)

The most confusing thing - why has nobody else reported this? Am I just doing something horribly wrong? :)

[rnorth]

I think that #813 and #790 are related to this ticket, but thanks for the detailed write-up. I'll have a think about this.

[rnorth]

Also from ffissore in #836:

I'm using testcontainers 1.8.3. I have a test db image on a private repo on AWS ECR.
When I run my tests with
eval $(aws ecr get-login --no-include-email); mvn clean test
testcontainers is unable to pull the image.

Error message is
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://REPOID.dkr.ecr.eu-west-1.amazonaws.com/v2/IMAGENAME/manifests/IMAGETAG: no basic auth credentials"}

My .docker/config.json is

{
"auths": {
"REPOID.dkr.ecr.eu-west-1.amazonaws.com": {
"auth": "AUTH"
},
"https://index.docker.io/v1/": {
"auth": "AUTH"
}
},
"HttpHeaders": {
"User-Agent": "Docker-Client/18.06.0-ce (linux)"
}
}
Attached a snippet of the logs log.txt

[rnorth]

And from @kiptix in #835:

It works with version 1.8.0, but with 1.8.1 i got the following exception:
Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: unauthorized: authentication required

OS is Linux.
Docker version 17.05.0-ce, build 89658be
~/.docker/config.json (with replaced url and token):
{
"auths": {
"registry.url": {
"auth": "token"
}
}
}

[rnorth]

@nefilim thanks for the draft PR - it's been a good basis for trying to figure this out while on a long plane journey :) I have some additions worked out.

This is still WIP, but tackles the two underlying issues that you and others have encountered:

• basic auth credentials not being parsed out of the config.json file (for all private repos)

• registry auth for index.docker.io being rejected. To resolve this I've overridden the effectiveAuthConfig method of DockerClientConfig for now, to try and come up with a solution that doesn't require changing docker-java (yet).

In the long term, this requires a fair amount of modification to docker-java to do it cleanly, but for now we can basically layer this on top and refactor later.

[reardonm]

FWIW, I can confirm that I'm experiencing the same problem as @nefilim.

[rnorth]

I think I've done the work necessary in #845! However, given that understanding/reproducing the registry setup is potentially the harder thing here, it'd be _extremely_ helpful if people affected by this could give this a try before we call the PR done. Would that be OK?

There is a jitpack build available with the following details:

Repository: maven { url 'https://jitpack.io' }
Dependency: com.github.testcontainers.testcontainers-java:testcontainers:auth-fixes-SNAPSHOT (or use 6d3d4ea86c for a recent non-snapshot version)

Thanks!

[reardonm]

Oh, I can now appreciate the problem. Nice solution. I've been trying it out on my workstation (macOS) as on a CI server with my work-around removed. Both are working correctly...
macOS:

DEBUG o.t.utility.RegistryAuthLocator - Looking up auth config for image: foo/stubservice:latest
DEBUG o.t.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /Users/myuser/.docker/config.json (exists) and commandPathPrefix:
DEBUG o.t.utility.RegistryAuthLocator - registryName [index.docker.io] for dockerImageName [foo/stubservice:latest]
DEBUG o.t.utility.RegistryAuthLocator - Executing docker credential helper: docker-credential-osxkeychain to locate auth config for: index.docker.io
DEBUG o.t.s.o.z.exec.ProcessExecutor - Executing [docker-credential-osxkeychain, get].
DEBUG o.t.s.o.z.exec.ProcessExecutor - Started java.lang.UNIXProcess@398dada8
DEBUG o.t.s.o.z.exec.WaitForProcess - java.lang.UNIXProcess@398dada8 stopped with exit code 0
DEBUG o.t.utility.RegistryAuthLocator - Credential helper provided auth config for: index.docker.io
DEBUG o.t.utility.RegistryAuthLocator - found creds store auth config [AuthConfig{username=mydockerhub, password=hidden non-blank value, auth=blank, email=null, registryAddress=index.docker.io, registryToken=blank}]

Centos:

DEBUG o.t.utility.RegistryAuthLocator - Looking up auth config for image: foo/stubservice:latest
DEBUG o.t.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /home/centos/jenkins/workspace/myservice_component-test-TXAW6D7VAHKETGC2PDRAEWPZFPITNCY62VAZZPLUCI2CNW5OCMOQ@tmp/3a0bfcca-b0da-4266-b6f0-0780fe87df9d/config.json (exists) and commandPathPrefix: 
DEBUG o.t.utility.RegistryAuthLocator - registryName [index.docker.io] for dockerImageName [foo/stubservice:latest]
DEBUG o.t.utility.RegistryAuthLocator - found existing auth config [AuthConfig{username=jenkins, password=hidden non-blank value, auth=hidden non-blank value, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]
DEBUG o.t.d.a.AuthDelegatingDockerClientConfig - effective auth config [AuthConfig{username=jenkins, password=hidden non-blank value, auth=hidden non-blank value, email=null, registryAddress=https://index.docker.io/v1/, registryToken=blank}]

Also, also thanks for avoiding the repository credentials in the logs. I was going to ask about that. This project is really looking great.

[rnorth]

Thanks @reardonm, it's useful to get feedback that this is working for you.

[kiptix]

The fix works for me, too

[nefilim]

Thanks so much @rnorth - looking good here too.

Sorry for the delay, was travelling and had some technical issues. Don't know gradle at all (use SBT) - not very trivial/clear to try and get it to publish to a local ivy repo, took some time to find a workaround :)