Testcontainers-java: Support for identitytoken auth type

[peter-evans]

Issue found when attempting to run the test in the spring boot example here:
https://github.com/testcontainers/testcontainers-java-examples/tree/master/spring-boot

Environment

• Ubuntu 16.04
• docker 1.13.1
• IntelliJ IDEA

This is the output from executing the maven test phase: build_output.txt

I verified the version by adding the following dependency to the pom.xml

        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>testcontainers</artifactId>
            <version>1.1.9</version>
        </dependency>

@bsideup identified issue https://github.com/testcontainers/testcontainers-java/pull/270 as the possible cause.

After testing with version 1.1.7 the test passed. So it's likely a regression caused by the above issue.

Just an extra detail. After the test passed using version 1.1.7 I recompiled with 1.1.9 and the test passed. I had to manually delete the redis image to cause the error to occur again. So if you already have the required image in your local environment you may need to remove it to reproduce.

[peter-evans]

When I found this issue I was testing on a machine that has a ~/.docker/config.json file present. I believe this file is used to store auth details for private repositories.

I just tried to reproduce the issue on another machine with the exact same environment. The only difference is that this machine does not have a ~/.docker/config.json file present. The test passed fine with version 1.1.9.

[rnorth]

Hi @peter-evans
Sorry - that's very odd. Could you possibly share the ~/.docker/config.json file as well (please do mask out any sensitive values, though!)

This will probably help reproduce and also identify if there are any edge cases for parsing that need to be handled upstream in docker-java. @tourea might be interested to know about this if there is anything!

Richard

[peter-evans]

Hi @rnorth

This is the file: config.json.txt

Except for the following two actions to mask sensitive details everything in the file is unchanged.

• Added asterisks to partially mask values
• Replaced my real company name with companyname

There is one entry in the file for a company internal docker registry. All the other entries have presumably been inserted by the Google Cloud SDK.

[peter-evans]

I noticed that the entry in the file for my company's internal docker registry is different. It uses identitytoken instead of email.

Removing that one entry from the file allows the file to be parsed and the spring boot project test passes with version 1.1.9. It seems likely that the parsing code for config.json isn't taking into account the identitytoken format.

The product my company is using is the Docker Datacenter. It comes with the Docker Trusted Registry component, which I believe is using the identitytoken format.

Update: Docker Datacenter has been re-branded Docker Enterprise Edition.

[rnorth]

@peter-evans Sorry for the slow response.
It looks like that field isn't supported in docker-java's AuthConfig DTO. I'm not sure why the changes to support registry v2 format broke things, though. Is it possible that 1.1.7 was working for you because docker-java ignored your registry config, whereas now it's trying and failing to parse it?

We'll need to look into this further.

[peter-evans]

@rnorth Yes, I believe that is why version 1.1.7 was working. Before this change docker-java didn't support config.json and so didn't attempt to parse it.

I think you are right about that AuthConfig DTO. The Docker Trusted Registry, which is a component of the newly branded Docker Enterprise Edition, is the official commercial offering from Docker, so it really should support it.

[peter-evans]

Spotify's docker-client also had the same issue and it seems to have been fixed recently here with an update here. That fix might help as a reference.

[mabn]

I have similar problem - parsing fails when config.json contains "credsStore".

There was an issue for this in docker-java (https://github.com/docker-java/docker-java/issues/806) and it's fixed, but new version wasn't released yet. It will be included in docker-java 3.0.12.

[email2liyang]

I have the same issue when I try to use version 1.3.1

[bsideup]

Hi @peter-evans, @mabn, and @email2liyang,

we just released 1.4.2, it should fix the issue, could you please try it? Thanks!

[email2liyang]

@bsideup , confirmed, it works in v 1.4.2, thanks very much

[mabn]

Yup, it works

[peter-evans]

I tested it but it still doesn't seem to recognise and parse the field identitytoken. See the error below.

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.009 s <<< FAILURE! - in com.github.peterevans.testcontainerstemplate.StringReverseClientIT
[ERROR] com.github.peterevans.testcontainerstemplate.StringReverseClientIT  Time elapsed: 0.008 s  <<< ERROR!
java.lang.ExceptionInInitializerError
    at com.github.peterevans.testcontainerstemplate.StringReverseClientIT.<clinit>(StringReverseClientIT.java:25)
Caused by: org.testcontainers.shaded.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: 
Unrecognized field "identitytoken" (class com.github.dockerjava.api.model.AuthConfig), not marked as ignorable (6 known properties: "serveraddress", "username", "auth", "password", "email", "registrytoken"])
 at [Source: N/A; line: -1, column: -1] (through reference chain: java.util.LinkedHashMap["auths"]->java.util.LinkedHashMap["docker-regi.intra.companyname-it.com"]->com.github.dockerjava.api.model.AuthConfig["identitytoken"])
    at com.github.peterevans.testcontainerstemplate.StringReverseClientIT.<clinit>(StringReverseClientIT.java:25)

This is the part of the config.json file it doesn't recognise (sensitive info masked):

{
    "auths": {
        "docker-regi.intra.companyname-it.com": {
            "auth": "ZXZh************Og==",
            "identitytoken": "981f****************************1e44"
        }
    }
}

[Nowheresly]

I have the same issue with the unrecognized identitytoken.
A PR has just been merged in project docker-java to solve this (docker-java/docker-java#1004).
I think this issue could be closed as soon as the new release of docker-java with the fix is available is used.

[rnorth]

Thanks @Nowheresly - will watch out for that being released.

[majusmisiak]

@Nowheresly are you aware of reliable workaround, until this issue is fixed in the upstream (other than removing "identitytoken": ... from ~/.docker/config.json)? Even with image pull disabled like:

    @Rule
    public RuleChain chain = RuleChain
            .outerRule(new DockerComposeContainer(new File(dockerComposeFile))
                    .withLocalCompose(true)
                    .withPull(false));

it is still failing with org.testcontainers.shaded.com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.

Should the config be read, if pulling is disabled?

[Nowheresly]

Hi @majusmisiak , unfortunately, my current workaround consist in remove the identityToken (by calling docker logout).
In fact the fix is located in only one class of docker-java project. Maybe if you try to put the correct version of the AuthConfig class somewhere in your classpath so that it takes precedence over the official one inside the docker-java jar, it may help.
The correct version of the class can be found here in this PR docker-java/docker-java#1083

[majusmisiak]

Hi @Nowheresly, patching this single class directly solved the problem. I used AuthConfig.java directly from master (sha1 153e06a4) with testcontainers:core 1.8.3 and testcontainers:docker-compose 0.9.9.

This seems more portable workaround than removing "identitytoken: ..., since I can run it as-it-is on any build server.

Many thanks, now awaiting docker-java 3.1.0 release.

[bsideup]

@majusmisiak have you tried running the code from #845 PR?
If not, could you please? @rnorth is currently testing it and any feedback will help 👍

[majusmisiak]

@bsideup it will give it a spin locally and add comment on PR if I come up with anything useful for you.

[rnorth]

I think we'll probably need to bump up to docker-java 3.1.0-rc4 to benefit from that PR. I think that would be worth including in #845...

[bsideup]

@rnorth yes, upgrading docker-java is a good idea 👍

[Nowheresly]

I agree as well since this bug is really annoying.

[rnorth]

We updated docker-java in #845 on Friday.

We have this out in a Release Candidate build (1.9.0-rc1) for anyone who is keen to try it!

Release notes

[kiview]

Fixed with #845.
We recommend trying out Testcontainers 1.9.1. If this problem persists, I'll gladly reopen the issue.

[jonkjenn]

Seems to me the support for identitytoken was not introduced until docker-java 3.1.0-rc5?