Testcontainers-java: Mirroring settings ignored since 1.15.0-rc1

[andrus]

Environment: Bamboo CI, Linux, Java 11, Docker 19.03.13. DockerHub is blocked on the CI agent network, and instead mirroring is setup to use a local registry.

Have a build that worked fine on TC "1.14.3". Upgraded to "1.15.0-rc2" to address MacOS build issue (#3166), and the builds started failing. Same failure on either "1.15.0-rc1" or "1.15.0-rc2". Looks like the new version attempts to connect to "registry-1.docker.io" despite mirroring settings, and predictably fails (see above).

Good build (1.14.3)

main INFO ? ? i.b.j.j.d.DriverDataSource: Connecting to 'jdbc:tc:mysql:5.7.24://localhost/mydb?TC_REUSABLE=true' as 'null'
main INFO ? ? o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy: Found docker client settings from environment
main INFO ? ? o.t.d.DockerClientProviderStrategy: Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock
main INFO ? ? o.t.DockerClientFactory: Docker host IP address is 172.18.0.1
main INFO ? ? o.t.DockerClientFactory: Connected to docker: 
  Server Version: 19.03.13
  API Version: 1.40
  Operating System: CentOS Linux 7 (Core)
  Total Memory: 3770 MB
main INFO ? ? o.t.DockerClientFactory: Ryuk started - will monitor and terminate Testcontainers containers on JVM exit

Failing build (1.15.0-rc*):

main INFO ? ? i.b.j.j.d.DriverDataSource: Connecting to 'jdbc:tc:mysql:5.7.24://localhost/mydb?TC_REUSABLE=true' as 'null'
main INFO ? ? o.t.d.DockerMachineClientProviderStrategy: docker-machine executable was not found on PATH ([/usr/local/sbin, /usr/local/bin, /usr/sbin, /usr/bin, /sbin, /bin])
main INFO ? ? o.t.d.DockerClientProviderStrategy: Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
main INFO ? ? o.t.DockerClientFactory: Docker host IP address is 172.18.0.1
main INFO ? ? o.t.DockerClientFactory: Connected to docker: 
  Server Version: 19.03.13
  API Version: 1.40
  Operating System: CentOS Linux 7 (Core)
  Total Memory: 3770 MB
docker-java-stream--210510714 ERROR ? ? c.g.d.a.a.ResultCallbackTemplate: Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
        at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
        at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
        at java.base/java.lang.Thread.run(Thread.java:834)

[bsideup]

mirroring settings

@andrus are you referring to registry-mirrors in Docker?

[andrus]

are you referring to registry-mirrors in Docker?

Yes

[bsideup]

@andrus are you confident that the issue is with Testcontainers? Because it is transparent to Testcontainers and we don't have a special handling for registry-mirrors /cc @rnorth

[andrus]

Appears to be so. I take the same code, and change TC version in the pom.xml:

• 1.15.0-rc1 - failure
• 1.15.0-rc2 - failure
• 1.14.3 - success

Perhaps 1.15.x added some explicit calls to the registry?

[andrus]

Ok, found it! A colleague from the ops team pointed to this change - switch to "testcontainers" prefix from "testcontainersofficial". Turns out our repo proxy had explicit prefix matching for images to avoid untrusted sources. Changed that, and everything works. Perhaps something to mention in the upgrade notes.

[bsideup]

@andrus thanks for reporting back! I am glad that it works well now :) We indeed had a bit of back and forth with the Docker Hub orgs, but it is in the release notes:
https://github.com/testcontainers/testcontainers-java/releases/tag/1.15.0-rc1

Use testcontainers/* Docker Hub images (#2850) @bsideup

[andrus]

Thanks for awesome support BTW!