Testcontainers-java: How to Specify Private Registries?

[austin127001]

I would like to understand how I can explicitly pass Docker auth/registry configuration to testcontainers. I see a lot of settings are processed as part of docker-java but I can't seem to get them to work as documented.

What I'm attempting:

• A gradle build occurs inside of a Docker container (I've read through: https://www.testcontainers.org/supported_docker_environment/continuous_integration/dind_patterns/)
  
  
  
  • The Docker unix socket is mounted into this container
  
  
  • The source-code for a given project is mounted in
  
  
  • The image does not contain a ~/.docker/config.json
  
  
  • Appropriate credentials, endpoints, etc. are passed in at container runtime (large org, multiple BUs)[HOW DO I DO THIS?]
  
  
• DiniD is not an option
• Never touch the outside world; all internal registry mirror/proxies

What I've tried:

• Explicitly creating a ~/.docker/config.json inside of the container
  
  
  
  • No setting was picked up
  
  
• Explicitly creating a config.json along with a DOCKER_CONFIG=....
  
  
  
  • No setting was picked up
  
  
• Passing the settings as part of the gradle build (e.g. gradle build -Pregistry.url=... or gradle build -Dregistry.url
  
  
  
  • No setting was picked up
  
  
• Creating a docker-java.properties (described: https://github.com/docker-java/docker-java#configuration)
  
  
  
  • No setting was picked up
  
  
• Setting the values in the test explicitly (No such withRegistryUrl):

@ClassRule
    public static GenericContainer rabbitMq = new GenericContainer<>("<INTERNAL_URL>/rabbitmq:management-alpine")
            .withExposedPorts(5672,15672)
                        .withRegistryUrl("<INTERNAL_URL>")
                        .withRegistryEmail("[email protected]");

No matter the configurations described above, it always fails with:

gradle build --info
...
> Task :test
Excluding []
Caching disabled for task ':test' because:
  Build cache is disabled
Task ':test' is not up-to-date because:
  Task has failed previously.
Starting process 'Gradle Test Executor 1'. Working directory: /workspace Command: /usr/lib/jvm/java-11-openjdk-11.0.1.13-3.el7_6.x86_64/bin/java -Dorg.gradle.native=false @/tmp/gradle-worker-classpath9403681164797392133txt -Xmx512m -Dfile.encoding=US-ASCII -Duser.country=US -Duser.language=en -Duser.variant -ea worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 1'
Successfully started process 'Gradle Test Executor 1'

test.containers.TestContainersTest STANDARD_OUT
    17:41:02.891 [Test worker] INFO  o.t.d.DockerClientProviderStrategy - Will use 'okhttp' transport
    17:41:13.062 [Test worker] ERROR o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - ping failed with configuration Environment variables, system properties and defaults. Resolved:
        dockerHost=unix:///var/run/docker.sock
        apiVersion='{UNKNOWN_VERSION}'
        registryUrl='https://index.docker.io/v1/'
        registryUsername='<BUILD_USER>'
        registryPassword='null'
        registryEmail='null'
        dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=<BUILD_USER>,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/workspace/config.json,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'
     due to org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
    org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
...

tl;dr:
Can I see working code where the following have been set explicitly as part of a mvn/gradle build process?

dockerHost=
apiVersion=
registryUrl=
registryUsername=
registryPassword=
registryEmail=
dockerConfig=

Version Info:
testCompile "org.testcontainers:junit-jupiter:1.11.3"

Gradle 5.4.1
------------------------------------------------------------

Build time:   2019-04-26 08:14:42 UTC
Revision:     261d171646b36a6a28d5a19a69676cd098a4c19d

Kotlin:       1.3.21
Groovy:       2.5.4
Ant:          Apache Ant(TM) version 1.9.13 compiled on July 10 2018
JVM:          11.0.1 (Oracle Corporation 11.0.1+13-LTS)
OS:           Linux 3.10.0-957.1.3.el7.x86_64 amd64

docker version
Client:
 Version:           18.09.8
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        0dd43dd87f
 Built:             Wed Jul 17 17:40:31 2019
 OS/Arch:           linux/amd64
 Experimental:      false

[rnorth]

Edit: I wrote the below in an attempt to help debug private registry auth, but re-reading your notes I think that actually the problem is entirely different. I'll still post the below for posterity, but please ignore for now. I'll post another comment with what I think the actual problem is.

I'd be happy to try and help, though unfortunately it may be more a case of comparing setups and working out the differences. We use private registry auth extensively at Skyscanner so I can let you know what I know works.

For starters though, could you perhaps adjust the logging level for org.testcontainers.utility.RegistryAuthLocator to DEBUG? It'll give you more visibility into what Testcontainers' registry auth code is doing.

Explicitly creating a ~/.docker/config.json inside of the container - No setting was picked up

We use this as the primary means of authentication for Docker/Testcontainers, so I find it extremely strange that it's not working for you. In our setup we use this as follows:

• with credHelpers and credsStore settings for local dev (developer machines authenticated to our private registry)
• with 'fixed' auth credentials in CI, which in our case are temporary credentials per build job generated by the aws ecr CLI tool.

I'd suggest sticking with ~/.docker/config.json and working out what's going wrong. Please could you confirm:

• That the config.json file is definitely in the same container as Testcontainers is being run in and is readable by the JVM process (sorry if this sounds basic!)
• That the config is valid - if you'd be able to get a docker binary into the build container execute a docker pull ..., it'd help

As mentioned, DEBUG level logs should help a lot, as they'll tell us what the RegistryAuthLocator is finding and give us an (obfuscated) dump of the credentials that it manages to resolve.

[rnorth]

The crucial part of the error message is:

    17:41:13.062 [Test worker] ERROR o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - ping failed with configuration Environment variables, system properties and defaults. Resolved:
        dockerHost=unix:///var/run/docker.sock
...

What this is saying is that Testcontainers cannot connect to the docker daemon on that UNIX socket path.

The important thing to know is that private registry authentication plays no part in the connection to the docker daemon - it's entirely unused until Testcontainers attempts to pull an image. In your case, it can't even connect to the socket.

I'm afraid anything to do with private registry authentication is a red herring at this point.

I'd suggest double and triple checking that the docker socket is mounted into the build container at the expected path successfully, including retaining the correct permissions. Again if you can try executing a command with the docker CLI tool inside of the build container, it would tell us a lot. If docker CLI can talk to the daemon via the socket, Testcontainers/docker-java should be able to as well.

[austin127001]

@rnorth I can confirm that the Docker sock is mounted correctly. I am able to:

• Pull new images
• Start new containers
• rinse repeat

So I believe that part to be working. I'm looking into your previous comment. Thanks for taking a look!

[rnorth]

Hmm, OK, it's still definitely not going to be a registry auth problem at this stage - the docker socket is unauthenticated.

Please could you capture Testcontainers logs (everything under org.testcontainers) at DEBUG level, so we can see what is happening in the environment detection and connection test code?

[ajatman]

[Same as @austin127001 ]

Revisiting the issue I still can't seem to get any settings to be picked up:

17:29:42.621 [INFO] [org.gradle.process.internal.DefaultExecHandle] Successfully started process 'Gradle Test Executor 1'
17:29:42.931 [DEBUG] [TestEventLogger]
17:29:42.932 [DEBUG] [TestEventLogger] Gradle Test Executor 1 STARTED
17:29:43.259 [DEBUG] [TestEventLogger]
17:29:43.259 [DEBUG] [TestEventLogger] test.containers.TestContainersTest STARTED
17:29:43.419 [DEBUG] [TestEventLogger]
17:29:43.419 [DEBUG] [TestEventLogger] test.containers.TestContainersTest STANDARD_OUT
17:29:43.419 [DEBUG] [TestEventLogger]     17:29:43.413 [Test worker] DEBUG o.t.u.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/home/internal_user/.testcontainers.properties
17:29:43.420 [DEBUG] [TestEventLogger]     17:29:43.417 [Test worker] DEBUG o.t.u.TestcontainersConfiguration - Testcontainers configuration overrides loaded from TestcontainersConfiguration(properties={docker.client.strategy=org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy, registry.url=internal.docker.registry})
17:29:43.434 [DEBUG] [TestEventLogger]     17:29:43.431 [Test worker] INFO  o.t.d.DockerClientProviderStrategy - Loaded org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy from ~/.testcontainers.properties, will try it first
17:29:43.452 [DEBUG] [TestEventLogger]     17:29:43.451 [Test worker] INFO  o.t.d.DockerClientProviderStrategy - Will use 'okhttp' transport
17:29:43.605 [DEBUG] [TestEventLogger]     17:29:43.604 [ducttape-0] DEBUG o.t.d.DockerClientProviderStrategy - Pinging docker daemon...
17:29:43.800 [DEBUG] [TestEventLogger]     17:29:43.799 [Test worker] INFO  o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
17:29:43.817 [DEBUG] [TestEventLogger]     17:29:43.812 [Test worker] INFO  o.t.d.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved:
17:29:43.817 [DEBUG] [TestEventLogger]         dockerHost=unix:///var/run/docker.sock
17:29:43.818 [DEBUG] [TestEventLogger]         apiVersion='{UNKNOWN_VERSION}'
17:29:43.818 [DEBUG] [TestEventLogger]         registryUrl='https://index.docker.io/v1/'
17:29:43.818 [DEBUG] [TestEventLogger]         registryUsername='internal_user'
17:29:43.818 [DEBUG] [TestEventLogger]         registryPassword='null'
17:29:43.818 [DEBUG] [TestEventLogger]         registryEmail='null'
17:29:43.818 [DEBUG] [TestEventLogger]         dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=internal_user,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/home/internal_user/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'
17:29:43.819 [DEBUG] [TestEventLogger]
17:29:43.896 [DEBUG] [TestEventLogger]     17:29:43.894 [Test worker] DEBUG o.t.utility.RegistryAuthLocator - Looking up auth config for image: alpine:3.5
17:29:43.896 [DEBUG] [TestEventLogger]     17:29:43.894 [Test worker] DEBUG o.t.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /home/internal_user/.docker/config.json (exists) and commandPathPrefix:
17:29:43.901 [DEBUG] [TestEventLogger]     17:29:43.897 [Test worker] DEBUG o.t.utility.RegistryAuthLocator - registryName [index.docker.io] for dockerImageName [alpine:3.5]
17:29:43.901 [DEBUG] [TestEventLogger]     17:29:43.897 [Test worker] DEBUG o.t.utility.RegistryAuthLocator - no matching Auth Configs - falling back to defaultAuthConfig [null]
17:29:43.901 [DEBUG] [TestEventLogger]     17:29:43.897 [Test worker] DEBUG o.t.d.a.AuthDelegatingDockerClientConfig - Effective auth config [null]
17:29:46.695 [LIFECYCLE] [org.gradle.process.internal.health.memory.MemoryManager]
17:29:46.695 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting OS memory status event {Total: 52590579712, Free: 46844362752}
17:29:46.695 [DEBUG] [org.gradle.launcher.daemon.server.health.LowMemoryDaemonExpirationStrategy] Received memory status update: {Total: 52590579712, Free: 46844362752}
17:29:46.695 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting JVM memory status event {Maximum: 536870912, Committed: 268435456}
17:29:51.695 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting OS memory status event {Total: 52590579712, Free: 46858334208}
17:29:51.695 [DEBUG] [org.gradle.launcher.daemon.server.health.LowMemoryDaemonExpirationStrategy] Received memory status update: {Total: 52590579712, Free: 46858334208}
17:29:51.695 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting JVM memory status event {Maximum: 536870912, Committed: 268435456}
17:29:53.287 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] DaemonExpirationPeriodicCheck running
17:29:53.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
17:29:53.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
17:29:53.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
17:29:53.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
17:29:53.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
17:29:53.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
17:29:56.695 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting OS memory status event {Total: 52590579712, Free: 46858293248}
17:29:56.695 [DEBUG] [org.gradle.launcher.daemon.server.health.LowMemoryDaemonExpirationStrategy] Received memory status update: {Total: 52590579712, Free: 46858293248}
17:29:56.695 [DEBUG] [org.gradle.process.internal.health.memory.MemoryManager] Emitting JVM memory status event {Maximum: 536870912, Committed: 268435456}
17:29:44.230 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger]
17:29:44.230 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] > Task :test
17:29:58.909 [DEBUG] [TestEventLogger]     17:29:58.909 [tc-okhttp-stream-937090908] ERROR c.g.d.c.async.ResultCallbackTemplate - Error during callback
17:29:58.910 [DEBUG] [TestEventLogger]     com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
17:29:58.910 [DEBUG] [TestEventLogger]
17:29:58.910 [DEBUG] [TestEventLogger]          at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:276)
17:29:58.910 [DEBUG] [TestEventLogger]          at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$1(OkHttpInvocationBuilder.java:292)
17:29:58.910 [DEBUG] [TestEventLogger]          at java.base/java.lang.Thread.run(Thread.java:834)
17:29:58.916 [DEBUG] [TestEventLogger]
17:29:58.916 [DEBUG] [TestEventLogger] test.containers.TestContainersTest > initializationError STARTE

The Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection error makes sense as this environment shouldn't have external internet access.

However, for the life of me, I cannot set the Docker client properties no matter what I do.

This file say's it's found but nothing seems to get picked up.

[internal_user@e274a182637b workspace]$ cat /home/internal_user/.docker/config.json
{
    "auths": {
        "internal.docker.registry": {
            "auth": "<AUTH>"
        }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/18.06.1-ce (linux)"
    }
}

This at least shows up in the logs (above) but it still doesn't actually set the Docker registry. This file get mutated by gradle to include some comments but that's about it.

[internal_user@e274a182637b workspace]$ cat /home/internal_user/.testcontainers.properties
registry.url=internal.docker.registry
[internal_user@e274a182637b workspace]$ cat gradle.properties
uriBase = https://internal.binary.store/artifactory
registry.url=internal.docker.registry

This is the test I'm trying to run:

[internal_user@e274a182637b workspace]$ cat src/test/java/test/containers/TestContainersTest.java
package test.containers;

import java.util.Map;

import org.junit.ClassRule;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestInstance;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.junit.jupiter.Testcontainers;

import io.micronaut.context.ApplicationContext;
import io.micronaut.context.env.PropertySource;
import io.micronaut.runtime.server.EmbeddedServer;

@Testcontainers
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class TestContainersTest {

    @ClassRule
    public static GenericContainer rabbitMq = new GenericContainer<>("internal.docker.registry/bu/rabbitmq:management-alpine")
            .withExposedPorts(5672,15672);

    private static EmbeddedServer server;

    @BeforeAll
    protected static void setupServer() {

        Map<String, Object> contextInitProperties = Map.of(
                "rabbitmq.servers.server-noms.uri", "amqp://" + rabbitMq.getContainerIpAddress() + ":5672"
        );

        server = ApplicationContext.run(EmbeddedServer.class, PropertySource.of("test", contextInitProperties));
    }

    @AfterAll
    protected static void stopServer() {

        if (server != null) {
            server.stop();
        }
    }

    @Test
    public void test_TestContainers() {

        TestPublishClient client = server.getApplicationContext().getBean(TestPublishClient.class);

        client.publishUpdate("Test message".getBytes());
    }
}

The environment is very minimal:

[internal_user@e274a182637b workspace]$ env
HOSTNAME=e274a182637b
TERM=xterm
USER=internal_user
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/workspace
JAVA_HOME=/usr/lib/jvm/java-11-openjdk
HOME=/home/internal_user
GRADLE_USER_HOME=/tmp
SHLVL=1
LESSOPEN=||/usr/bin/lesspipe.sh %s
_=/usr/bin/env

[rnorth]

Oh, OK. @austin127001 based on your logs do you have the same issue? @ajatman this may be a different problem.

If you both have this common factor, then you might have related issues: @austin127001 do you also have firewall restrictions preventing you from reaching index.docker.io?

[ajatman]

@rnorth sorry for the confusion: @austin127001 == @ajatman (account hell)

Exactly, I have no ability to reach outside of the internal network.

I had some success in setting the actual properties that I'm concerned by:

$ cat build.gradle
...
test {
    useJUnitPlatform()
    systemProperty 'registry.url', findProperty('registry.url')
}
bash-4.2$ cat gradle.properties
registry.url = internal.docker.registry

But this is the only mechanism I've found to have the properties recognized.

So that's the first hurdle dealt with, however it seems like a lot of configuration that "should" work doesn't seem to :/.

What seems very odd to me is that I only seem to be able to override the images (https://www.testcontainers.org/features/configuration/#customizing-images) by specifying their values in ${HOME}/.testcontainer.properties. I can't seem to get them set any other way.

Ideally: I would like to have all properties required for these builds to be encapsulated in the repository itself (no reliance on how the user has anything set up).
How can a specify these as a local properties file as oppose to some user setting?

[rnorth]

Right, OK. You're on the right track with looking at customizing the images Testcontainers is using. That's the first step in preventing it from reaching out to public registries.

testcontainers.properties is loaded from the classpath first, and then supplemented with ~/.testcontainers.properties from the file system. (Note no . preceding the classpath filename).

So you really should be able to modify the tinyimage.container.image and ryuk.container.image to point to your private registry via this classpath config, so that the configuration is indeed encapsulated within your project.

Have you tried creating a file named src/main/resources/testcontainers.properties with these image name overrides?

If we can get it to a point where Testcontainers isn't reaching out to Docker Hub, then that would be progress! Hopefully the private registry authentication should be activated after that, and you should see these images start to be pulled from your private registry.

Thanks for your patience.

[ajatman]

@rnorth not at all, thanks for your assistance with this!

That was the ticket! For anyone else attempting a similar setup (internal only, inside a Docker container with the unix sock mounted):

• Set any internal images via ~/.testcontainers.properties or src/main/resources/testcontainers.properties
• Set any docker-java properties via the modification of the test task (e.g. test { systemProperty 'registry.url', findProperty('registry.url') } with the appropriate props passed a build/test. Others may have luck setting these via the Docker config but I was unable to get this to happen.
• Have environment variable TESTCONTAINERS_RYUK_DISABLED=true (assuming container will handle reaping)
• Follow https://www.testcontainers.org/supported_docker_environment/continuous_integration/dind_patterns/

[bsideup]

⚠️⚠️⚠️

Have environment variable TESTCONTAINERS_RYUK_DISABLED=true

This should really be avoided, especially if you mount the socket.

assuming container will handle reaping

The container will not handle the reaping. Only the shutdown hook will do the "best effort cleanup", but there are many situations when it won't happen (kill -9 is only one of them).

[rnorth]

Glad to hear you got it working. As @bsideup says, please only disable Ryuk as a last resort!

I think we should update the docs to make private registry usage clearer, so let's leave this issue open as a reminder.

[ajatman]

@bsideup & @rnorth Thanks for all the helpful suggestions!

I'm running into an issue when attempting to employ ryuk as part of this process. Specifically it reports:

17:01:31.899 [DEBUG] [TestEventLogger]     17:01:31.896 [Test worker] DEBUG o.t.utility.RegistryAuthLocator - Looking up auth config for image: internal.docker.registry/alpine:3.6
17:01:31.899 [DEBUG] [TestEventLogger]     17:01:31.896 [Test worker] DEBUG o.t.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /home/internal_user/.docker/config.json (does not exist) and commandPathPrefix:
17:01:31.899 [DEBUG] [TestEventLogger]     17:01:31.896 [Test worker] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: internal.docker.registry/alpine:3.6, configFile: /home/internal_user/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/internal_user/.docker/config.json (No such file or directory)
17:01:31.899 [DEBUG] [TestEventLogger]     17:01:31.897 [Test worker] DEBUG o.t.d.a.AuthDelegatingDockerClientConfig - Effective auth config [null]
17:01:31.912 [DEBUG] [TestEventLogger]     17:01:31.907 [Test worker] DEBUG c.g.d.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.command.CreateContainerCmdImpl@607ca2c0[name=<null>,hostName=<null>,domainName=<null>,user=<null>,attachStdin=<null>,attachStdout=<null>,attachStderr=<null>,portSpecs=<null>,tty=<null>,stdinOpen=<null>,stdInOnce=<null>,env=<null>,cmd={sh,-c,ip route|awk '/default/ { print $3 }'},entrypoint=<null>,image=internal.docker.registry/alpine:3.6,volumes=com.github.dockerjava.api.model.Volumes@293bb2ec,workingDir=<null>,macAddress=<null>,networkDisabled=<null>,exposedPorts=com.github.dockerjava.api.model.ExposedPorts@2cd46fa9,stopSignal=<null>,hostConfig=com.github.dockerjava.api.model.HostConfig@a69cf3d[binds=<null>,blkioWeight=<null>,blkioWeightDevice=<null>,blkioDeviceReadBps=<null>,blkioDeviceReadIOps=<null>,blkioDeviceWriteBps=<null>,blkioDeviceWriteIOps=<null>,memorySwappiness=<null>,capAdd=<null>,capDrop=<null>,containerIDFile=<null>,cpuPeriod=<null>,cpuShares=<null>,cpuQuota=<null>,cpusetCpus=<null>,cpusetMems=<null>,devices=<null>,diskQuota=<null>,dns=<null>,dnsSearch=<null>,extraHosts=<null>,links=<null>,logConfig=<null>,lxcConf=<null>,memory=<null>,memorySwap=<null>,memoryReservation=<null>,kernelMemory=<null>,networkMode=<null>,oomKillDisable=<null>,autoRemove=<null>,oomScoreAdj=<null>,portBindings=<null>,privileged=<null>,publishAllPorts=<null>,readonlyRootfs=<null>,restartPolicy=<null>,ulimits=<null>,volumesFrom=<null>,pidMode=<null>,securityOpts=<null>,cgroupParent=<null>,volumeDriver=<null>,shmSize=<null>,pidsLimit=<null>,runtime=<null>,tmpFs=<null>],labels={org.testcontainers=true, org.testcontainers.sessionId=c8929d74-b61a-46ec-9fbf-c9584ec8a494},networkingConfig=<null>,ipv4Address=<null>,ipv6Address=<null>,aliases=<null>,authConfig=<null>,execution=com.github.dockerjava.core.exec.CreateContainerCmdExec@5a573d64]
17:01:31.991 [DEBUG] [TestEventLogger]     17:01:31.990 [Test worker] DEBUG c.g.d.core.command.AbstrDockerCmd - Cmd: 9135f4fe3a08da65407fd7ad141d6cca3e69def1d6e7477ed014fcb6debf6106,com.github.dockerjava.core.exec.StartContainerCmdExec@602075c1
17:01:32.208 [DEBUG] [TestEventLogger]     17:01:32.208 [tc-okhttp-stream-1253780825] DEBUG c.g.d.c.c.LogContainerResultCallback - STDOUT: 172.17.0.1
17:01:32.272 [DEBUG] [TestEventLogger]     17:01:32.271 [Test worker] DEBUG c.g.d.core.command.AbstrDockerCmd - Cmd: 9135f4fe3a08da65407fd7ad141d6cca3e69def1d6e7477ed014fcb6debf6106,true,true,com.github.dockerjava.core.exec.RemoveContainerCmdExec@cab136e
17:01:32.322 [DEBUG] [TestEventLogger]     17:01:32.320 [Test worker] INFO  o.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
17:01:32.322 [DEBUG] [TestEventLogger]     17:01:32.321 [Test worker] DEBUG c.g.d.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.exec.InfoCmdExec@6d2b25c6
17:01:32.360 [DEBUG] [TestEventLogger]     17:01:32.359 [Test worker] DEBUG c.g.d.core.command.AbstrDockerCmd - Cmd: com.github.dockerjava.core.exec.VersionCmdExec@34bcc241
17:01:32.365 [DEBUG] [TestEventLogger]     17:01:32.362 [Test worker] INFO  o.testcontainers.DockerClientFactory - Connected to docker:
17:01:32.365 [DEBUG] [TestEventLogger]       Server Version: 18.06.1-ce
17:01:32.365 [DEBUG] [TestEventLogger]       API Version: 1.38
17:01:32.365 [DEBUG] [TestEventLogger]       Operating System: CentOS Linux 7 (Core)
17:01:32.365 [DEBUG] [TestEventLogger]       Total Memory: 50154 MB
17:01:32.365 [DEBUG] [TestEventLogger]     17:01:32.363 [Test worker] DEBUG c.g.d.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=internal.docker.registry/testcontainers/ryuk:0.2.3 ,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@517ed480]
17:01:32.390 [DEBUG] [TestEventLogger]     17:01:32.378 [Test worker] DEBUG o.t.utility.RegistryAuthLocator - Looking up auth config for image: internal.docker.registry/testcontainers/ryuk:0.2.3
17:01:32.391 [DEBUG] [TestEventLogger]     17:01:32.378 [Test worker] DEBUG o.t.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /home/internal_user/.docker/config.json (does not exist) and commandPathPrefix:
17:01:32.391 [DEBUG] [TestEventLogger]     17:01:32.378 [Test worker] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: internal.docker.registry/testcontainers/ryuk:0.2.3 , configFile: /home/internal_user/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/internal_user/.docker/config.json (No such file or directory)
17:01:32.391 [DEBUG] [TestEventLogger]     17:01:32.378 [Test worker] DEBUG o.t.d.a.AuthDelegatingDockerClientConfig - Effective auth config [null]
17:01:32.391 [DEBUG] [TestEventLogger]     17:01:32.383 [tc-okhttp-stream-2125521173] ERROR c.g.d.c.async.ResultCallbackTemplate - Error during callback
17:01:32.391 [DEBUG] [TestEventLogger]     com.github.dockerjava.api.exception.BadRequestException: {"message":"invalid reference format"}
17:01:32.391 [DEBUG] [TestEventLogger]
17:01:32.391 [DEBUG] [TestEventLogger]          at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:266)
17:01:32.391 [DEBUG] [TestEventLogger]          at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$1(OkHttpInvocationBuilder.java:292)
17:01:32.391 [DEBUG] [TestEventLogger]          at java.base/java.lang.Thread.run(Thread.java:834)

So the alpine image is working just fine (and has been) but ryuk seems to be another story.

I believe this is due to the ryuk not having access to the /var/run/docker.sock:

[internal_user@9ac425cb3bfe workspace]$ docker run -it internal.docker.registry/testcontainers/ryuk:0.2.3
2019/08/06 16:14:53 Starting on port 8080...
panic: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

goroutine 1 [running]:
main.main()
    /go/src/github.com/testcontainers/moby-ryuk/main.go:31 +0xe2e

Which makes sense as adding the volume mount seems to make it work:

[internal_user@9ac425cb3bfe workspace]$ docker run -v /var/run/docker.sock:/var/run/docker.sock -it internal.docker.registry/testcontainers/ryuk:0.2.3
2019/08/06 16:26:32 Starting on port 8080...
^C

So it seems that I need to somehow tell testcontainers that it needs to also mount the /var/run/docker.sock to the ryuk container that get's spawned? Is this possible?
I've attempted ryuk.container.privileged = false but this doesn't resolve the problem it seems.

Thanks again for your assistance with this! I can make this a new ticket (as the problem is different)? It still seems associated with the goal of: How do I use this in an isolated environment.

[FranckKe]

I had trouble setting up testcontainers with private registry in Gitlab CI as well but ended up working.

What I tried:

• Configured docker-java through sytem properties
• Had issues with Ryuk, disabled it

But then docker-java or testcontainer, not sure which, tried to pull all images through our private registry without fallbacking on dockerhub, thus failing on pulling regular images.

I stopped at this point and created a custom image with dind and maven packaged together which works fine:

Dockerfile

FROM docker:19.03.12-dind # Alpine 3.11

RUN apk update \
    && apk add --no-cache \
    openjdk8=8.242.08-r2 \
    gcompat=0.4.0-r1 \
    maven=3.6.3-r0

.gitlab-ci.yaml

java_test:
  # This image does not exists, this is an example
  image: dind-maven:3.6.3-jdk-8 
  script:
    # Configure your secrets in the ci settings
    - docker login 
      -u "${DOCKER_PRIVATE_REGISTRY_USER}" 
      -p "${DOCKER_PRIVATE_REGISTRY_PASSWORD}" 
      "${DOCKER_PRIVATE_REGISTRY_URL}"
    - mvn test
...

[HaMatthias]

Having similiar issues by using a private registry.

I want to pull the images like ryuk from our private registry, by customizing the image in the testcontainers.properties. This isn't a problem at all. Testcontainers is pulling those images from our private registry.

Furthermore, I want to avoid using the path /home/user/.docker/config.json for my config.json and put it in a custom location. For the Docker-Compose-Module this is possible, by editing the dockerConfigFile-Property (https://www.testcontainers.org/modules/docker_compose/#using-private-repositories-in-docker-compose), but not for the images given by Testcontainers like mentioned in https://www.testcontainers.org/features/configuration/#customizing-images.

Is there any way to provide a custom location for the auth-config (config.json) to pull images from private registry?

[bastiat]

I have testcontainers.properties in src/test/resources with
registry.url=https://nexus.local:443/
This local repository doesn't contain external rabbitmq image, but I see other problem in logs - it tries to access https://registry-1.docker.io/v2/ although in testcontainers.properties local repo is configured registry.url=https://nexus.local:443/
On this gitlab runner there is of course no internet access.

In logs I can see

21:31:05.190 [main] INFO org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 18.09.7
  API Version: 1.39
  Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
  Total Memory: 15867 MB
21:31:05.190 [main] DEBUG org.testcontainers.DockerClientFactory - Ryuk is enabled
21:31:05.193 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=testcontainersofficial/ryuk:0.3.0,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0]
21:31:05.219 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainersofficial/ryuk:0.3.0 at registry: index.docker.io
21:31:05.220 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /opt/gitlab-runner/.docker/config.json (exists) and commandPathPrefix: 
21:31:05.223 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [index.docker.io] for dockerImageName [testcontainersofficial/ryuk:0.3.0]
21:31:05.223 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
21:31:05.223 [main] DEBUG org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig - Effective auth config [null]
21:31:20.236 [tc-okhttp-stream-278166606] ERROR com.github.dockerjava.api.async.ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
    at com.github.dockerjava.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:293)
    at com.github.dockerjava.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$4(OkHttpInvocationBuilder.java:317)
    at java.base/java.lang.Thread.run(Thread.java:834)
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 15.633 s <<< FAILURE! - in acme.FooTest
[ERROR] acme.FooTest  Time elapsed: 15.633 s  <<< ERROR!
org.testcontainers.containers.ContainerLaunchException: Container startup failed
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=rabbitmq:3-management, imagePullPolicy=DefaultPullPolicy())
Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: 
{"message":"Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}

What is prcecedence of config files? Is testcontainers.properties from src/test/resources read when there is a log RegistryAuthLocator has configFile: /opt/gitlab-runner/.docker/config.json (exists) and commandPathPrefix:?

If there are not appropriate images in local repository, does it start searching in https://registry-1.docker.io/v2/?

What could I try? How can I investigate?

[bsideup]

@bastiat registry.url in testcontainers.properties makes no sense and not supported.

See https://www.testcontainers.org/features/image_name_substitution/#automatically-modifying-docker-hub-image-names for the available options or use docker-java's config (not recommended).

Closing this issue as the automatic prefixing is available now.

[bastiat]

@bsideup where can I configure registry.url?
How can I make sure, that this registry.url configuration is used by testcontainers?

[bsideup]

registry.url is a custom configuration property of docker-java, not Testcontainers. Please check with the docker-java project on how to configure it.

Also nore that we do not recommend using it. Instead, configure the image prefix as described in the docs (see the link I provided), or use a custom image name substitutor.

[bastiat]

@bsideup thx for reply
I checked the link you provided.
-bash-4.2$ echo $TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX
shows link to our internal docker repository host.internal:8443
but in logs I still see

13:22:33.820 [main] DEBUG org.testcontainers.DockerClientFactory - Ryuk is enabled
13:22:33.822 [main] DEBUG com.github.dockerjava.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=testcontainersofficial/ryuk:0.3.0,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0]
13:22:33.868 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - Looking up auth config for image: testcontainersofficial/ryuk:0.3.0 at registry: index.docker.io
13:22:33.868 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - RegistryAuthLocator has configFile: /opt/gitlab-runner/.docker/config.json (exists) and commandPathPrefix: 
13:22:33.872 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - registryName [index.docker.io] for dockerImageName [testcontainersofficial/ryuk:0.3.0]
13:22:33.872 [main] DEBUG org.testcontainers.utility.RegistryAuthLocator - No matching Auth Configs - falling back to defaultAuthConfig [null]
13:22:33.873 [main] DEBUG org.testcontainers.dockerclient.auth.AuthDelegatingDockerClientConfig - Effective auth config [null]
13:22:48.907 [tc-okhttp-stream-2096598149] ERROR com.github.dockerjava.api.async.ResultCallbackTemplate - Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"}
    at com.github.dockerjava.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:293)

Can the problem be caused by that in our docker repo there is no testcontainersofficial/ryuk:0.3.0?
Must testcontainersofficial/ryuk:0.3.0 also be placed in internal docker repo?
How can I check if TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX env variable is used by testcontainers?

[bsideup]

@bastiat check logs for ImageNameSubstitutor. Also, since I see testcontainersofficial in the logs, it most probably means that you're not using the latest version of Testcontainers. Please always try the latest version before reporting issues.