Testcontainers-java: Wrong gateway is detected starting with Testcontainers 1.12.4

[ghostbuster91]

Hi,

I have a jenkins CI with a master node and two slaves. All tests are running smoothly on slaves but they fail once run on master. Master and slaves differs in terms of ubuntu version and docker version.
The master is:
```Server Version: 19.03.4
API Version: 1.40
Operating System: Ubuntu 18.04.3 LTS

While the slaves are:

Server Version: 19.03.0
API Version: 1.40
Operating System: Ubuntu 16.04.6 LTS

I use testcontainers 1.12.4

Exceptions I am getting on master node:

13:11:08 [info] org.testcontainers.containers.ContainerLaunchException: Container startup failed
13:11:08 [info] at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:322)
13:11:08 [info] at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:302)
13:11:08 [info] at com.dimafeng.testcontainers.SingleContainer.start(Container.scala:42)
13:11:08 [info] ...
13:11:08 [info] Cause: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageNameFuture=java.util.concurrent.CompletableFuture@687cf499[Completed normally], imagePullPolicy=DefaultPullPolicy(), dockerClient=LazyDockerClient.INSTANCE)
13:11:08 [info] at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1265)
13:11:08 [info] at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:600)
13:11:08 [info] at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:311)
13:11:08 [info] at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:302)
13:11:08 [info] at com.dimafeng.testcontainers.SingleContainer.start(Container.scala:42)
13:11:08 [info] ...
13:11:08 [info] Cause: java.lang.IllegalStateException: Can not connect to Ryuk
13:11:08 [info] at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:150)
13:11:08 [info] at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:135)
13:11:08 [info] at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
13:11:08 [info] at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
13:11:08 [info] at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
13:11:08 [info] at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
13:11:08 [info] at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
13:11:08 [info] at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:62)
13:11:08 [info] at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:25)
13:11:08 [info] at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)

and

12:11:37.140 [testcontainers-ryuk] WARN o.t.u.ResourceReaper - Can not connect to Ryuk at localhost:32965
13:11:37 java.net.SocketException: Connection reset
13:11:37 at java.net.SocketInputStream.read(SocketInputStream.java:210)
13:11:37 at java.net.SocketInputStream.read(SocketInputStream.java:141)
13:11:37 at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
13:11:37 at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
13:11:37 at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
13:11:37 at java.io.InputStreamReader.read(InputStreamReader.java:184)
13:11:37 at java.io.BufferedReader.fill(BufferedReader.java:161)
13:11:37 at java.io.BufferedReader.readLine(BufferedReader.java:324)
13:11:37 at java.io.BufferedReader.readLine(BufferedReader.java:389)
13:11:37 at org.testcontainers.utility.ResourceReaper$FilterRegistry.waitForAcknowledgment(ResourceReaper.java:406)
13:11:37 at org.testcontainers.utility.ResourceReaper$FilterRegistry.register(ResourceReaper.java:402)
13:11:37 at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:128)
13:11:37 at java.lang.Thread.run(Thread.java:748)

Although when I call docker logs on master node 

• docker ps
  CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
  7b115c90863a quay.io/testcontainers/ryuk:0.2.3 "/app" 32 seconds ago Up 31 seconds 0.0.0.0:32965->8080/tcp testcontainers-ryuk-75c3e431-6dd5-4faf-9b54-df86d395ff4e
  ++ docker ps -q
• docker logs 7b115c90863a
  2020/01/12 10:08:16 Starting on port 8080...
  ```
  I can see that ryuk has started.

I assume that this is not a fault of ubuntu neither of docker version (mainly because I use similar versions on my local machine and it all works), but rather some specific configuration of master node. Unfortunately I have no idea how to debug it further, so any help would be appreciated.

Cheers!

[worldtiki]

I have the same issue. Downgrading to 1.12.3 seems to solve this.

[bsideup]

@worldtiki is it with 1.12.5 or 1.12.4?

[worldtiki]

@worldtiki is it with 1.12.5 or 1.12.4?

I have this issue with 1.12.5. I'm unable to test with 1.12.4 because of https://github.com/testcontainers/testcontainers-java/pull/2119

It's difficult for me to troubleshoot as I don't have access to the jenkins slaves where my tests run, but my guess is that they have different configurations of versions of docker as the error is intermittent (likely because it only affects some of the agents).

[bsideup]

@worldtiki you can test 1.12.4 just by calling DockerClientFactory.instance().client() somewhere before you start LocalStackContainer

[worldtiki]

So, quick update. With 1.12.4 it works ok (with the hack you suggested).

With 1.12.5 this is the error:

17:14:20.548 [main] INFO  o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
17:14:20.553 [main] INFO  o.t.d.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock
17:14:20.769 [main] INFO  o.testcontainers.DockerClientFactory - Docker host IP address is localhost
17:14:20.790 [main] INFO  o.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 18.09.9-ce
  API Version: 1.39
  Operating System: Amazon Linux 2
  Total Memory: 31367 MB
17:14:21.070 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.3, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:14:21.982 [testcontainers-ryuk] WARN  o.t.utility.ResourceReaper - Can not connect to Ryuk at localhost:53414
java.net.ConnectException: Connection refused (Connection refused)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    at java.base/java.net.Socket.connect(Socket.java:609)
    at java.base/java.net.Socket.connect(Socket.java:558)
    at java.base/java.net.Socket.<init>(Socket.java:454)
    at java.base/java.net.Socket.<init>(Socket.java:231)
    at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:114)
    at java.base/java.lang.Thread.run(Thread.java:834)

With 1.12.4

17:27:23.493 [main] INFO  o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
17:27:23.498 [main] INFO  o.t.d.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock
17:27:23.986 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: alpine:3.5, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:27:25.065 [main] INFO  o.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
17:27:25.088 [main] INFO  o.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 18.09.9-ce
  API Version: 1.39
  Operating System: Amazon Linux 2
  Total Memory: 31367 MB
17:27:25.316 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.3, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:27:25.939 [main] INFO  o.testcontainers.DockerClientFactory - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
        ?? Checking the system...
        ? Docker version should be at least 1.6.0
        ? Docker environment should have more than 2GB free disk space
17:27:26.466 [main] INFO  docker[localstack/localstack:0.9.4] - HOSTNAME_EXTERNAL environment variable set to 172.17.0.1 (to match host-routable address for container)
17:27:26.468 [main] INFO  docker[localstack/localstack:0.9.4] - Creating container for image: localstack/localstack:0.9.4
17:27:26.468 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: localstack/localstack:0.9.4, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:27:26.695 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: alpine:3.5, configFile: /home/jenkins/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /home/jenkins/.docker/config.json (No such file or directory)
17:27:27.803 [main] INFO  docker[localstack/localstack:0.9.4] - Starting container with ID: 6075f529b3e1837404eda90ca0fcdc5be830abe59534f719ab71cc999646a94b

[ddierickx]

Can confirm that I have a similar regression with 1.12.5.

For the DockerComposeContainer, it did not use the correct Docker hostname when running inside docker (gitlab). Instead it used localhost. For a regular DockerContainer this worked fine and returned the correct hostname (IP address iso. localhost). Reverting to 1.12.4 fixed the issue for me.

[bsideup]

@ddierickx thanks, we will look at it. I wonder if this change is the one to blame:
https://github.com/testcontainers/testcontainers-java/pull/2151

[bsideup]

@ddierickx can you debug it on your side? Here are the steps:

1. Use 1.12.4
2. Obtain DockerClient by using DockerClientFactory.instance().client()
3. Run the following code:

var network = client.inspectNetworkCmd()
                         .withNetworkId("bridge")
                         .exec();
System.out.println(network);

var gateway = network
                         .getIpam()
                         .getConfig()
                         .stream()
                         .filter(it -> it.getGateway() != null)
                         .findAny()
                         .map(Network.Ipam.Config::getGateway)
                         .orElse("localhost");

assertEquals(DockerClientProviderStrategy.getDockerHostIpAddress(), gateway);

Please do share the log output if it fails

[ddierickx]

Changed the code a little bit since DockerClientProviderStrategy is not in 1.12.4, so I used EnvironmentAndSystemPropertyClientProviderStrategy instead.

        val client = DockerClientFactory.instance().client()
        val network = client.inspectNetworkCmd()
            .withNetworkId("bridge")
            .exec();
        println("Network: $network");

        val gateway = network
            .getIpam()
            .getConfig()
            .stream()
            .filter { it -> it.getGateway() != null }
            .findAny()
            .map { it.gateway }
            .orElse("localhost");

        System.out.println("Gateway: $gateway")

        EnvironmentAndSystemPropertyClientProviderStrategy()
            .getDockerHostIpAddress()
            .shouldBeEqualTo(gateway)

The output when running on our gitlab (with support for docker-in-docker by passing the docker socket) is:

Network: com.github.dockerjava.api.model.Network@4947a298[id=0742db67f2ff5b4706b8a3340248cd27e6d93c785304116e3c9b4911c1ea5a36
    ,name=bridge
    ,scope=local
    ,driver=bridge
    ,enableIPv6=false
    ,internal=false
    ,ipam=com.github.dockerjava.api.model.Network$Ipam@66efc28c[driver=default
    ,config=[com.github.dockerjava.api.model.Network$Ipam$Config@6ada782c]
    ,options=<null>]
    ,containers={b5de758269581432f830f7e78f8fe1e79ea5921eab554415f0c87102f25c90f9=com.github.dockerjava.api.model.Network$ContainerNetworkConfig@2fd37407[endpointId=afebcc4c3cf2e94b4309ccd2ee494ff73cdd9b7b0c4e44b360c93ef0f8084dc0
    ,macAddress=02:42:ac:11:00:02
    ,ipv4Address=172.17.0.2/16
    ,ipv6Address=]
    , edbef2cd0d08fca3f54e9149d0a3fdae91c76bafb1b8245bab101384ed450f7d=com.github.dockerjava.api.model.Network$ContainerNetworkConfig@5bf45007[endpointId=f20d40ccd6067864805411fd14c3d030f9be8fc8d5aed144b60c6540b9eed902
    ,macAddress=02:42:ac:11:00:03
    ,ipv4Address=172.17.0.3/16
    ,ipv6Address=]}
    ,options={com.docker.network.bridge.default_bridge=true
        , com.docker.network.bridge.enable_icc=true
        , com.docker.network.bridge.enable_ip_masquerade=true
        , com.docker.network.bridge.host_binding_ipv4=0.0.0.0
        , com.docker.network.bridge.name=docker0
        , com.docker.network.driver.mtu=1500}
        ,attachable=false
        ,labels={}]
Gateway: localhost
java.lang.AssertionError: Expected localhost actual 172.17.0.1.

Thanks for your help.

[NersesAM]

We have the same problem
1.12.5+ fails with WARN org.testcontainers.utility.ResourceReaper Can not connect to Ryuk at localhost:32768

INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock
INFO  org.testcontainers.DockerClientFactory Docker host IP address is localhost
INFO org.testcontainers.DockerClientFactory Connected to docker: 
Server Version: 18.09.7
API Version: 1.39
Operating System: Ubuntu 18.04.3 LTS (containerized)

1.12.4
```
INFO org.testcontainers.dockerclient.DockerClientProviderStrategy Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///var/run/docker.sock
INFO org.testcontainers.DockerClientFactory Docker host IP address is 172.18.0.1
INFO org.testcontainers.DockerClientFactory Connected to docker:
Server Version: 18.09.7
API Version: 1.39
Operating System: Ubuntu 18.04.3 LTS (containerized)

[bsideup]

@ddierickx thanks!

config=[com.github.dockerjava.api.model.Network$Ipam$Config@6ada782c]

it seems that docker-java did not add toString to Network$Ipam$Config.

Could you please try adding the following?

println("Network: ${JsonOutput.toJson(network)}");

[NersesAM]

@bsideup this is what I get on our CI

{
    "Id": "466fb33eef145fba9793a3bdbaef8b595df2267b5de806b4647485cd1ab827aa",
    "Name": "bridge",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "Internal": false,
    "IPAM": {
        "Driver": "default",
        "Config": [
            {
                "Subnet": "172.18.0.0/16",
                "IPRange": null,
                "Gateway": null,
                "NetworkID": null
            }
        ],
        "Options": null
    },
    "Containers": {
        "85e912f17b5501f6a683a70369bba46b1645a9a06c1e6e22388a7e051e918229": {
            "EndpointID": "<removed>",
            "MacAddress": "<removed>",
            "IPv4Address": "172.18.0.3/16",
            "IPv6Address": ""
        },
        "9391acd00344eaabf34f40f2c7116bde5f2209475ec8cdd968b03e345cbd6581": {
            "EndpointID": "<removed>",
            "MacAddress": "<removed>",
            "IPv4Address": "172.18.0.2/16",
            "IPv6Address": ""
        }
    },
    "Options": {
        "com.docker.network.bridge.default_bridge": "true",
        "com.docker.network.bridge.enable_icc": "true",
        "com.docker.network.bridge.enable_ip_masquerade": "true",
        "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
        "com.docker.network.bridge.name": "docker0",
        "com.docker.network.driver.mtu": "1500"
    },
    "Attachable": false,
    "Labels": {}
}

[bsideup]

@NersesAM thanks! It looks like a regression indeed. Not sure why Gateway is null, but I guess our assumption was incorrect and we need to fallback to running a container. Oh well... 😅

[bsideup]

@NersesAM @ddierickx @ghostbuster91 @worldtiki I just pushed the following change, it should fix it:
https://github.com/testcontainers/testcontainers-java/pull/2429

You can get it via Jitpack as described here:
https://www.testcontainers.org/jitpack_dependencies/

Something like:

repositories {
     maven { url 'https://jitpack.io' }
}

dependencies {
    testCompile 'com.github.testcontainers.testcontainers-java:testcontainers:PR2429-SNAPSHOT`
}

[NersesAM]

@bsideup Can confirmed the snapshot build worked on our CI. Thanks for the fix

[bsideup]

@NersesAM thanks a lot for trying! We will release it soon, together with #2297 and a few other neat PRs :)

[captainju]

Is it fixed with the 1.13.0 release ?

[bsideup]

@captainju fixed in 1.14.0

[SudhakarKonduru]

I'm getting the same issue. I tied with 1.14.0, 1.14.3
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.068 s <<< FAILURE! - in package.DaoTCTest
[ERROR] package.ReadPlanDaoTCTest Time elapsed: 0.006 s <<< ERROR!
org.testcontainers.containers.ContainerLaunchException: Container startup failed
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=postgres:12.3, imagePullPolicy=DefaultPullPolicy())
Caused by: com.github.dockerjava.api.exception.NotFoundException:
{"message":"network bridge not found"}

Appreciate any help

[saeed-aghaee]

I'm having the same issue with 1.14.3 and Jenkins.

{"message":"network bridge not found"}

[rnorth]

@SudhakarKonduru @saeed-aghaee
Yikes, I almost missed this (comments on closed issues tend not to get noticed). Please could you raise a new issue, as I think this is a new bug related to, but not the same as this one.

[SudhakarKonduru]

Thank you. Created new one: https://github.com/testcontainers/testcontainers-java/issues/2921