Testcontainers-java: Can't connect to Ryuk container when using remote Docker host

Created on 2 Feb 2018  路  43Comments  路  Source: testcontainers/testcontainers-java

Version 1.6.0
Not sure if this came from multiple threads but never seen it with single thread.
Many tests use several types of containers.

java.lang.ExceptionInInitializerError
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:58)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:69)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:48)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
    at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
    at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:105)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:355)
    at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:158)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:116)
    at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:126)
    at org.testcontainers.containers.JdbcDatabaseContainer.<init>(JdbcDatabaseContainer.java:35)
    at org.testcontainers.containers.PostgreSQLContainer.<init>(PostgreSQLContainer.java:28)
    at com.tests.container.DbContainer.<init>()
    at com.tests.MyTest.<clinit>()
    ... 27 more
resolutioacknowledged
Source
picture alexcase52
👍7

Most helpful comment

Maybe I'm wrong but this could be a 'known docker bug'.
I experienced several issues with this before.

Long story short:
Everyone can access docker-proxied port, except for container in the same host.
Basically two containers with ip 172.17.0.2 and 172.17.0.3 with exposed ports can't connect to each other on these ports using docker host ip 172.17.0.1:(exposedport)
You get 'No route to host'.

It is a docker/firewall/iptables issue.

You have to permit it on iptables in filter/INPUT chain an exact ip or docker network.
FirewallD example

firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -s 172.17.0.0/24 -j ACCEPT firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT_direct 0 -s 172.17.0.0/24 -j ACCEPT

Or iptables example:
iptables -t filter -I INPUT 1 -s 172.17.0.0/24 -j ACCEPT

Where 172.17.0.0/24 is default docker network out of box.

This will INSERT rule in filter/INPUT chain as FIRST one.
To delete:
iptables t filter -D INPUT 1

To test it, just run two containers like centos (install telnet there) and nginx with exposed port.
And try to connect via telnet to your nginx server via docker-host ip and nginx exposed port.
Keep in mind, ping always works, don't rely on that.

Just FYI my docker-compose.yml, config.toml and gitlab-ci.yml in attached zip file.

1.zip

picture ghost on 2 Aug 2019
👍4

All 43 comments

Hi @alexcase52,

Could you please try with 1.5.1 + multiple threads? I want to make sure that this is a regression in 1.6.0 :)

bsideup picture bsideup on 2 Feb 2018

Hi @alexcase52,

I get also the error only on 1.6.0, 1.5.1 is fine.
Testing on Windows 10 with Java9.

Is deamon exposition on tcp://xy:2375 without TLS mandatory ?

llfbandit picture llfbandit on 7 Feb 2018

@llfbandit

Is deamon exposition on tcp://xy:2375 without TLS mandatory ?

This is currently still mandatory (until we will be able to support npipes or Docker for Windows will support Unix sockets).

kiview picture kiview on 7 Feb 2018

https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/

Well... kind of tiny scope for now...

llfbandit picture llfbandit on 7 Feb 2018

I'm experiencing a similar issue when running on Jenkins, with a DOCKER_HOST pointing to some other host:

  2018-04-03 17:59:19.719  WARN --- [containers-ryuk[] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at 10.17.48.167:32800
  java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:111)
    at org.testcontainers.utility.ResourceReaper$$Lambda$41/1921545475.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)

Downgrading to 1.5.1 fixes the issue.

alvarosanchez picture alvarosanchez on 4 Apr 2018
👍4

@alvarosanchez
Thanks for the info, is 10.17.48.167 the ip of your DOCKER_HOST? Should be. Could this be a problem with a firewall?
I assume unrelated, but I've experienced this problem when inside a network with a captive portal, so this can be some DNS hickup.

kiview picture kiview on 14 May 2018

is 10.17.48.167 the ip of your DOCKER_HOST?

Yes, it is.

Could this be a problem with a firewall?

No, because then version 1.5.1 wouldn't work either. Note that the client machine (a Jenkins instance) is able to run Docker commands on that DOCKER_HOST via docker CLI and also the Gradle plugin.

alvarosanchez picture alvarosanchez on 14 May 2018

@alvarosanchez
With version 1.6.0 we've introduced a new implementation for the ResourceReaper component, which will now run outside of the JVM in its own Docker container (the container is called Ryuk and the error appears when trying to connect to this container).
Since we don't have an automated CI build for this scenario (remote DOCKER_HOST), I'm not sure if this implementation broke the compatibility with some remote scenarios.

I've changed the title of the issue to better reflect the current state of knowledge we have, I hope it's okay for everyone involved?

@alexcase52 Were you able to find out if this issue is related to multiple Gradle threads?

kiview picture kiview on 14 May 2018
👍1

@alvarosanchez
Have you tried using 1.7.0? Maybe the container startup ran into a timeout as mentioned in #621.

kiview picture kiview on 15 May 2018

@kiview 1.7.2 :)

bsideup picture bsideup on 15 May 2018

Hi, Just encountered this on 1.7.3 running a test on our Jenkins server:

9:26:52.538 [main] INFO  org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
09:26:52.555 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=tcp://NAME_OF_MY_DOCKER_HOST:2375
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='jenkins'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=tcp://NAME_OF_MY_DOCKER_HOST:2375,registryUsername=jenkins,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/var/lib/jenkins/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

09:26:52.557 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is NAME_OF_MY_DOCKER_HOST
09:26:52.921 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 17.09.0-ce
  API Version: 1.32
  Operating System: CentOS Linux 7 (Core)
  Total Memory: 7823 MB
09:26:54.359 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at NAME_OF_MY_DOCKER_HOST:32774
java.net.NoRouteToHostException: No route to host (Host unreachable)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
    at java.lang.Thread.run(Thread.java:748)
lon-pilot-mpf picture lon-pilot-mpf on 23 May 2018

Hi, I have similar error on 1.7.3 inside Jenkins image.
In docker processes I see that Ruak image has been started, but it impossible to connect to it for some Reason...

10:00:00.109 [main] INFO  org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
10:00:00.129 [main] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

10:00:00.989 [main] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
10:00:01.134 [main] INFO  org.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 17.03.2-ce
  API Version: 1.27
  Operating System: Ubuntu 14.04.5 LTS
  Total Memory: 16047 MB
10:00:02.686 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:39972
java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
    at java.lang.Thread.run(Thread.java:748)
10:00:03.692 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:39972
java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
    at `java.lang.Thread.run(Thread.java:748)
deshchenko picture deshchenko on 24 May 2018

Actually, I resolved my problem. Root cause is that moby-ryuk image isn't accessible via External Docker IP (172.17.0.1 in my case) inside Jenkins docker container due to firewall rule. I open necessary ports and it works fine now.

deshchenko picture deshchenko on 29 May 2018

@eschenko could you please share why you have a firewall affecting Docker? Is it something custom on your instances or it is configured by default? :)

bsideup picture bsideup on 29 May 2018

That instance had been configured by other person before I started using it. So I suppose it had custom configuration.

Anyway that problem happened because of running inside Jenkins container. Testcontainers used External Docker API for connection to moby-ryuk (172.17.0.1 in my case). Ports were closed and I couldn't connect to it. But I was able to connect via internal container IP ( 172.22.0.6 in my case) and it worked fine (just via telnet command).

I couldn't find proper way to configure testcontainers to access moby-ryuk via internal IP. Is it possible? Or we can add some notes to documentation about access to moby-ryuk from other containers like Jenkins etc. It was not obvious that root cause were closed ports.

deshchenko picture deshchenko on 30 May 2018

I'm seeing exactly the same trying to run in Drone CI with the host Docker API exposed via IP instead of mounting /var/run/docker.sock, I suspect with the same cause as @eschenko. If I run it on a laptop using drone exec and mounting /var/run/docker.sock as a volume it works without issue. I've reverted back to 1.5 for now as ambassador seems to work OK in this scenario.

GJKrupa picture GJKrupa on 29 Jun 2018

We are seeing a similar issue on our jenkins agents running centos with selinux.

14:22:41.936 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at localhost:32797
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[na:1.8.0_131]
    at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_131]
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[na:1.8.0_131]
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[na:1.8.0_131]
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[na:1.8.0_131]
    at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.8.0_131]
    at java.io.BufferedReader.fill(BufferedReader.java:161) ~[na:1.8.0_131]
    at java.io.BufferedReader.readLine(BufferedReader.java:324) ~[na:1.8.0_131]
    at java.io.BufferedReader.readLine(BufferedReader.java:389) ~[na:1.8.0_131]
    at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:139) ~[testcontainers-1.6.0.jar:na]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]

When starting the ryuk container manually, we see a socket permission problem. 

$ docker run -p8080:8080 -v /var/run/docker.sock:/var/run/docker.sock --name ryuk -P -d quay.io/testcontainers/ryuk:0.2.2
$ docker logs ryuk
panic: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/_ping: dial unix /var/run/docker.sock: connect: permission denied

The access is blocked by selinux for security reasons, see https://danwalsh.livejournal.com/74095.html
We could allow this globally on the agents, but given the outlined security issues, this is probably not a good idea.

jmaicher picture jmaicher on 9 Aug 2018

I am having a similar issue. But I'm getting this locally on Fedora. When using 1.8.1, I get this issue with the Ryuk container on localhost:32788

My other Docker containers that I run locally are all on 172.17.0.2. I'm not sure if the Ryuk one was supposed to run on that one too? Also the first time I tried running with 1.8.1 a warning popped that I didn't get to click on from SELinux. I can't seem to get that message to pop up again however.

I tried using 1.5.1 as well, however I get another issue:
ERROR [馃惓 [mysql:5.7.15]] - Retry limit reached while trying to pull image: mysql:5.7.15. Please check output of 'docker pull mysql:5.7.15' [thread=main]

I have a local container that is running that exact image and when re-pulling I don't get any error messages.

My issues seems to be similar, though I'm not running remotely.

TheShed412 picture TheShed412 on 29 Aug 2018

Just tried starting a ryuk container manually as suggested my @jmaicher and I got the same error from the logs:
panic: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/_ping: dial unix /var/run/docker.sock: connect: permission denied goroutine 1 [running]: main.main() /go/src/github.com/bsideup/moby-ryuk/main.go:31 +0xe2e

TheShed412 picture TheShed412 on 29 Aug 2018

I also have a Ubuntu 18 VM with docker installed and starting the ryuk image manually actually works. Could this be a CentOS/Fedora issue?

TheShed412 picture TheShed412 on 29 Aug 2018

Is there any further update on this.

My scenario is Jenkins. We run our tests inside docker containers and then mount the docker socket in to talk with other containers.

The issue seems sporadic for us. Some builds fail, others pass.

Error Message

java.lang.IllegalStateException: Can not connect to Ryuk

Stacktrace

java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:166)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:119)
    at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:142)
    at com.bossanova.helper.ConsulContainerIntegrationTest.beforeClass(ConsulContainerIntegrationTest.java:19)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Edit:

We have downgraded from 1.8 to 1.7 and the issue remains in both.

bearrito picture bearrito on 7 Sep 2018

Is the expecation that https://github.com/testcontainers/testcontainers-java/pull/843 will also resolve this issue?

bearrito picture bearrito on 7 Sep 2018

@bearrito Would be great if you can test with 1.9.1.

kiview picture kiview on 1 Oct 2018
👍1

I have tested 1.9.1 in bitbucket pipelines, 119 tests (1 thread) and 1 fails (seemingly random, different test fail in both actual test and execution order).
We were running 1.6.0 and 1.7.0 up until last week with no issues until we were forced to upgrade due to another issue.

reidarbrenna picture reidarbrenna on 2 Oct 2018

@kiview I tested with a single job in a fairly up to date jenkins install.

We did not see any regressions when running a single build, I also ran multiple builds in parallel (not sure this matters) and didn't see any regressions in that case either.

We are tentatively upgrading from 1.7.1 to 1.9.1. If I see regressions so I continue to report them to this ticket?

bearrito picture bearrito on 9 Oct 2018
👍1

@bearrito Thanks, would be great!

kiview picture kiview on 9 Oct 2018

1.9.1 doesn't appear to have fixed my variant on the issue. I suspect it's due to routing/iptables configuration on the Drone CI instance we're using that prevents the build container from accessing the docker0 network and only allowing Docker port-forwarding.

GJKrupa picture GJKrupa on 23 Nov 2018

@GJKrupa I actually finally got around this yesterday with 1.10.1. We also had to disable using our remote docker host (not 100% sure that's a requirement but we had no reason for using it except 'that's how it was set up N years ago') and use the locally installed docker daemon. You might try with that version and see if it fixes your issue.

lon-pilot-mpf picture lon-pilot-mpf on 23 Nov 2018

I tried switching to 1.10.1 yesterday after ambassador stopped working on my 1.5.1-based tests. 1.10.1 didn't solve the issue in my case and I'm still seeing the same errors:

4:44:02.499 [testcontainers-ryuk] WARN  o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32774
java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:112)
    at java.lang.Thread.run(Thread.java:748)

Unfortunately, disabling the remote host isn't an option for me as our CI tool is a shared resource being managed by a central team and they only support $DOCKER_HOST.

(as an aside, the ambassador issue actually seems to be a networking issue in docker-compose 1.23.1 and was fixed when I downgraded back to 1.22.0).

GJKrupa picture GJKrupa on 23 Nov 2018

I faced with this issue at version 1.8.3 although it worked in the past and no firewall constrain.
But it work well when I change version up to 1.9.1.

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>testcontainers</artifactId>
    <version>1.9.1</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>mssqlserver</artifactId>
    <version>1.9.1</version>
    <scope>test</scope>
</dependency>

Hope it help.

3t-dev picture 3t-dev on 19 Dec 2018

we are facing the same issue on a Jenkins Docker Agent
Version 1.10.5

2019-01-17 16:02:03.115  INFO   --- [           main] o.t.d.DockerClientProviderStrategy       : Will use 'okhttp' transport
2019-01-17 16:02:03.684  INFO   --- [           main] tAndSystemPropertyClientProviderStrategy : Found docker client settings from environment
2019-01-17 16:02:03.698  INFO   --- [           main] o.t.d.DockerClientProviderStrategy       : Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='<user>'
    registryPassword='null'
    registryEmail='null'

Error log:

019-01-17 16:02:35.379  WARN   --- [containers-ryuk] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at 172.17.0.1:37385
java.net.NoRouteToHostException: No route to host (Host unreachable)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:211)
    at org.testcontainers.utility.ResourceReaper.lambda$start$1(ResourceReaper.java:112)
    at java.lang.Thread.run(Thread.java:748)
lwiedenhoeft picture lwiedenhoeft on 17 Jan 2019

Same here on teamcity:

[11:08:33]  [Step 4/12] 10:08:33.527 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at localhost:33256
[11:08:33]  [Step 4/12] java.net.SocketException: Broken pipe (Write failed)
[11:08:33]  [Step 4/12]     at java.net.SocketOutputStream.socketWrite0(Native Method)
[11:08:33]  [Step 4/12]     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
[11:08:33]  [Step 4/12]     at java.net.SocketOutputStream.write(SocketOutputStream.java:134)
[11:08:33]  [Step 4/12]     at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:144)
[11:08:33]  [Step 4/12]     at java.lang.Thread.run(Thread.java:748)

[11:08:33]  [Step 4/12] 10:08:33.528 [testcontainers-ryuk] WARN  org.testcontainers.utility.ResourceReaper - Can not connect to Ryuk at localhost:33256
[11:08:33]  [Step 4/12] java.net.SocketException: Connection reset
[11:08:33]  [Step 4/12]     at java.net.SocketInputStream.read(SocketInputStream.java:210)
[11:08:33]  [Step 4/12]     at java.net.SocketInputStream.read(SocketInputStream.java:141)
[11:08:33]  [Step 4/12]     at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
[11:08:33]  [Step 4/12]     at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
[11:08:33]  [Step 4/12]     at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
[11:08:33]  [Step 4/12]     at java.io.InputStreamReader.read(InputStreamReader.java:184)
[11:08:33]  [Step 4/12]     at java.io.BufferedReader.fill(BufferedReader.java:161)
[11:08:33]  [Step 4/12]     at java.io.BufferedReader.readLine(BufferedReader.java:324)
[11:08:33]  [Step 4/12]     at java.io.BufferedReader.readLine(BufferedReader.java:389)
[11:08:33]  [Step 4/12]     at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:147)
[11:08:33]  [Step 4/12]     at java.lang.Thread.run(Thread.java:748)
nWidart picture nWidart on 18 Jan 2019

We ran into the same problem as well

java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:148)
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:64)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:124)
    at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:142)
    at org.testcontainers.containers.JdbcDatabaseContainer.<init>(JdbcDatabaseContainer.java:45)
    at org.testcontainers.containers.MariaDBContainer.<init>(MariaDBContainer.java:26)

We're using Testcontainers inside a custom JUnit-Rule which

  1. creates a Network
  2. creates two containers using this network
  3. starts the containers inside the before-method within independent threads
  4. stops those containers inside the after-method

This might not be the polite way to use Testcontainers but this replaced some legacy code and works much better than before.
We've started the tests several times, this error seems to occur in a non-deterministic way.

Using Testcontainers version 1.10.5.

StefanHufschmidt picture StefanHufschmidt on 6 Feb 2019

have tried creating the docker group? it solves permission issues

# create new group
sudo groupadd docker
# add current user to the group
sudo usermod -aG docker $USER
# log in to the group
newgrp - docker
ciri-cuervo picture ciri-cuervo on 15 Apr 2019

Please see #1274 . It is very likely that the local UNIX socket is not opened on your docker host.
Adding -H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375 when running the docker server should solve your issue.

mariusneo picture mariusneo on 15 Apr 2019
👍3

Ryuk seems to be listening on random port(33088 and 33089 and so on). This is unacceptable when docker host is behind firewall. I can't open all the port only for Ryuk.

skyline75489 picture skyline75489 on 18 Jun 2019

@skyline75489
It is not just Ryuk, but every container we start will be listening on a random port.

bsideup picture bsideup on 18 Jun 2019
👍2

@bsideup
And is there any way to force Ryuk to bind on the specific port not the random one ? We are having the similar issues with docker-host behind the firewall.

Wosin picture Wosin on 12 Jul 2019

@Wosin no, sorry. As I said - it is not just Ryuk, but any container we start.

bsideup picture bsideup on 12 Jul 2019
👍1

Maybe I'm wrong but this could be a 'known docker bug'.
I experienced several issues with this before.

Long story short:
Everyone can access docker-proxied port, except for container in the same host.
Basically two containers with ip 172.17.0.2 and 172.17.0.3 with exposed ports can't connect to each other on these ports using docker host ip 172.17.0.1:(exposedport)
You get 'No route to host'.

It is a docker/firewall/iptables issue.

You have to permit it on iptables in filter/INPUT chain an exact ip or docker network.
FirewallD example

firewall-cmd --direct --add-rule ipv4 filter INPUT_direct 0 -s 172.17.0.0/24 -j ACCEPT firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT_direct 0 -s 172.17.0.0/24 -j ACCEPT

Or iptables example:
iptables -t filter -I INPUT 1 -s 172.17.0.0/24 -j ACCEPT

Where 172.17.0.0/24 is default docker network out of box.

This will INSERT rule in filter/INPUT chain as FIRST one.
To delete:
iptables t filter -D INPUT 1

To test it, just run two containers like centos (install telnet there) and nginx with exposed port.
And try to connect via telnet to your nginx server via docker-host ip and nginx exposed port.
Keep in mind, ping always works, don't rely on that.

Just FYI my docker-compose.yml, config.toml and gitlab-ci.yml in attached zip file.

1.zip

ghost picture ghost on 2 Aug 2019
👍4

I've run into a similar problem but on localhost. After starting a docker container, the program hanged for two minutes trying to connect to it (only info log is saved, but there were requests every second in debug log). After that, while obtaining connection, it raised com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure Caused by: java.net.ConnectException: Connection refused (Connection refused) (stacktrace).

Reboot solved the problem. I think that the reason for such behavior was a nearly full load of the RAM with enabled swap, that had an about quarter load.

errandir picture errandir on 6 Sep 2019

If someone comes here having broken bitbucket pipeline with error org.testcontainers.utility.ResourceReaper : Can not connect to Ryuk at localhost:32768 java.net.SocketException: Broken pipe (Write failed) then here is the solution:
https://www.testcontainers.org/supported_docker_environment/continuous_integration/bitbucket_pipelines/

vkopichenko picture vkopichenko on 9 Sep 2019

Closing as outdated, since the issue seemed to be caused by network misconfiguration (see https://github.com/testcontainers/testcontainers-java/issues/572#issuecomment-517831833 for an advice on how to fix it)

bsideup picture bsideup on 9 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

andredasilvapinto picture andredasilvapinto  路  3Comments
lovepoem picture lovepoem  路  3Comments
oneiros-de picture oneiros-de  路  3Comments
dabraham02124 picture dabraham02124  路  3Comments
naderghanbari picture naderghanbari  路  3Comments