Testcontainers-java: `Can not connect to Ryuk` due to missing Docker iptables rule

[vmassol]

Context:

• TC is working fine for me locally and on our Jenkins CI when the Jenkins agent is itself not running inside Docker.
• We're moving our Jenkins agents to use Docker and thus we need Docker in Docker.

After setting up everything as indicated on https://www.testcontainers.org/supported_docker_environment/continuous_integration/dind_patterns/, I still got the following error for starting Ryuk:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.xwiki.menu.test.ui.MenuIT
17:44:20.106 [main] INFO  o.t.d.DockerClientProviderStrategy - Will use 'okhttp' transport
17:44:20.714 [main] INFO  o.t.d.EnvironmentAndSystemPropertyClientProviderStrategy - Found docker client settings from environment
17:44:20.740 [main] INFO  o.t.d.DockerClientProviderStrategy - Found Docker environment with Environment variables, system properties and defaults. Resolved: 
    dockerHost=unix:///var/run/docker.sock
    apiVersion='{UNKNOWN_VERSION}'
    registryUrl='https://index.docker.io/v1/'
    registryUsername='root'
    registryPassword='null'
    registryEmail='null'
    dockerConfig='DefaultDockerClientConfig[dockerHost=unix:///var/run/docker.sock,registryUsername=root,registryPassword=<null>,registryEmail=<null>,registryUrl=https://index.docker.io/v1/,dockerConfigPath=/root/.docker,sslConfig=<null>,apiVersion={UNKNOWN_VERSION},dockerConfig=<null>]'

17:44:20.941 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: alpine:3.5, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
17:44:22.398 [main] INFO  o.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1
17:44:22.512 [main] INFO  o.testcontainers.DockerClientFactory - Connected to docker: 
  Server Version: 18.06.1-ce
  API Version: 1.38
  Operating System: Debian GNU/Linux 8 (jessie)
  Total Memory: 32221 MB
17:44:22.543 [main] WARN  o.t.utility.RegistryAuthLocator - Failure when attempting to lookup auth config (dockerImageName: quay.io/testcontainers/ryuk:0.2.2, configFile: /root/.docker/config.json. Falling back to docker-java default behaviour. Exception message: /root/.docker/config.json (No such file or directory)
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 33.495 s <<< FAILURE! - in org.xwiki.menu.test.ui.MenuIT
[ERROR] org.xwiki.menu.test.ui.MenuIT  Time elapsed: 33.494 s  <<< ERROR!
java.lang.IllegalStateException: Can not connect to Ryuk
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:148)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:125)
    at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:142)
    at org.testcontainers.containers.PortForwardingContainer.createSSHSession(PortForwardingContainer.java:30)
    at org.testcontainers.containers.PortForwardingContainer.getSshConnection(PortForwardingContainer.java:24)
    at org.testcontainers.containers.PortForwardingContainer.exposeHostPort(PortForwardingContainer.java:60)
    at org.testcontainers.Testcontainers.exposeHostPorts(Testcontainers.java:11)
    at org.xwiki.test.docker.junit5.XWikiDockerExtension.beforeAll(XWikiDockerExtension.java:108)
...

After a lot of debugging I found that the issue (I think) is that the Ryuk container is started and listens to port 32768 but on the IPv6 address, while TC tries to connect to it on the same port but on the IPv4 address (i.e. on 172.17.0.1).

vmassol@ks4:~$ netstat -tuplen
...
tcp6       0      0 :::32768

Open questions:

• Why is Docker exposing the port only on the IPv6 address and not on the IPv4 one for Ryuk. For example for the Jenkins agent docker image we use it exposes port 22 and I can see with netstat that port 22 is bound to both IPv4 and IPv6 addresses.
• I still need to figure out to make this work. I'm going to try disabling IPv6 to see if it helps.

Issue reported at request of @bsideup

Discussion thread at https://testcontainers.slack.com/archives/C1SUBPZK6/p1549555885186300

[vmassol]

Actually this is not enough to prove the problem. I see for example that when it works fine (outside of docker), the ryuk or ssh container ports are bound only on the IPv6 interface and yet it works, so there must be something else.

EDIT: It works probably because "localhost" is used and thus binds to the IPv6 interface... while when executed inside Docker it's the host IP that is used and thus binds to IPv4.

[vmassol]

Actually it wasn't a problem of IPv4 vs IPv6 in the end...

For some reasons there was one iptables rule missing on this machine and I can't figure out why since that should have been added by Docker. But adding the following makes it work (I still have problems downstream but they're probably different, need to work on them now):

iptables -A INPUT -i docker0 -j ACCEPT

Thx

[rnorth]

I've just clarified the title of this issue following the comment Actually it wasn't a problem of IPv4 vs IPv6 in the end...