Testcontainers-java: Docker for Mac - cannot connect to Ryuk

[rnorth]

I believe that this is most likely a problem with Docker for Mac's userland proxy but am creating a ticket here for initial notes and visibility.

In the last couple of days I've noticed the following test failure on a Mac:

    java.lang.IllegalStateException: Can not connect to Ryuk
        at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:149)
        at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:125)
        at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:138)

Various observations:

• These exceptions persist - rerunning the build results in the same failure.
• No test can pass, as Testcontainers can't start up.
• Once the system has entered this broken state, the only way to fix is to restart Docker for Mac. Deleting containers and running docker system prune makes no difference.
• Ryuk is definitely running, and eventually shuts itself down (after 1 minute, after the Java tests have failed)
• Netcat from the Mac host to Ryuk seems to yield a hung socket - no responses come back, and there's no evidence that Ryuk is receiving any packets. Furthermore, the netcat connection stays 'open' even after the Ryuk container has shut down and been deleted.

On this machine I have Docker Desktop for Mac, Version 2.0.0.3 (31259) - stable version.

This happens after I have run a sufficient number of testcontainers builds (not that many; maybe 5-20?). I can't narrow down a repro case further yet.

I tried to trigger this problem with the following bash snippet, but it did not reoccur:

while true; do 
  ID=$(docker run -p 8080:8080/tcp -v /var/run/docker.sock:/var/run/docker.sock -d quay.io/testcontainers/ryuk:0.2.3)
  echo "foo=bar" | nc localhost 8080
  docker kill $ID
done

[rnorth]

It looks like this might be a related problem: https://forums.docker.com/t/port-forward-not-working-on-a-macos-installation-but-works-on-others/77401

After reaching the broken state, I experience identical symptoms to the author of this forum post - exposed ports are broken for any container I launch.

[rnorth]

There are all sorts of similar-sounding issues, but this looks most similar: https://github.com/docker/for-mac/issues/3487

Maybe there's a fix due imminently? https://github.com/docker/for-mac/issues/3487#issuecomment-509176608

[rnorth]

It seems that using the Edge version of Docker (2.0.5.0 (35318)) resolves this issue. Hopefully this will be released as Stable soon.

I'll close this, as it is not a Testcontainers bug.

[AndreasEK]

I'm seeing this issue again on Docker Desktop for Mac v2.4.0.0 and Catalina 10.15.7 - anybody else or is it just me?

2020-10-02 14:44:56.722  WARN 7172 --- [containers-ryuk] o.testcontainers.utility.ResourceReaper  : Can not connect to Ryuk at localhost:32768

java.net.ConnectException: Connection refused (Connection refused)
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
        at java.base/java.net.Socket.connect(Socket.java:591)
        at java.base/java.net.Socket.connect(Socket.java:540)
        at java.base/java.net.Socket.<init>(Socket.java:436)
        at java.base/java.net.Socket.<init>(Socket.java:213)
        at org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:121)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:119)
        at java.base/java.lang.Thread.run(Thread.java:834)

[bsideup]

@AndreasEK see https://github.com/testcontainers/testcontainers-java/issues/3166

[AndreasEK]

ok, upgrading testcontainers to 1.15.0-rc2 solved the issue, 1.14.3 still showed it.