Testcontainers-java: Docker4mac compatibility

[jeantil]

Hello,

I open this issue in order to link and track the dependencies in other projects which are needed to make this happen.

I already requested such compatibility in docker-java (https://github.com/docker-java/docker-java/issues/537) and this seems dependant on a netty issue (https://github.com/netty/netty/issues/4231)

once both of these are resolved, it would be nice to have a new test containers release that works using docker4mac and doesn't require docker-machine anymore :)

Thanks !

[bsideup]

@jeantil it will be great if we will have docker4mac to work on it :D

My request wasn't accepted yet, unfortunately.

[jeantil]

I suggest trying to contact docker on twitter and/or leaving a word in the docker4mac forum @ https://forums.docker.com/c/docker-for-mac to get faster access. Saying you are a maintainer of a docker related tool should get you a ticket pretty fast.

In the meantime if there are any testcase you want me to run, paste them here or point me to a branch of the project, I'll run them asap. I also suggest looking at the docker-java issue where I have documented all my attempts.
There is a combo where i manage to get the docker-java client (and testcontainers) to connect to my docker4mac server. It fails to actually start a container with an error 500 though (I think it's a protocol incompatibility between 1.18 and 1.22 but I am completely out of my depth here)

[rnorth]

@jeantil - thank you for starting to look into this!

Actually as it happens I've already made it in to the beta and will be playing with testcontainers integration ASAP :)

I'd be keen to learn from what you've noticed already - will have a look at the referenced issues for starts.

Thanks
Rich

[rnorth]

@jeantil - thanks for being ahead of me on this. I've just started looking into this now, so I'll note my observations in case it helps others.

As the latter comments in docker-java/docker-java#537 suggest, native unix socket support is not working, so right now it's necessary to do:

export DOCKER_HOST=tcp://docker.local:2375
export DOCKER_TLS_VERIFY=0

I suspect it might be a good thing if we modify DockerClientFactory to explicitly try these settings during its discovery process, as we want to default to trying the right thing unless overridden.

[outofcoffee]

@rnorth do we have a branch where we're tracking this?

[outofcoffee]

Here's one: https://github.com/outofcoffee/testcontainers-java/tree/feature/docker-local-config

[outofcoffee]

OK, so the changes in the above branch seem to work well on the new Docker for Mac beta:

• Client: Docker version 1.11.0, build 4dc5990)
• Mac app: Version 1.11.0-beta8 (build: 6072) 3c1bfeb0e86a9403f82302edfea4c4987cc2cb32

Need to check using Docker for Windows beta as well.

[jeantil]

I just got an email from the docker teams aying docker.local is being
removed from beta 9 making this workaround unusable :(

would it make sense to investigate https://github.com/spotify/docker-client instead of docker-java ? docker-java is about to move to netty (if I understand correclty it's in order to get exec capabilites with input and output stream redirection which isn't so useful for testcontainers)
netty isn't poised to get unix socket support anytime soon so docker-java isn't either.

docker-client seems to support unix sockets through the use of https://github.com/jnr/jnr-unixsocket and it should work on mac since at least one of the developers is on mac (https://github.com/spotify/docker-client/issues/146#issuecomment-109656816)

I have no idea if it would be "easy" to swap out docker-java for docker-client ... maybe have both around and use whichever works on the platform.

On Thu, Apr 28, 2016 at 8:43 PM Pete Cornish [email protected]
wrote:

OK, so the changes in the above branch seem to work well on the new Docker
for Mac beta:

• _Client:_ Docker version 1.11.0, build 4dc5990)
• _Mac app:_ Version 1.11.0-beta8 (build: 6072)
  3c1bfeb0e86a9403f82302edfea4c4987cc2cb32

Need to check using Docker for Windows beta as well.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/testcontainers/testcontainers-java/issues/126#issuecomment-215524049

[rnorth]

Yikes, I'd not read that email yet. This is a bit disappointing.

Looks like we'll need changes in docker-java to enable this. I'll re-read docker-java/docker-java/issues/537...

[rnorth]

@jeantil yes, the spotify docker-client is one of the options I'm considering. Thanks for the pointer to jnr-unixsocket.

In its earliest versions testcontainers actually used the spotify client library so switching might not be too painful.

[rnorth]

Just some early observations from tinkering yesterday:

• the spotify docker client library does seem to be more promising in initial testing with DfM beta9. A few basic API operations seem to work with the unix socket connection.
• I've started on a (currently unfinished) refactoring to use the spotify client: https://github.com/testcontainers/testcontainers-java/tree/spotify-docker-client

One of the reasons I switched to docker-java before was the difficulty of shading the spotify libraries - although it turned out docker-java still needed quite a bit of work. I suspect now that I've worked out the shading approach to use, the spotify client may not be too painful to shade.

[bsideup]

My "-1" to spotify:

Docker-java is widely used everywhere, including different Jenkins plugins.
It has some potential, especially with upcoming refactoring (switch to
Netty from Jersey)

APIs are being integrated pretty soon after their release. Stuff like
docker4mac, with breaking changes each beta release, obviously, will not be
integrated that soon until it hits some RC stage, so it's expect (more or
less)

I'm not agains of doing implementation-agnostic test containers btw, but
just switching to Spotify sounds bad for me.

[rnorth]

@bsideup thanks - this kind of discussion is needed (I'd not want to just go all in on a switch without discussing). The branch is just there while I'm trying to understand the implications of the spotify-client approach.

Did you get around to trying the netty upgrade for docker-java with testcontainers? @outofcoffee also mentioned he was trying it out along with beta 9, but had some issues I think. If it works, or if there are some improvements I can help with on docker-java to make it work, I'd be more than happy - I just need to get my head around what needs doing!

[rnorth]

Just pushed another branch to try and understand the tasks associated with using docker-java netty support to solve this: docker-java-3.

This branch still doesn't actually work, but:

• Uses the latest docker-java v3 release candidate
• Incorporates most (all?) of the changes necessary to testcontainers to adapt to the docker-java API changes
• uses the netty CmdExecFactory instead of JAX-RS (in the main client creation for unix sockets mode at least)
• Excludes the unix-socket-factory library and replaces it with junixsocket
• Does some maven dependency management and exclusion of test dependencies which were needed to avoid borking the specific version of netty that docker-java 3 needs.

I think I understand the core problem a little better having been through this exercise so will share what I've learned (though I suspect many people involved in solving this have already figured this out):

• com.github.dockerjava.netty.DockerCmdExecFactoryImpl:221 deals with creation of an EventLoopGroup specialised towards Unix sockets. For this, it instantiates io.netty.channel.epoll.EpollEventLoopGroup
• unfortunately EpollEventLoopGroup is linux only, being part of a library which relies on JNI to use linux native epoll features
• so basically per netty/netty#4231 we're going to need a native library for netty kqueue support to serve as an OS X substitute for the netty-transport-native-epoll lib...

[rnorth]

I've prepared another experimental branch which is essentially similar to the socat workaround that's been floating around: a Java TCP->Unix domain socket proxy which allows netty to continue connecting to a TCP socket on OS X. This uses docker-java 3 as well for API compatibility. The socket proxy is a separate component I wrote, here.

This is an ugly approach, but potentially a usable, temporary, option to turn to if Docker for Mac is released before netty/netty#4231 can be solved.

The proxy itself uses an obviously suboptimal model of threads and blocking I/O. However, at the volume of connections we'll have, I'm not too worried about performance yet.

Thoughts?

[outofcoffee]

Looks good, @rnorth. As long as we document the constraints I'm comfortable with us including this; essentially treating it as the lesser of two evils.

One suggestion: we could consider implementing it as one of the pluggable config strategies, and isolating that code to a separate dependency so it can be killed easily in future. This way people can opt in if they want to use it, rather than making part of the default config strategy list.

I've added Windows support here: https://github.com/outofcoffee/testcontainers-java/tree/feature/docker-java-3-socket-proxy-windows-support so that we get Docker for Windows beta as well.

[rnorth]

Word on the Docker forums is that Windows will also lose the TCP socket in the near future, to be replaced with Windows named pipes.

I'm guessing that it's even less likely that netty and docker-java will support that 😬...

[gesellix]

In case you're interested in a proof of concept for adapting named pipes as sockets, you can check out this one: https://github.com/gesellix/docker-client/blob/master/src/main/groovy/de/gesellix/docker/client/protocolhandler/urlstreamhandler/HttpOverNamedPipeClient.groovy#L38

It's yet another hack, but kind of minimal hook into the standard Java connection workflow, which consequently doesn't arise any issues with other high level HTTP libraries like Apache HttpClient or Netty.

[jeantil]

looks like docker-java is not going to be fixed any time soon, any hope for switching to the spotify client ?

I woudl prefer to avoid socat like tricks as workarounds ...

the beta is now public this could start impacting a much larger user base