Testcontainers-java: Add support for running containers inside WSL 2.0

[jetersen]

Would be great to support running the containers inside wsl by running wsl docker instead of the regular docker

This currently requires windows insiders and docker edge channel to run.

I would be happy to contribute a PR as a draft and have tests disable until CI can support it.

[bsideup]

Isn't WSL 2.0 runs just a "normal" Docker? If so, it is already supported (as long as the env vars are set correctly)

[jetersen]

you would have to prefix the cli with wsl to avoid hitting the regular docker daemon that could be running in Windows mode.

[bsideup]

But isn't it what the users want, to use their main "Docker for Windows" installation?
Also, I guess soon Docker for Windows will use WSL 2.0 under the hood.

[jetersen]

You can actually run both Linux and Windows containers at the same time with WSL 2.0 :)
Just by prefixing the docker command with wsl

This way in the future you don't have to be concerned about the OSType.
This will still use their main docker for windows installation.

You actually don't have to install docker inside the Linux subsystem with WSL 2.0

[bsideup]

Ok, two things I see here:
1) in the strategy, we should check that the OS type is Linux, so that we avoid a positive result of the strategy if it cannot be used later - this sounds like a trivial change, also a good first issue btw ;)
2) somehow detect "WSL 2.0" docker installation - this one is tricky (and hard to develop since it requires a special insider access). Would love to see a draft of it tho 👍

[rnorth]

Would be great to support running the containers inside wsl by running wsl docker instead of the regular docker

you would have to prefix the cli with wsl

You're probably aware, but just to make sure it's clear for all readers: we don't invoke the docker CLI at all, and access Docker via its API.

I have zero knowledge about how wsl docker is implemented, but we'd need there to be feature parity for this in the API as a basic prerequisite.

[jetersen]

wsl is just a way to execute a command inside the WSL. Docker edge will detect the context and runs the container inside WSL.

[jetersen]

I wasn't aware that you access docker via API.

[jetersen]

Any help in how to explore the docker API? maybe there is a way to do the same from API?

[kiview]

How can you access network ports that are running inside WSL 2.0? Are they automatically/transparently mapped to the host? If yes, a similar workaround to WSL 1.0 would work, just the other way around. Expose the Docker daemon inside WSL 2.0 over TCP.

Else you would need to expose the Unix socket from inside WSL to the Windows host, I don't think this is currently possible.

[jetersen]

Here are some of the benefits when using WSL 2.0 context: https://github.com/docker/for-win/issues/4374#issuecomment-517338773

[jetersen]

I think this could be solved in upstream by adding support for named pipes on Windows: https://github.com/docker-java/docker-java/issues/765#issuecomment-523428713

[jetersen]

Or be added here: https://github.com/testcontainers/testcontainers-java/blob/52d7126c19cbde76937dcabe1b589662fd15f9a5/core/src/main/java/org/testcontainers/dockerclient/NpipeSocketClientProviderStrategy.java#L17

[bsideup]

@casz we could have two strategies, one for //./pipe/docker_engine, and one for //./pipe/docker_wsl

[jetersen]

I'll see what I can do 😅 Hopefully have some time in the weekend to focus on this 👍

[jetersen]

@bsideup I haven't looked too much into the code base but perhaps a strategy that would

• check if docker_wsl pipe is available
• if not try and use docker_engine while also checking if OSType is Linux.

[rnorth]

Two discrete strategies would do the trick, I think.

Working out which order to evaluate them in could be interesting - I'm not sure what the right answer would be, but we have to assume that one will be attempted first, and will normally win if available.

What's going to be the safest default?

FWIW, users can manually override the strategy via a setting in ~/.testcontainers.properties, but we can't end up in a situation where a majority of users need to do this. We should be aiming to make a sensible choice automatically.

[jetersen]

I think once Windows 10 1909 is out the default should be docker_wsl IMHO. As that will run the container outside the very limited HyperV VM that the old docker_engine uses for running containers.

[jetersen]

As you can see below:
the newer WSL 2 docker daemon has access to all 16 threads and 32 GB of memory. Whereas the old docker daemon only has 2 threads and 2 GB of memory.

So this would help improve the developer experience when running testcontainers 😆

[kiview]

I also don't see an disadvantage of using docker_wsl if available.

On the other hand, at the moment it is still labelled as tech preview:
https://docs.docker.com/docker-for-windows/wsl-tech-preview/

[jetersen]

Well it could stay in draft PR till ready :)

[jetersen]

Here is an announcement that says they will switch people over to WSL 2 when generally available: https://engineering.docker.com/2019/10/new-docker-desktop-wsl2-backend/

[jetersen]

Ended up being a won't do due to the blog post above from docker engineering.

As of Docker Desktop (Edge) version 2.1.5.0 (40323) and Windows 10 Insider 19013.1000\
The named pipe is back to being named \\.\pipe\docker_engine at the moment it just a tick box away.

See the blog post above how to enable WSL 2 engine while it is still in preview

The future of Docker Desktop
Once Microsoft makes WSL 2 generally available, we plan to enable the WSL 2 engine on all supported Windows versions by default. We will still support the Hyper-V backend until Microsoft stops supporting Windows versions without WSL 2 though, but only as a fallback mechanism.

By moving to this “WSL 2 first” approach, we also want to take advantage of its unique characteristics to unlock new features in the future. As an example, WSL 2 is supported on Windows 10 Home. We want to take advantage of that to reach new users in the future (nothing to announce yet, but it is definitely in our backlog).

This new backend paves the way for exciting new features to come, and we are eager to hear your feedback.

We will be releasing the new WSL 2 architecture as part of the next Docker Desktop Edge release. For Windows users already with WSL 2 Download Edge today to get access to the latest Docker architecture in the next couple of weeks.

> get-childitem \\.\pipe\

    Directory:  \\.\pipe

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
------       01-01-1601     01:00        2   docker_engine
------       01-01-1601     01:00       18   docker_engine_linux
------       01-01-1601     01:00        1   dockerBackend
------       01-01-1601     01:00        2   dockerBackendApiServer
------       01-01-1601     01:00        3   dockerBackendApiServerForGuest
------       01-01-1601     01:00        2   dockerDiagnosticd
------       01-01-1601     01:00        2   dockerExport
------       01-01-1601     01:00        2   dockerGuiToDriver
------       01-01-1601     01:00        4   dockerLifecycleServer
------       01-01-1601     01:00        1   dockerLogs
------       01-01-1601     01:00        3   dockerMemlogdq
------       01-01-1601     01:00        2   dockerVolume
------       01-01-1601     01:00        5   dockerVpnkit
------       01-01-1601     01:00        2   dockerVpnKitControl
------       01-01-1601     01:00        2   dockerVpnkitData
------       01-01-1601     01:00        4   dockerVpnKitDiagnostics
------       01-01-1601     01:00        6   dockerWebApiServer
------       01-01-1601     01:00        2   dockerWsl2BootstrapExposePorts