Podman: Support TestContainers

[gastaldi]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

This is a follow-up of https://github.com/testcontainers/testcontainers-java/issues/2088.
It would be nice if podman could be used with TestContainers.

[mheon]

Can you link any current issues that need to be resolved here?

[zhangguanzhang]

this issue https://github.com/containers/podman/issues/7923
I was unable to recreate on master, I believe it was taken care of in https://github.com/containers/podman/commit/cde367c1c0b348366fcaf4d8fac3b753de8456d5

[ruddy32]

"//var/run/docker.sock" is not setup dynamically when API start.
It seem to be a problem with TestContainer.

[rhatdan]

You need to install podman-docker, for this to happen.

[bsideup]

Hey @rhatdan,

does it mean that all known issues with Podman and Testcontainers are fixed now and that 100% of TC's test suite passes with Podman? 🎉

[rhatdan]

I have no idea. But we need specific issues reported to what is broken. No one on the Podman team is concentrating on testing TestContainers. Community members have been taking the lead on this, and reporting issues, that we will attempt to fix ASAP.

[psakar]

I was able to successfully run test involving postgresql db using podman built from 2.2.0.RC1 with fixes from https://github.com/containers/podman/pull/8440, https://github.com/containers/podman/pull/8429 and https://github.com/containers/podman/pull/8423

podman (rootless) started with

podman system service -t 0 tcp:localhost:8880 &

test containers started with following environment variables set

DOCKER_HOST=tcp://localhost:8880
TESTCONTAINERS_RYUK_DISABLED=true

Test containers version 1.14.3 or 1.15.0

[ruddy32]

Does it means podman API do not provide a docker compatible socket?

Applying changes to /usr/lib/systemd/user/podman.service with
ExecStart=/usr/bin/podman system service -t 0 tcp://localhost:8880
and remove socket Requires and After instruction

... and updating maven build configuration with

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <systemPropertyVariables>
            <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
        </systemPropertyVariables>
        <environmentVariables>
            <DOCKER_HOST>tcp:///localhost:8880</DOCKER_HOST
            <TESTCONTAINERS_RYUK_DISABLED>true</TESTCONTAINERS_RYUK_DISABLED>
            <api.version>v2</api.version>
            <KEYCLOAK_SERVICE>http://localhost:8180</KEYCLOAK_SERVICE>
        </environmentVariables>
    </configuration>
</plugin>

... and running podman api using command systemctl --user start podman.service

give the following build result in maven test phase

Caused by: java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$7(DockerClientProviderStrategy.java:214)

Is there something wrong in my configuration?

[psakar]

@ruddy32 your comment does not provide enough information so anybody can provide you some useful help.
Best would be either link to your project with description how to reproduce or in case it is private project please create public "minimal" project reproducing your problem.

General hints based on your comment

1. Have you build the podman containing the fixes mentioned above ? (ATM only two are merged to master, you have to cherry pick the third one or wait for PR to be reviewed / possibly further fixed and merged)
2. Make it work without systemd service first - that rules out problems cased by possibly wrong setup of service
3. The exception mentioned in the end of your comment has root cause (which you have not pasted here).I've seen for example case, when root cause was ClassNotFound exception because of fasterxml related classes required for test containers "clashed" with old fasterxml dependency used by the project under test. That can be but does not have to be your case - hard to say without knowing the root cause of "Could not find a valid Docker environment"

[ruddy32]

Is there something wrong in my configuration?

I was using podman 2.1.1, which does not include last changes listed by @psakar :-(
Sorry for disturb!

[zkirill]

I was able to successfully run my Spring Boot test involving PostgreSQL with podman 2.2.0-dev (4fe7c3f7bc8a1d20a23e87783b6a17b887c782e1) and testcontainers 1.15.0 after introducing a third environment variable [1] in addition to the ones @psakar specified:

TESTCONTAINERS_CHECKS_DISABLE=true

Also, my testcontainers setup completely ignored any normal location for testcontainers environment variables [1] and I had to set them in the Run/Debug Configurations of IntelliJ for the test. Otherwise, the environment variables would never be picked up by EnvironmentAndSystemPropertyClientProviderStrategy for whatever reason.

-- EDIT 2 --

The following approach also worked for me after I learned about podman-docker and referenced the Podman API documentation [2].

I could not use podman-docker by itself because I kept getting access denied exceptions on /var/run/docker.sock.

1. Install podman-docker (is this still necessary given step 2?).
2. Run # podman system service -t 0
3. Set environment variable DOCKER_HOST=unix:///run/user/1000/podman/podman.sock.
4. Make sure the environment variable to disable Ryuk is still set to true.

-- EDIT 3 --

Adding a reference to podman-system-service [3] explaining "rootful" service, "rootless" service and "$XDG_RUNTIME_DIR" to hopefully save someone else some time. It took me a while to piece this together because I did not understand what I was searching for.

[1] https://www.testcontainers.org/features/configuration/#configuration-locations
[2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/using-the-container-tools-api_using-the-container-tools-cli
[3] https://github.com/containers/podman/blob/master/docs/source/markdown/podman-system-service.1.md

[ruddy32]

Running podman 2.2.1 with

 systemctl --user start podman.service

Installing host with podman-docker 2.2.1.

Configuring maven 3.6.3 build with

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
                <configuration>
                    <environmentVariables>
                        <DOCKER_HOST>unix:///run/user/1000/podman/podman.sock</DOCKER_HOST>
                        <TESTCONTAINERS_RYUK_DISABLED>false</TESTCONTAINERS_RYUK_DISABLED>
                        <api.version>v2</api.version>
                        ...
                    </environmentVariables>
                </configuration>
            </plugin>

Running maven test produce following message

ERROR] fakeTest  Time elapsed: 0.009 s  <<< ERROR!
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: 
Status 500: {"cause":"incorrect volume format, should be [host-dir:]ctr-dir[:option]","message":"fill out specgen: unix:///run/user/1000/podman/podman.sock:/var/run/docker.sock:rw: incorrect volume format, should be [host-dir:]ctr-dir[:option]","response":500}

DockerJava concat both DOCKER_HOST definition and '/var/run/docker.sock' instead of using DOCKER_HOST definition only (unix:///run/user/1000/podman/podman.sock:rw).

It seems to be a problem with DockerJava or TestContainer host configuration. Could it be a problem with Podman host configuration?

[mheon]

Providing the JSON payload for the failing container create call would be greatly helpful here - seems likely that Testcontainers is sending valid JSON but we're not parsing the volumes bit correctly.

[ruddy32]

Providing the JSON payload for the failing container create call would be greatly helpful here.

The container used is KeyCloack 10 and 11. How to capture JSON payload?

[mheon]

You can use socat to capture the HTTP traffic to and from the Podman socket; @baude knows how to do so better than I.