Testcontainers-java: Fails on m1 mac docker preview, java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native

[timja]

https://www.docker.com/blog/download-and-try-the-tech-preview-of-docker-desktop-for-m1/

(possibly wrong repo to report this to)

UnixSocketClientProviderStrategy: failed with exception RuntimeException (java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native). Root cause NoClassDefFoundError (Could not initialize class com.sun.jna.Native)

   1.490 [id=16]    WARNING o.t.u.TestcontainersConfiguration#readProperties: Attempted to read Testcontainers configuration file at file:/Users/timja/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /Users/timja/.testcontainers.properties (No such file or directory)
   1.496 [id=16]    INFO    o.t.d.DockerMachineClientProviderStrategy#isApplicable: docker-machine executable was not found on PATH ([/Library/Java/JavaVirtualMachines/zulu-11.jdk/Contents/Home/bin, /Users/timja/bin, /usr/local/bin, /usr/local/bin, /usr/bin, /bin, /usr/sbin, /sbin, /Users/timja/code/jenkins/backend-commit-history-parser/bin])
   1.675 [id=16]    SEVERE  o.t.d.DockerClientProviderStrategy#lambda$getFirstValidStrategy$7: Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
   1.675 [id=16]    SEVERE  o.t.d.DockerClientProviderStrategy#lambda$getFirstValidStrategy$7:     UnixSocketClientProviderStrategy: failed with exception RuntimeException (java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native). Root cause NoClassDefFoundError (Could not initialize class com.sun.jna.Native)
   1.675 [id=16]    SEVERE  o.t.d.DockerClientProviderStrategy#lambda$getFirstValidStrategy$7: As no valid configuration was found, execution cannot continue

[bsideup]

The root cause is JNA and its lacking M1 support - consider reporting to them. Also, I was able to run it with Rosetta (since my IDEA is running with Rosetta already)

FYI even tho the "core" functionality of Testcontainers works (Ryuk, alpine checks, connectivity), some modules (like Kafka) are failing. I hope future versions of Docker Desktop preview would fix it.

[Stexxen]

Probably related.
I am attempting to use as much native as possible so I am using the following :

• M1 Native intelliJ IDE (build 6494 aarch64)
• Azul M1 Native JDK 16-ea+28
• Docker Desktop for M1
• Docker image mysql/mysql-server:latest for aarch64, (mysql/mysql doesn't have a arm64 version)

This provides the following output quite similiar to @timja, but as I am using all M1 native systems, @bsideup is it still a problem with JNA?

T 15:31:56.345  INFO 16779 --- [           main] .t.d.DockerMachineClientProviderStrategy : docker-machine executable was not found on PATH ([/usr/local/bin, /usr/bin, /bin, /usr/sbin, /sbin])
T 15:31:56.600 ERROR 16779 --- [           main] o.t.d.DockerClientProviderStrategy       : Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
T 15:31:56.600 ERROR 16779 --- [           main] o.t.d.DockerClientProviderStrategy       :     UnixSocketClientProviderStrategy: failed with exception RuntimeException (java.lang.UnsatisfiedLinkError: /Users/stexxen/Library/Caches/JNA/temp/jna16779827993017480427.tmp: dlopen(/Users/stexxen/Library/Caches/JNA/temp/jna16779827993017480427.tmp, 1): no suitable image found.  Did find:
    /Users/stexxen/Library/Caches/JNA/temp/jna16779827993017480427.tmp: no matching architecture in universal wrapper
    /Users/stexxen/Library/Caches/JNA/temp/jna16779827993017480427.tmp: no matching architecture in universal wrapper). Root cause UnsatisfiedLinkError (/Users/stexxen/Library/Caches/JNA/temp/jna16779827993017480427.tmp: dlopen(/Users/stexxen/Library/Caches/JNA/temp/jna16779827993017480427.tmp, 1): no suitable image found.  Did find:
    /Users/stexxen/Library/Caches/JNA/temp/jna16779827993017480427.tmp: no matching architecture in universal wrapper
    /Users/stexxen/Library/Caches/JNA/temp/jna16779827993017480427.tmp: no matching architecture in universal wrapper)
T 15:31:56.600 ERROR 16779 --- [           main] o.t.d.DockerClientProviderStrategy       : As no valid configuration was found, execution cannot continue

[bsideup]

@Stexxen yes, definitely something with JNA, nothing we can do here

[Stexxen]

@bsideup Sorry to bother you again.
Currently there is a PR https://github.com/java-native-access/jna/pull/1238 that compiles and the M1 Native /build of IntelliJ also contains a jna.jar and jna-platform.jar for aarch64

Could you provide some advice on how I could try to get testcontainers to make use of these jars?
This is out of my area so not sure what steps take.

[bsideup]

@Stexxen industry wide M1 support is going to take time, I would suggest running Rosetta JVM for now, so that everything works just fine.

FTR I am running Testcontainers on M1 Mac with Rosetta, JNA works just fine

[Stexxen]

@bsideup I understand and I'm quite happy with running everything on intel as before, but I have acquired an M1 to try things out on and now I have everything else executing natively the performance difference is quite a amazing.
However I cannot run any tests that use testcontainers.

I know im pushing this, but in my view thats how we get things working. You seem ok with waiting until all the tools are production ready and available before looking at this, and I'm also happy to step away and let that happen.
However I'm also happy to help get it to work and report problems, but using JNA is my sticking block and thats why I'm asking for advice.

[bsideup]

@Stexxen you can always override the JNA dependency version and use a newer one.

Also, as I said, I am also on M1, and I have no issues with JNA + Rosetta. Once JNA with M1 support is released, we will update the version we depend on 👍

[Stexxen]

@Stexxen you can always override the JNA dependency version and use a newer one.

Yes thats all I'm trying to do, to take advantage of the native JNA jars out there, but I have very little JNA experience, do you have a starting point to me at?

[elimydlarz-sp]

Looks like this is fixed in the newest version JNA. Will you publish a new version of testcontainers for Java that includes the newest JNA version and fixes this problem?

Not sure why this issue is closed since the problem is real and fixable.

[bsideup]

@elimydlarz-sp yes, next version of Testcontainers will depend on the latest version of JNA that comes with M1 support, just to make it easier for our users.

The issue is closed because it wasn't actionable - there was nothing we could do to fix it, as it required a fix in a 3rd party library. BTW even now, the JNA dependency comes from docker-java where it should be updated, and then we will receive an automated update of docker-java once it is released.

[pie-ai]

I added: test dependencies to net.java.dev.jna:jna5.8.0 to get it work until update is ready.

Thanks @bsideup

[todor-kolev]

@bsideup when is the latest version expected to be released?

[todor-kolev]

@pie-ai how did you manage to override the jna library? The dependency hierarchy (in my case) is testcontainers.postgres -> testcontainers.jdbc -> testcontainers.database-commons -> testcontainers.testcontainers -> docker-java.docker-java-transport-zerodep -> jna.

[pie-ai]

@todor-kolev I added the dependency to my project.

[bsideup]

@todor-kolev soon, but you don't have to wait for it to be released, just add JNA 5.8.0 to your dependencies list (as @pie-ai did). Also, docker-java 3.2.8 is already released and depends on 5.8.0, you can update the docker-java dependency as an alternative

[davsclaus]

Thanks docker-java 3.2.8 worked for me on M1 to detect docker from test-containers (Docker Desktop RC 3).
Version 3.2.7 did not detect docker.

        <dependency>
            <groupId>com.github.docker-java</groupId>
            <artifactId>docker-java-api</artifactId>
            <version>3.2.8</version>
        </dependency>
        <dependency>
            <groupId>com.github.docker-java</groupId>
            <artifactId>docker-java-transport-zerodep</artifactId>
            <version>3.2.8</version>
        </dependency>

[bsideup]

3.2.7 depends on JNA 5.5.0 (not compatible with M1), 3.2.8 on 5.8.0 (compatible).

Also, we just released Testcontainers 1.15.3 and it depends on JNA 5.8.0 🎉

[todor-kolev]

Hmm, I am still getting NoClassDefFoundError even after updating to Testcontainers 1.15.3, any ideas?

{"instant":{"epochSecond":1618487919,"nanoOfSecond":469898000},"thread":"zio-default-async-3","level":"ERROR","loggerName":"org.testcontainers.dockerclient.DockerClientProviderStrategy","message":"Could not find a valid Docker environment. Please check configuration. Attempted configurations were:","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":16,"threadPriority":5}
{"instant":{"epochSecond":1618487919,"nanoOfSecond":469942000},"thread":"zio-default-async-3","level":"ERROR","loggerName":"org.testcontainers.dockerclient.DockerClientProviderStrategy","message":"    UnixSocketClientProviderStrategy: failed with exception RuntimeException (java.lang.UnsatisfiedLinkError: /Users/tkolev/Library/Caches/JNA/temp/jna5997803519083406963.tmp: dlopen(/Users/tkolev/Library/Caches/JNA/temp/jna5997803519083406963.tmp, 1): no suitable image found.  Did find:\n\t/Users/tkolev/Library/Caches/JNA/temp/jna5997803519083406963.tmp: no matching architecture in universal wrapper\n\t/Users/tkolev/Library/Caches/JNA/temp/jna5997803519083406963.tmp: no matching architecture in universal wrapper). Root cause UnsatisfiedLinkError (/Users/tkolev/Library/Caches/JNA/temp/jna5997803519083406963.tmp: dlopen(/Users/tkolev/Library/Caches/JNA/temp/jna5997803519083406963.tmp, 1): no suitable image found.  Did find:\n\t/Users/tkolev/Library/Caches/JNA/temp/jna5997803519083406963.tmp: no matching architecture in universal wrapper\n\t/Users/tkolev/Library/Caches/JNA/temp/jna5997803519083406963.tmp: no matching architecture in universal wrapper)","endOfBatch":true,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":16,"threadPriority":5}
{"instant":{"epochSecond":1618487919,"nanoOfSecond":469950000},"thread":"zio-default-async-3","level":"ERROR","loggerName":"org.testcontainers.dockerclient.DockerClientProviderStrategy","message":"    UnixSocketClientProviderStrategy: failed with exception RuntimeException (java.lang.NoClassDefFoundError: Could not initialize class com.github.dockerjava.transport.DomainSocket). Root cause NoClassDefFoundError (Could not initialize class com.github.dockerjava.transport.DomainSocket)","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":16,"threadPriority":5}
{"instant":{"epochSecond":1618487919,"nanoOfSecond":469956000},"thread":"zio-default-async-3","level":"ERROR","loggerName":"org.testcontainers.dockerclient.DockerClientProviderStrategy","message":"As no valid configuration was found, execution cannot continue","endOfBatch":true,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":16,"threadPriority":5}

- Unexpected failure 0ms
  IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration

  DockerClientProviderStrategy.java:215    org.testcontainers.dockerclient.DockerClientProviderStrategy#lambda$getFirstValidStrategy$7
  Optional.java:403                        java.util.Optional#orElseThrow
  DockerClientProviderStrategy.java:207    org.testcontainers.dockerclient.DockerClientProviderStrategy#getFirstValidStrategy
  DockerClientFactory.java:136             org.testcontainers.DockerClientFactory#getOrInitializeStrategy
  DockerClientFactory.java:178             org.testcontainers.DockerClientFactory#client
  LazyDockerClient.java:14                 org.testcontainers.LazyDockerClient#getDockerClient
  LazyDockerClient.java:12                 org.testcontainers.LazyDockerClient#authConfig
  GenericContainer.java:310                org.testcontainers.containers.GenericContainer#start

I did delete the jna cache before trying and did a clean in the build tool.