Testcontainers-java: Unable to run test containers with Apple Silicon Mac

[sdelamo]

% docker version                                                                                      
Client: Docker Engine - Community
 Cloud integration: 1.0.8
 Version:           20.10.3
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        48d30b5
 Built:             Fri Jan 29 14:28:27 2021
 OS/Arch:           darwin/amd64 (rosetta)
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.3
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       46229ca
  Built:            Fri Jan 29 14:32:07 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Running docker via Docker Desktop Preview Version 3.1.0(60984)
Disabled: Use gRPC FUSE for file sharing

Error:

Caused by: java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$7(DockerClientProviderStrategy.java:215)
    at java.base/java.util.Optional.orElseThrow(Optional.java:408)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:207)
    at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:136)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:178)
    at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
    at org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
    at example.mycompany.MySql.init(MySql.groovy:16)
    at example.mycompany.MySql.<clinit>(MySql.groovy:10)
    ... 5 more

Using test containers in a Gradle build

    testImplementation(platform("org.testcontainers:testcontainers-bom:1.15.2"))
    testImplementation("org.testcontainers:mysql")
    testImplementation "org.testcontainers:selenium"

[bsideup]

@sdelamo please share the full log

[ansotolab]

@bsideup I have the same issue and I wrote it here https://github.com/testcontainers/testcontainers-java/issues/3810

[bsideup]

@ansotolab I do not think you have the same issue. In your case, the environment detection works fine.

[ansotolab]

@bsideup sorry it could be my fault because I wrote two comments on the same issue, I'm talking about the first one https://github.com/testcontainers/testcontainers-java/issues/3810#issuecomment-783221149 you can see there is an error log dropdown. In it appears:

• 09:54:24.871 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
• java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration

The second comment https://github.com/testcontainers/testcontainers-java/issues/3810#issuecomment-783231622 is a temporary solution. If I run the testcontainers inside Docker I didn't receive any error but it's not the best option for development.

[sdelamo]

@bsideup Not sure how if this is the log you mentioned:

16:30:05.406 [Test worker] WARN  o.t.u.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/Users/sdelamo/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /Users/sdelamo/.testcontainers.properties (No such file or directory)
16:30:05.413 [Test worker] INFO  o.t.d.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/usr/bin, /bin, /usr/sbin, /sbin])
16:30:05.630 [Test worker] ERROR o.t.d.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
16:30:05.631 [Test worker] ERROR o.t.d.DockerClientProviderStrategy -     UnixSocketClientProviderStrategy: failed with exception RuntimeException (java.lang.UnsatisfiedLinkError: /Users/sdelamo/Library/Caches/JNA/temp/jna14927840726792078170.tmp: dlopen(/Users/sdelamo/Library/Caches/JNA/temp/jna14927840726792078170.tmp, 1): no suitable image found.  Did find:
    /Users/sdelamo/Library/Caches/JNA/temp/jna14927840726792078170.tmp: no matching architecture in universal wrapper
    /Users/sdelamo/Library/Caches/JNA/temp/jna14927840726792078170.tmp: no matching architecture in universal wrapper). Root cause UnsatisfiedLinkError (/Users/sdelamo/Library/Caches/JNA/temp/jna14927840726792078170.tmp: dlopen(/Users/sdelamo/Library/Caches/JNA/temp/jna14927840726792078170.tmp, 1): no suitable image found.  Did find:
    /Users/sdelamo/Library/Caches/JNA/temp/jna14927840726792078170.tmp: no matching architecture in universal wrapper
    /Users/sdelamo/Library/Caches/JNA/temp/jna14927840726792078170.tmp: no matching architecture in universal wrapper)
16:30:05.631 [Test worker] ERROR o.t.d.DockerClientProviderStrategy - As no valid configuration was found, execution cannot continue

java.lang.ExceptionInInitializerError
    at java.base/java.lang.Class.forName(Class.java:398)
    at uk.co.metadataworks.mdximport.MySqlTestPropertyProvider$Trait$Helper.url(MySqlTestPropertyProvider.groovy:8)
    at uk.co.metadataworks.mdximport.DatabaseTestPropertyProvider$Trait$Helper.getProperties(DatabaseTestPropertyProvider.groovy:12)
    at uk.co.metadataworks.mdximport.repositories.DataModelRepositorySpec.getConfiguration(DataModelRepositorySpec.groovy:17)
    at uk.co.metadataworks.mdximport.ApplicationContextSpecification.$spock_initializeSharedFields(ApplicationContextSpecification.groovy:12)
Caused by: java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$7(DockerClientProviderStrategy.java:215)
    at java.base/java.util.Optional.orElseThrow(Optional.java:408)
    at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:207)
    at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:136)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:178)
    at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
    at org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
    at uk.co.metadataworks.mdximport.MySql.init(MySql.groovy:16)
    at uk.co.metadataworks.mdximport.MySql.<clinit>(MySql.groovy:10)
    ... 5 more


DataModelRepositorySpec > initializationError FAILED
    java.lang.ExceptionInInitializerError at DataModelRepositorySpec.groovy:17
        Caused by: java.lang.IllegalStateException at DataModelRepositorySpec.groovy:17
1 test completed, 1 failed

[bsideup]

@sdelamo it is, thanks :)

Your problem is related to JNA:

/Users/sdelamo/Library/Caches/JNA/temp/jna14927840726792078170.tmp: no matching architecture in universal wrapper

Consider either using JNA 5.7.0 that is compatible with M1 or use x86 build of JDK.

[ikattey]

I can confirm that adding the following Maven dependency fixed this issue for me:

 <dependency>
               <groupId>net.java.dev.jna</groupId>
                <artifactId>jna</artifactId>
               <version>5.7.0</version>
                <scope>test</scope>
</dependency>

[sdelamo]

Thanks @bsideup and @ikattey

Now I get a different error:

java.lang.ExceptionInInitializerError
    at java.base/java.lang.Class.forName(Class.java:398)
    at uk.co.metadataworks.mdximport.MySqlTestPropertyProvider$Trait$Helper.url(MySqlTestPropertyProvider.groovy:8)
    at uk.co.metadataworks.mdximport.DatabaseTestPropertyProvider$Trait$Helper.getProperties(DatabaseTestPropertyProvider.groovy:12)
    at uk.co.metadataworks.mdximport.repositories.DataModelRepositorySpec.getConfiguration(DataModelRepositorySpec.groovy:17)
    at uk.co.metadataworks.mdximport.ApplicationContextSpecification.$spock_initializeSharedFields(ApplicationContextSpecification.groovy:12)
Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:330)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
    at uk.co.metadataworks.mdximport.MySql.init(MySql.groovy:16)
    at uk.co.metadataworks.mdximport.MySql.<clinit>(MySql.groovy:10)
    ... 5 more
Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=mysql:5.7.28, imagePullPolicy=DefaultPullPolicy())
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1286)
    at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:615)
    at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:320)
    ... 8 more
Caused by: org.testcontainers.containers.ContainerFetchException: Failed to get Docker client for mysql:5.7.28
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:99)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:27)
    at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
    at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1284)
    ... 10 more
Caused by: com.github.dockerjava.api.exception.DockerClientException: Could not pull image: no matching manifest for linux/arm64/v8 in the manifest list entries
    at com.github.dockerjava.api.command.PullImageResultCallback.checkDockerClientPullSuccessful(PullImageResultCallback.java:97)
    at com.github.dockerjava.api.command.PullImageResultCallback.throwFirstError(PullImageResultCallback.java:112)
    at com.github.dockerjava.api.async.ResultCallbackTemplate.awaitCompletion(ResultCallbackTemplate.java:93)
    at org.testcontainers.images.TimeLimitedLoggedPullImageResultCallback.awaitCompletion(TimeLimitedLoggedPullImageResultCallback.java:52)
    at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:82)
    ... 14 more

I am able to download this docker image in an Intel Mac.

[dizney]

@sdelamo mysql does not provide arm images. For example https://hub.docker.com/_/mariadb does provide them, so you might try that.

[rdehuyss]

For those working with Gradle, adding the following solved the issue for me:

testImplementation 'net.java.dev.jna:jna-platform:5.8.0'

[bsideup]

FYI Testcontainers 1.15.3 is released now and depends on 5.8.0 by default.

[rdehuyss]

@bsideup indeed, I see it - however, for me it only works if I add the testImplementation 'net.java.dev.jna:jna-platform:5.8.0'. Can't explain why yet - maybe because I'm using gradle platform to manage my dependencies?

To reproduce, see:
https://github.com/jobrunr/jobrunr/blob/silicon-mac/core/build.gradle#L52-L53

Any test that uses TestContainers will fail like LettuceRedisStorageProvider

[bsideup]

@rdehuyss it is really a question to Gradle and not Testcontainers :)

Someone pointed out that visible-assertions also depends on JNA (5.2.0) and that some build systems for some reason prioritize it over docker-java-transport:
https://github.com/testcontainers/testcontainers-java/issues/4008#issuecomment-824617867

To help with that, since visible-assertions isn't used by our main classpath anymore, we plan to remove it as a dependency in the next release. You can also try excluding org.rnorth.visible-assertions:visible-assertions already today, just to check whether that's the case