Quarkus: Quarkus and Testcontainers

[gboro54]

Describe the bug
When upgrading to version 1.3.0.Final of Quarkus I started getting issues while trying to leverage TestContainers to spin-up a MySQL instance as part of unit testing. I was getting the following error:
`java.util.ServiceConfigurationError: org.testcontainers.dockerclient.DockerClientProviderStrategy: org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy not a subtype

at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:588)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1236)
at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1264)
at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1299)
at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1384)
at java.base/java.lang.Iterable.forEach(Iterable.java:74)
at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:111)
at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:134)
at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:62)
at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:25)
at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:20)
at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:27)
at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1263)
at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:600)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:311)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:302)
at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:173)
at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.access$100(TestcontainersExtension.java:160)
at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$null$1(TestcontainersExtension.java:41)
at org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$getOrComputeIfAbsent$0(ExtensionValuesStore.java:81)
at org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.get(ExtensionValuesStore.java:182)
at org.junit.jupiter.engine.execution.ExtensionValuesStore.getOrComputeIfAbsent(ExtensionValuesStore.java:84)
at org.junit.jupiter.engine.execution.NamespaceAwareStore.getOrComputeIfAbsent(NamespaceAwareStore.java:53)
at org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$beforeAll$2(TestcontainersExtension.java:41)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at org.testcontainers.junit.jupiter.TestcontainersExtension.beforeAll(TestcontainersExtension.java:41)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$7(ClassBasedTestDescriptor.java:359)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:359)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:189)
at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:78)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:132)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:248)
at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$5(DefaultLauncher.java:211)
at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:226)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:199)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:132)
at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)`

This would appear to be tied to a class loader issue that I am unable to figure out. This is happening as part of a normal JVM build process

Expected behavior
I would expect the test container to start as it did in 1.2.0.Final

Actual behavior

To Reproduce
Steps to reproduce the behavior:

1. 
   
   
   2.
   
3.

Configuration

# Add your application.properties here, if applicable.

Screenshots
(If applicable, add screenshots to help explain your problem.)

Environment (please complete the following information):

• Output of uname -a or ver: Darwin WeissMis-MBP.fios-router.home 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64
• Output of java -version:
  openjdk version "11.0.1" 2018-10-16
  OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
  OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)
• GraalVM version (if different from Java): graalvm-ce-java11-19.3.1
• Quarkus version or git rev: 1.3.0.final
• Build tool (ie. output of mvnw --version or gradlew --version): Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T14:41:47-04:00)

Additional context
TestContainers version 1.13.0.

Also see link in TestContainers IssueTracker: https://github.com/testcontainers/testcontainers-java/issues/2470

[gastaldi]

@stuartwdouglas this looks like another classloader bug, similar to https://github.com/quarkusio/quarkus/issues/7996#issuecomment-601390914

[stuartwdouglas]

The fix for #7996 will likely help here, but in general we need a better stragegy around TestContainers, as their default lifecycle does not work with QuarkusTest. The best approach at the moment is to use a QuarkusTestResource to manage the container lifecycle.

[geoand]

I am wondering if we should have something in the docs about testcontainers and Quarkus.
It's really easy to use both together, it's just that most people aren't aware of QuarkusTestResource.

Also https://github.com/quarkusio/quarkus/pull/7897 might help with this issue as well.

@gboro54 if you have a reproducer, I can take a look and make sure what we are proposing will actual work and confirm that there aren't any more underlying issues that we might need to fix.

[gboro54]

@geoand - I can create a test project later but here is the code I have for my test cases as of now:

`import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
import com.fasterxml.jackson.datatype.jsr310.deser.LocalDateDeserializer;
import com.fasterxml.jackson.datatype.jsr310.ser.LocalDateSerializer;
import io.quarkus.test.junit.QuarkusTest;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.MySQLContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;

import org.jboss.logging.Logger;
import com.fasterxml.jackson.databind.ObjectMapper;

import javax.ws.rs.core.MediaType;
import java.time.LocalDate;
import java.time.format.DateTimeFormatter;

import static io.restassured.RestAssured.given;
import static org.hamcrest.CoreMatchers.is;

@QuarkusTest
@Testcontainers
public class TestClass {

private static final Logger LOGGER = Logger.getLogger(TestClass.class);


@Container
protected static MySQLContainer DATABASE = new MySQLContainer<>("mysql/mysql-server:5.7")
        .withDatabaseName("dbname")
        .withUsername("user")
        .withPassword("test")
        .withInitScript("import.sql");

@BeforeAll
private static void configure(){
    System.setProperty("quarkus.datasource.url",DATABASE.getJdbcUrl());
    System.setProperty("quarkus.datasource.username",DATABASE.getUsername());
    System.setProperty("quarkus.datasource.password",DATABASE.getPassword());
    LOGGER.info("DB and props init complete");

}

@Test
public void test() { }

}`

[gboro54]

For the QuarkusTestResource I assume I just do the container startup in there and set the connection info rather then pulling it from the image. I will give that a try

[geoand]

@gboro54 The prefered way to use testcontainers is to use it with @QuarkusTestResource. See this for an example: https://github.com/quarkusio/quarkus-quickstarts/blob/master/kafka-quickstart/src/test/java/org/acme/kafka/KafkaResource.java

The reason this is needed is to ensure that testcontainers plays nicely with the Quarkus lifecycle.

[gboro54]

@geoand - Thanks Ill give it a look. I hope the above helps but perhaps doing it with test resources this is a none-issue anyway

[sheepy85]

@geoand Thank you, just what I need.

However with @QuarkusTestResource start new containers for each test.
It would be nice a singleton option

My workaround:

//@QuarkusTestResource(MariaDbTestResource.class)
public class TestClass {

    static MariaDBTestResource mariaDB = new MariaDBTestResource();

    @BeforeAll
    static void setup() {
        mariaDB.start();
    }

    @AfterAll
    static void cleanUp() {
        mariaDB.stop();
    }
...
}

public class MariaDBTestResource implements QuarkusTestResourceLifecycleManager {

    private MariaDBContainer DATABASE = new MariaDBContainer<>();

    @Override
    public Map<String, String> start() {
        DATABASE = new MariaDBContainer<>("mariadb:10.4.12-bionic")
                .withDatabaseName("myschema")
                .withInitScript("import.sql")
                .withStartupTimeoutSeconds(60);

        DATABASE.start();

        System.setProperty("quarkus.datasource.url", DATABASE.getJdbcUrl());
        System.setProperty("quarkus.datasource.username", DATABASE.getUsername());
        System.setProperty("quarkus.datasource.password", DATABASE.getPassword());

        return Collections.emptyMap();
    }

    @Override
    public void stop() {
        DATABASE.close();
    }
}

[geoand]

I am surprised it starts one container per test.
I need to confirm that because I am pretty sure it was just one

[geoand]

@sheepy85 I just tested it again, and no matter how many tests run, when I use testcontainers inside a QuarkusTestResourceLifecycleManager, the container only starts once.

So the combination of testcontainers and QuarkusTestResourceLifecycleManager works marvelously :).

[geoand]

In any case @gboro54 if you provide a small reproducer for us to check, that would be great.

[sheepy85]

@geoand Indeed the container only starts once.

Tested again in a new clean project with maven and Intellij Allow parallel run option too.

Sorry for the confusion.

[geoand]

No worries :)

[schulzp]

Hi, I ran into the same class loading issue with quarkus 1.3.1.Final.

I created a shared maven artifact com.exmaple:test-shared which provides classes to be used with @QuarkusTestResource. Those implement QuarkusTestResourceLifecycleManager and launch multiple, pre-defined and custom containers, e.g. the org.testcontainers.containers.MySQLContainer and a custom com.example.KafkaContainer.

In my main project depend on com.exmaple:test-shared in test scope and use the test resource via annotation. If I run the test, MySQLContainer starts fine, KafkaContainer runs into the aforementioned class-loading issue, however. The problem occurs once its method getImageName() gets called which at some point tries to resolve a service (see OP), which fails because of different class loaders.

[geoand]

@schulzp would you be able to provide a reproducer we can check?

[schulzp]

@geoand, I tried but it's quite flaky behavior. Here is what I found out: The problem comes from code running inside the common ForkJoinPool whose threads might have not set a QuarkusClassLoader set via Thread.setContextClassLoader but use the system default jdk.internal.loader.ClassLoaders.AppClassLoader.

In my case I start multiple testcontainers in parallel using Stream.of(container1, container2, ...).parallel().forEach(GenericContainer::start) and sometimes the thread they are executed in is already bootstrapped with the correct QuarkusClassLoader and sometimes not. Seems like a race condition.

The following snippet allows me to work around this issue:

    final ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();
    Stream.of(container1, container2)
        .parallel().forEach(container -> {
          if (Thread.currentThread().getContextClassLoader() != contextClassLoader) {
              Thread.currentThread().setContextClassLoader(contextClassLoader);
          }
          container.start();
    });

[mklueh]

Running into the same issue now with 1.4.1.Final :(

[geoand]

@mklueh do you have a reproducer?

[mklueh]

@geoand No I don´t have any reproducer, but I´ve used it the way it is supposed to be used according to the the testcontainers documentation https://www.testcontainers.org/modules/vault/

I´ve now wrapped everything into QuarkusTestResourceLifecycleManager and what I´ve tested so far is working. Will there be an attempt to make it work normally or is it not possible?

[cemnura]

I have had experience with testcontainers in the past with quarkus in a service I developed. Are you developing a test for quarkus? If it is your own project/demo is there a specific reason why you are using QuarkusTestResourceLifecycleManager?

I may be in the wrong by not using QuarkusTestResourceLifecycleManager in the project. I simply used the annotations documented at testcontainer documentation for junit5.

https://www.testcontainers.org/test_framework_integration/junit_5/

I am launching a elasticsearch docker for testing such as;

  @Container
  private static final GenericContainer container =
      new FixedHostPortGenericContainer<>("docker.elastic.co/elasticsearch/elasticsearch:7.5.0")
          .withEnv("discovery.type", "single-node")
          .withFixedExposedPort(9200, 9200)
          .withFixedExposedPort(9300, 9300)
          .waitingFor(Wait.forHttp("/")); // Wait until elastic start;

The real trick here is to annotate the @TestContainers before @QuarkusTest

@Testcontainers
@QuarkusTest
public class ResourceTest {
}

Otherwise the containers are not launched before quarkus.

As I said I might be in the wrong here.

If you are developing a integration test for quarkus I recently experienced some difficulties using testcontainers which @geoand would confirm 😄

[geoand]

Will there be an attempt to make it work normally or is it not possible?

We might if we have per test application launch

[mklueh]

I have had experience with testcontainers in the past with quarkus in a service I developed. Are you developing a test for quarkus? If it is your own project/demo is there a specific reason why you are using QuarkusTestResourceLifecycleManager?

I may be in the wrong by not using QuarkusTestResourceLifecycleManager in the project. I simply used the annotations documented at testcontainer documentation for junit5.

https://www.testcontainers.org/test_framework_integration/junit_5/

I am launching a elasticsearch docker for testing such as;

  @Container
  private static final GenericContainer container =
      new FixedHostPortGenericContainer<>("docker.elastic.co/elasticsearch/elasticsearch:7.5.0")
          .withEnv("discovery.type", "single-node")
          .withFixedExposedPort(9200, 9200)
          .withFixedExposedPort(9300, 9300)
          .waitingFor(Wait.forHttp("/")); // Wait until elastic start;

The real trick here is to annotate the @TestContainers before @QuarkusTest

@Testcontainers
@QuarkusTest
public class ResourceTest {
}

Otherwise the containers are not launched before quarkus.

As I said I might be in the wrong here.

If you are developing a integration test for quarkus I recently experienced some difficulties using testcontainers which @geoand would confirm

Wow, thanks for that hint. I did not know that the order plays a role with annotations. It helped me indeed to go beyond that exception, although I now run into another issue 😄

[cemnura]

I have had experience with testcontainers in the past with quarkus in a service I developed. Are you developing a test for quarkus? If it is your own project/demo is there a specific reason why you are using QuarkusTestResourceLifecycleManager?
I may be in the wrong by not using QuarkusTestResourceLifecycleManager in the project. I simply used the annotations documented at testcontainer documentation for junit5.
https://www.testcontainers.org/test_framework_integration/junit_5/
I am launching a elasticsearch docker for testing such as;

  @Container
  private static final GenericContainer container =
      new FixedHostPortGenericContainer<>("docker.elastic.co/elasticsearch/elasticsearch:7.5.0")
          .withEnv("discovery.type", "single-node")
          .withFixedExposedPort(9200, 9200)
          .withFixedExposedPort(9300, 9300)
          .waitingFor(Wait.forHttp("/")); // Wait until elastic start;

The real trick here is to annotate the @TestContainers before @QuarkusTest

@Testcontainers
@QuarkusTest
public class ResourceTest {
}

Otherwise the containers are not launched before quarkus.
As I said I might be in the wrong here.
If you are developing a integration test for quarkus I recently experienced some difficulties using testcontainers which @geoand would confirm

Wow, thanks for that hint. I did not know that the order plays a role with annotations. It helped me indeed to go beyond that exception, although I now run into another issue 😄

Glad I could help 🙂I was amazed as well when the order played a role 👍

[aesteve]

It's really easy to use both together, it's just that most people aren't aware of QuarkusTestResource.

So the combination of testcontainers and QuarkusTestResourceLifecycleManager works marvelously :).

I tried, and for me it's absolutely not "working marvelously"
Sure, it started my container, ...

... , for EVERY test class of my project. Not just the ones declaring my @QuarkusTestResource container.

Maybe because I'm using Kotlin, or Gradle, or both @geoand . But although I thought @QuarkusTestResource was my way to workaround a bug reported in 1.1.0.Final, 5 months ago, to use TestContainers in my test, unfortunately it's not.

Are some folks using testcontainers in @QuarkusTests, with Gradle and kotlin, past the very basic example of a single test per project?

[geoand]

@aesteve if you have reproducer project that you could share, we would be happy to take a look.

[aesteve]

https://github.com/aesteve/quarkus-testresources-reproducer

~I tried to, but faced NoSuchFileException while running the tests, instead of the failure I was trying to reproduce... So I guess something else is wrong, but can't find what. Gonna experiment with other Quarkus versions~
=> fixed in https://github.com/aesteve/quarkus-testresources-reproducer/commit/7f6c6c9439e03514eef5669669984f5fc9440ae1
(you have to add at least one class under src/main/kotlin, the FileNotFoundException is quite a weird bug in this case, but I can understand it's not happening in real life, careful with Gradle libraries that would have all their code in testFixtures though : like utility testing libraries, they would have the same kind of issue).

---

~I tried to reproduce, but faced:~

Caused by: java.lang.IllegalStateException: Unable to locate CDIProvider

~instead.~
Fixed by: https://github.com/aesteve/quarkus-testresources-reproducer/commit/fa196db2c04afbb59e3d38318d9d4d11bb5f1af7
(you have to add at least one quarkus dependency apart from quarkus-junit for it to work. Sounds fine, since in a real project, you'll obviously have one "real" dependency in the classpath)

---

So, the reproducer does show something interesting:
if you run gradle -i test you'll see that the test resource is indeed instanciated just once, for the right test. If you run TestWithoutResource from IntellijIDEA, though, and put a breakpoint in SomeResource.start() you'll see it gets invoked! (same goes with gradle -i test --tests "...." which is what IntelliJ is doing anyway)
=> https://github.com/aesteve/quarkus-testresources-reproducer/blob/master/README.md