Testcontainers-java: ConcurrentModificationException using new Startable.dependsOn(startable)

[aguibert]

I ran into an error in one of my TravisCI builds using the new dependsOn mechanism:

    13780 ERROR 🐳 [postgres:9.6.12]  - Could not start container
    java.util.ConcurrentModificationException
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1660)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
        at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
        at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
        at org.testcontainers.containers.GenericContainer.applyConfiguration(GenericContainer.java:510)
        at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:301)
        at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:285)
        at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
        at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:283)
        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:272)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
        at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)

My containers are defined like this:

    @Container
    public static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>()
                    .withNetworkAliases("testpostgres")
                    .withDatabaseName("testdb");

    @Container
    public static MicroProfileApplication<?> app = new MicroProfileApplication<>()
                    .withEnv("POSTGRES_HOSTNAME", "testpostgres")
                    .withEnv("POSTGRES_PORT", "5432")
                    .withAppContextRoot("/myservice")
                    .dependsOn(postgres);

Using the latest version of testcontainers (1.12.0), the exception lines up with this bit of code:

    private void applyConfiguration(CreateContainerCmd createCommand) {
        HostConfig hostConfig = buildHostConfig();
        createCommand.withHostConfig(hostConfig);

        // Set up exposed ports (where there are no host port bindings defined)
        ExposedPort[] portArray = exposedPorts.stream()
                .map(ExposedPort::new)
                .toArray(ExposedPort[]::new);  // << --- CONCURRENT MOD EX HERE

It should also be noted that I am starting my containers in parallel like this:

containersToStart.parallelStream().forEach(GenericContainer::start);

which should be OK, since I believe the main value of dependsOn is that it allows us to do parallel container start without needing to worry about dependencies.

[rnorth]

Thanks, this seems really odd. Without looking into it yet, it seems to me that the parallel start of all containersToStart plus the chained dependency start behaviour could be causing a double-start of the same container.

Quite why we're getting that exception at that line seems strange, but overall we should be preventing the same container from starting concurrently in two threads.

I'll have a look this weekend.

[bsideup]

Workaround: do not use parallel streams and use Startables.deepStart instead

[aguibert]

thanks for the quick replies @rnorth and @bsideup! I will look into using Startables.deepStart() as a workaround for now.

I reviewed my code again and the closest suspect I can find is that I'm calling addExposedPorts() in the MicroProfileApplication ctor, but this is not an uncommon thing to do AFAIK, and the port list is set up before we fork off into multiple threads and doesn't get mutated again after that.

FYI: You can find the full class in question here: https://github.com/dev-tools-for-enterprise-java/system-test/blob/master/testcontainers/src/main/java/org/testcontainers/containers/microprofile/MicroProfileApplication.java

[pivovarit]

Thanks, this seems really odd. Without looking into it yet, it seems to me that the parallel start of all containersToStart plus the chained dependency start behaviour could be causing a double-start of the same container.

I managed to reproduce the issue and can confirm that's the case indeed - if multiple threads start containers with some dependencies between them, internal Startables#deepStart call will try to start them at the same time which will lead to multiple undefined side-effects (with multiple started containers being only one of them).

Will give it a closer look and propose a solution soon along with some docs update (it's not really clear if the tool is thread-safe)