Testcontainers-java: ResourceReaper.stopContainer sometimes fails to kill what is dead

[KantarBenedictAdamson]

One of my testcontainers integration tests (ran by JUnit 5 in Maven) failed because ResourceReaper.stopContainer threw an exception, which I don't expect. If I interpret the failure right, the essence of the complaint is it "Cannot kill container" because it "is not running". But killing should be idempotent: killing something that is already dead (not running) should be considered a success, and not throw an exception. So I guess this is either a problem in the use of com.github.dockerjava.api by ResourceReaper.stopContainer, or a problem in com.github.dockerjava.api itself.

I should point out that this particular integration test is unusual in that I expect the container to exit quickly; it is a test of error handling and shutdown in the case of a fatal configuration error.

Somewhat related to allowing graceful stopping of containers (issue 1000), I think.

Here is the log and stacktrace:

[INFO] Running com.XXXX.WrongUsageIT
...
14:20:35.235 [main] INFO  🐳 [testcontainers/8y6zrds9fbg4sb4z] - Starting container with ID: c69c45b039ed0a4d13105834b6aaa6d264cf295589ea291f89a2d33b10efbf17
14:20:35.818 [main] INFO  🐳 [testcontainers/8y6zrds9fbg4sb4z] - Container testcontainers/8y6zrds9fbg4sb4z is starting: c69c45b039ed0a4d13105834b6aaa6d264cf295589ea291f89a2d33b10efbf17
14:20:35.824 [main] INFO  🐳 [testcontainers/8y6zrds9fbg4sb4z] - Container testcontainers/8y6zrds9fbg4sb4z started
Apr 10, 2019 2:20:36 PM org.junit.jupiter.engine.execution.JupiterEngineExecutionContext close
SEVERE: Caught exception while closing extension context: org.junit.jupiter.engine.descriptor.MethodExtensionContext@1bb1fde8
java.lang.reflect.UndeclaredThrowableException
    at com.sun.proxy.$Proxy26.exec(Unknown Source)
    at org.testcontainers.utility.ResourceReaper.stopContainer(ResourceReaper.java:232)
    at org.testcontainers.utility.ResourceReaper.stopAndRemoveContainer(ResourceReaper.java:211)
    at org.testcontainers.containers.GenericContainer.stop(GenericContainer.java:322)
    at org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.close(TestcontainersExtension.java:140)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.execution.ExtensionValuesStore.closeAllStoredCloseableValues(ExtensionValuesStore.java:61)
    at org.junit.jupiter.engine.descriptor.AbstractExtensionContext.close(AbstractExtensionContext.java:73)
    at org.junit.jupiter.engine.execution.JupiterEngineExecutionContext.close(JupiterEngineExecutionContext.java:53)
    at org.junit.jupiter.engine.descriptor.JupiterTestDescriptor.cleanUp(JupiterTestDescriptor.java:191)
    at org.junit.jupiter.engine.descriptor.JupiterTestDescriptor.cleanUp(JupiterTestDescriptor.java:54)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$cleanUp$9(NodeTestTask.java:151)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.cleanUp(NodeTestTask.java:151)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:83)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
    at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
    at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:124)
    at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.testcontainers.dockerclient.AuditLoggingDockerClient.lambda$wrappedCommand$14(AuditLoggingDockerClient.java:98)
    ... 49 more
Caused by: com.github.dockerjava.api.exception.ConflictException: {"message":"Cannot kill container: c69c45b039ed0a4d13105834b6aaa6d264cf295589ea291f89a2d33b10efbf17: Container c69c45b039ed0a4d13105834b6aaa6d264cf295589ea291f89a2d33b10efbf17 is not running"}

    at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:274)
    at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:254)
    at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.post(OkHttpInvocationBuilder.java:115)
    at com.github.dockerjava.core.exec.KillContainerCmdExec.execute(KillContainerCmdExec.java:30)
    at com.github.dockerjava.core.exec.KillContainerCmdExec.execute(KillContainerCmdExec.java:11)
    at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
    at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
    at com.github.dockerjava.core.command.KillContainerCmdImpl.exec(KillContainerCmdImpl.java:50)
    ... 54 more

[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.742 s <<< FAILURE! - in com.XXXX.WrongUsageIT
[ERROR] starts  Time elapsed: 1.4 s  <<< ERROR!
java.lang.reflect.UndeclaredThrowableException
Caused by: java.lang.reflect.InvocationTargetException
Caused by: com.github.dockerjava.api.exception.ConflictException: 
{"message":"Cannot kill container: c69c45b039ed0a4d13105834b6aaa6d264cf295589ea291f89a2d33b10efbf17: Container c69c45b039ed0a4d13105834b6aaa6d264cf295589ea291f89a2d33b10efbf17 is not running"}

[bsideup]

but killing should be idempotent

"What is dead may never die", huh :D
Yes, it seems we're missing a catch block for that error. The container will still be killed by Ryuk, but what you observe is for sure not a good behaviour.

Would you like to contribute a fix for it? ☺️

[KantarBenedictAdamson]

The container will still be killed by Ryuk

Unfortunately, because I want to run the test suite inside to Jenkins Docker agent, I've disabled Ryuk using the TESTCONTAINERS_RYUK_DISABLED environment variable.

[bsideup]

because I want to run the test suite inside to Jenkins Docker agent, I've disabled Ryuk

That does not sound right. You should not disable Ryuk if your build system does not destroy everything after a run (like CircleCI, TravisCI and others do)

[KantarBenedictAdamson]

(Best not to get bogged down with the Ryuk problem here)

I see that the ConflictException thrown by the Docker API corresponds to an HTTP status of 409 (Conflict):

The 409 (Conflict) status code indicates that the request could not
be completed due to a conflict with the current state of the target
resource. This code is used in situations where the user might be
able to resolve the conflict and resubmit the request. The server
SHOULD generate a payload that includes enough information for a user
to recognize the source of the conflict.

so you could pedantically argue that Docker itself is at fault for claiming that the "request could not be completed". ;-)

Simply catching a ConflictException (and NotFoundException too?) would fix the immediate problem. But there would be the lingering nagging doubt that Docker might report a Conflict for other kinds of problems. The reply does indeed seem to have "a payload that includes enough information for a user to recognize the source of the conflict" in this case: a message saying "Container ... is not running". So the catch block could check the exception message for those magic words. But the code would break if Docker changed the message text. That however might be the best that testcontainers could do.

[chungngoops]

@bsideup I would like to contribute a fix for this issue, and I agree with @KantarBenedictAdamson, we should catch the ConflictException with the error message Cannot kill container ... is not running to make sure we are catching the right one. Do you have any other ideas?

[pitschr]

Hey. Thanks for fixing that. I am facing this issue now as well. I'm just curious if a release date is already defined so that the fix is available on maven repo as well?! Luckily it happens rarely, so it is not urgent.

Thanks again for spotting and fixing that

[bsideup]

@pitschr expect a release very soon, most probably this weekend.

Shout out to @chungngoops for fixing it! 👏

[chungngoops]

@pitschr : yes, I'm happy to fix!

[rnorth]

Fix was released in 1.12.0. Thanks @chungngoops!