Testcontainers-java: Thread "blocked" on ResultCallbackTemplate.close

[reda-alaoui]

Sometimes, I have builds hanging infinitly on my CI machine with the following stack trace:

"main" #1 prio=5 os_prio=0 cpu=4130622.05ms elapsed=4134.60s tid=0x00007f7484011800 nid=0x62db runnable  [0x00007f7489a8a000]
   java.lang.Thread.State: RUNNABLE
    at org.testcontainers.shaded.okio.Buffer.indexOf(Buffer.java:1475)
    at org.testcontainers.shaded.okio.RealBufferedSource.indexOf(RealBufferedSource.java:352)
    at org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230)
    at org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:224)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:489)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
    at org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
    at org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
    at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
    at org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
    at org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
    at org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
    at org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
    at org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
    at org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
    at org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
    at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$761/0x00000008006c9040.close(Unknown Source)
    at com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
    at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:202)
    at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
    - locked <0x000000012dac0658> (a [Ljava.lang.Object;)
    at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
    at org.testcontainers.LazyDockerClient.authConfig(LazyDockerClient.java:12)
    at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:310)
    at com.cos.framework.core.data_test.postgres.PostgresTestServer.<init>(PostgresTestServer.java:73)

Here is the full jstack: test-container-jstack.txt

[bsideup]

@reda-alaoui are you on the latest version of Testcontainers (1.15.1)?

And, if so, could you please try transport.type=httpclient5?

[reda-alaoui]

@bsideup , yes I am.
I will try, thanks.

[reda-alaoui]

After 5 days of httpclient5, not a single frozen thread so far.
When will this become the default? :)

[bsideup]

Thanks for reporting! Very soon, I guess :) Maybe even 1.16.0 :)

[QazerLab]

We had this issue too with 1.15.2 (Ryuk startup was hanging infinitely on ResultCallbackTemplate.close in 0.5-1% of test launches). I confirm that switching the transport to httpclient5 (TESTCONTAINERS_TRANSPORT_TYPE=httpclient5) fixes the issue (no hangings for roughly 3k test launches).

[jonathan-meier]

Looking at the thread dump, this is an exact duplicate of #3531 with threads #1 (main) and #130 (docker-java-stream--218727894) producing a race condition when reading the response buffer.