Testcontainers-java: Elasticsearch `availableProcessors is already set` error when using Testcontainers netty transport

[rnorth]

So far I've only seen this affect our internal tests. On one of my machines I consistently get this failure:

Gradle Test Executor 5 > org.testcontainers.elasticsearch.ElasticsearchContainerTest > transportClientClusterHealth FAILED
    java.lang.IllegalStateException: availableProcessors is already set to [4], rejecting [4]
        at io.netty.util.NettyRuntime$AvailableProcessorsHolder.setAvailableProcessors(NettyRuntime.java:51)
        at io.netty.util.NettyRuntime.setAvailableProcessors(NettyRuntime.java:87)
        at org.elasticsearch.transport.netty4.Netty4Utils.setAvailableProcessors(Netty4Utils.java:83)
        at org.elasticsearch.transport.netty4.Netty4Transport.<init>(Netty4Transport.java:112)
        at org.elasticsearch.transport.Netty4Plugin.lambda$getTransports$0(Netty4Plugin.java:86)
        at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:189)
        at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:283)
        at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:128)
        at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:114)
        at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:104)
        at org.testcontainers.elasticsearch.ElasticsearchContainerTest.transportClientClusterHealth(ElasticsearchContainerTest.java:103)

I believe this is an instance of a known Elasticsearch client issue (https://github.com/elastic/elasticsearch/issues/25741). This will be arising because both Elasticsearch and docker-java can use netty.

It is heavily mitigated by okhttp being our default transport now, but this bug does mean that we can't test the netty transport any more.

We should probably either:

• Wait for https://github.com/elastic/elasticsearch/issues/25741 to be resolved more elegantly (I agree with https://github.com/elastic/elasticsearch/issues/25741#issuecomment-430063966, but I'm not sure if it will be resolved that way)
• Modify our tests to set the workaround system property
• Modify the ElasticsearchContainer class to automatically set the workaround system property

@dadoonet you might have a better understanding of this - do you have any suggestions?

[dadoonet]

• Modify our tests to set the workaround system property

I believe that you need to set System.setProperty("es.set.netty.runtime.available.processors", "false"); indeed.

• Modify the ElasticsearchContainer class to automatically set the workaround system property

May be we should just document this workaround in case someone is getting this error as well?

[antonpious]

I looked at the following closed issue: [create-elasticsearch-client-throws-a-netty-illegalstateexception]
(https://github.com/elastic/elasticsearch/issues/25741)
and the
[Contradictory and, sometimes, poor, advice given for es.set.netty.runtime.available.processors]
(https://discuss.elastic.co/t/contradictory-and-sometimes-poor-advice-given-for-es-set-netty-runtime-available-processors/148014)

why wasn't a simpler solution implemented instead of ego clash between the developer and the user. Is there a review process for the developer comments on closing tickets?

The setting is required for elastic search client to connect to elastic search server. The default use case of any client is that it would run within another server or application that would initialize the cpu of Netty. So the client code should first check if the available processor setting is 0 before trying to set it similar to what Netty runtime is doing. The usage between elastic search server and elastic search client usage are different.

Please change the code to do a pre-check similar to what Netty runtime does.

Netty Runtime io.netty.util.NettyRuntime.setAvailableProcessors
synchronized void setAvailableProcessors(final int availableProcessors) { ObjectUtil.checkPositive(availableProcessors, "availableProcessors"); if (this.availableProcessors != 0) { final String message = String.format( Locale.ROOT, "availableProcessors is already set to [%d], rejecting [%d]", this.availableProcessors, availableProcessors); throw new IllegalStateException(message); } this.availableProcessors = availableProcessors; }

Code change required on org.elasticsearch.transport.netty4.Netty4Utils.setAvailableProcessors
Before this if statement put another if to check if available processor value is greater than 0

use method
public static int availableProcessors()

if (isAvailableProcessorsSet.compareAndSet(false, true)) { NettyRuntime.setAvailableProcessors(availableProcessors); } else if (availableProcessors != NettyRuntime.availableProcessors()) { /* * We have previously set the available processors yet either we are trying to set it to a different value now or there is a bug * in Netty and our previous value did not take, bail. */ final String message = String.format( Locale.ROOT, "available processors value [%d] did not match current value [%d]", availableProcessors, NettyRuntime.availableProcessors()); throw new IllegalStateException(message); }

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.

[bsideup]

Since 1.11.0, we no longer include Netty transport and fully migrated to OkHttp.
I'm closing this issue as not relevant anymore :)

[rfelgent]

You have my respect that you have taken the effort to remove completely the dependency of Netty transport!

Thank you people!