Elasticsearch: create-elasticsearch-client-throws-a-netty-illegalstateexception

[holygost]

detail: https://stackoverflow.com/questions/45116826/create-elasticsearch-client-throws-a-netty-illegalstateexception

The first time to new PreBuiltTransportClient() ,it throws the following exception, but the second time ,it creates PreBuiltTransportClient successfully, and this problem also happens in ElasticSearch-5.5.0:

Exception in thread "Full-Scan-Thread" java.lang.IllegalStateException: availableProcessors is already set to [8], rejecting [8] at io.netty.util.NettyRuntime$AvailableProcessorsHolder.setAvailableProcessors(NettyRuntime.java:51) at io.netty.util.NettyRuntime.setAvailableProcessors(NettyRuntime.java:87) at org.elasticsearch.transport.netty4.Netty4Utils.setAvailableProcessors(Netty4Utils.java:82) at org.elasticsearch.transport.netty4.Netty4Transport.<init>(Netty4Transport.java:138) at org.elasticsearch.transport.Netty4Plugin.lambda$getTransports$0(Netty4Plugin.java:93) at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:174) at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:265) at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:130) at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:116)

[jasontedor]

You need to set the system property es.set.netty.runtime.available.processors to false.

[jasontedor]

Taking a closer look at this, I don't see why you're running into this, we already guard against it being set twice. I think that there's something that you're not telling us about your application. Let's take this up on the forum post, we reserve GitHub for verified bug reports and feature requests.

[jasontedor]

Would you please open a topic on the forum and I will help you there? Also, I suspect that you are using Netty in another component of your application, would you please confirm (if you are, then my first answer is already going to be the correct guidance)?

[holygost]

I will analyze how this issue happens, and then i will give you the response. Now it works when using System.setProperty("es.set.netty.runtime.available.processors", "false");

I find that there is the same topic here : https://discuss.elastic.co/t/elasticsearch-5-4-1-availableprocessors-is-already-set/88036/2

[jasontedor]

Okay, what I think is happening is this: you are using Netty elsewhere in your application. That is leading the number of processors to be initialized before you've even instantiated a transport client. When a transport client is instantiated, we try to initialize the number of processors. Since you're using Netty elsewhere, this is already initialized and Netty guards against this so this first instantiated fails with the illegal state exception that you're seeing. Also, in the course of instantiating the transport client, we flip a boolean guard so that if you instantiate another one we do not try to set the number of processors again (this violating the aforementioned Netty constraint that it is only set once). The second time that you try to instantiate the transport client, this guard is active so we do not try to set the number of processors again and instantiation succeeds.

[ItamarBenjamin]

Hi, im facing the same issue. i think it is quite reasonable to use netty in another place in the application. elastic client should check if the availableProcessors were set already by any component, not just elastic.
what do you think?

[jasontedor]

@ItamarBenjamin The Netty API does not support this, and it would be subject to time of check to time of use race conditions (check that it is not set, then if not try to set it but another thread could have already set it meanwhile). Instead, this is on application developers to know their application and either:

• set the system property I indicated above
• allow Elasticsearch to configure Netty first

[fpompermaier]

Hi to all,
I have the same problem in an Apache Flink job where clients are created almost simultaneously on the same machine (if the number of slots per TaskManager is > 1).
Thus, that variable should not be static (IMHO)

[Rorschachdb]

Hi all,

Stumbled accross this too. It arrose using spring boot latest RC2, in this context it is impossible to enforce elastic client to be bootstraped first and we're left with the system property solution.

Though I understand the above reason for doing it this way, I find it weird to use a vm wide system property to avoid such multiple configuration failure, wouldn't it be possible to use the org.elasticsearch.common.settings.Settings class instead ?

See spring-projects/spring-boot#12242 for the problem I first exposed to spring-boot team and a small project to reproduce it.

Cheers,

R

[jasontedor]

Though I understand the above reason for doing it this way, I find it weird to use a vm wide system property to avoid such multiple configuration failure, wouldn't it be possible to use the org.elasticsearch.common.settings.Settings class instead ?

It's worth considering, but what's the problem with the system property?

[StuAtGit]

@jasontedor - ran into this myself. With respect to:

.It's worth considering, but what's the problem with the system property?

I can only respond with this:

https://discuss.elastic.co/t/need-info-for-es-set-netty-runtime-available-processors/111322

A thread where you explain that this setting is not documented because it shouldn't be used except for "some special cases".

If the options are basically:

• have a deep understanding of every dependency your application pulls in, and all the configuration values those dependencies have (including purposefully undocumented ones), how they might conflict, and in what order I should initialize every object in my application, in particular, with respect to the configuration values those objects load.

• or, have elastic search have some nice documentation that covers the "some special cases", and what to do about them.

I feel like option #2 would be a more sustainable solution ;)

[StuAtGit]

An re-iterate what's been mentioned elsewhere - we see Netty used all over the place in our dependencies, it's very common.

Yet, I've only run into this issue in elasticsearch. It seems like, if all the libs never had a problem with this setting, there might be a workable solution for elasticsearch. True, I don't have the solution myself, but it does seem like you folks are deflecting a bit here.

[StuAtGit]

Actually, here is a solution: if you basically are recommending everyone set some undocumented property to ignore a particular code path that trips an exception in a rather common situation (again, lots of libs use netty), why not just try to set the variable, and catch the exception that indicates it's already set, print a warning, and move on?

Why crash out the whole application, just because someone set a netty variable before elasticsearch did?

[StuAtGit]

Opened a topic covering this here:

https://discuss.elastic.co/t/contradictory-and-sometimes-poor-advice-given-for-es-set-netty-runtime-available-processors/148014

[ealexhaywood]

Setting a system property is a fine workaround, but it's not a solution. I'm running into this issue using Spring Boot Webflux, which runs on Netty by default.

[jasontedor]

Would you clarify what in your mind is the difference that makes this a workaround but not a solution?

[ealexhaywood]

As your question implies, whether it's good practice to use a system property here is debatable.

It's code smelly to me because it's not for the system -- it's for elasticsearch code. If I have to add a comment into my code with a link to an obscure github issue explaining why I'm doing it, it's probably not the best way to do it.

That said, it's really not a big deal. It's just another _thing_ we have to be aware of now, and it's a good idea to reduce the amount of _things_ you have to keep track of.

[jonfreedman]

It's a workaround because you have to google a stack trace and follow a chain of several github issues - if it was a solution then it would prevent the issue from occurring in the first place.

I encountered this when adding elasticsearch to a spring-boot project that already uses redis (as per https://github.com/spring-projects/spring-boot/issues/12242#issuecomment-394940171).

[jtxyz]

This caught me out when adding seemingly innocuous dependency to an application. The Azure Storage SDK also uses Netty under the hood and _safely_ calls NettyRuntime.availableProcessors() when initializing.

In some applications the order of initialisation of components is hard to control, and in some environments settings VM args can be difficult or undesirable. An alternative solution, such as the ability to configure using the Settings object instead of a system property, would be much appreciated.

[gjesse]

hard to believe that the system property solution was deemed to be acceptable here, rather than a simple code change in the initialization.