Elasticsearch: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED

[panchen66]

i use
BulkResponse bulkResponse = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
Accidental occurrence
java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED at org.apache.http.util.Asserts.check(Asserts.java:46) at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123) at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:531) at org.elasticsearch.client.RestClient.performRequestAsyncNoCatch(RestClient.java:516) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:228) at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1593) at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1563) at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1525) at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:416)
And restart follow-up can not be restored

This is the version of the project I quoted
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-elasticsearch</artifactId> <exclusions> <exclusion> <artifactId>spring-data-commons</artifactId> <groupId>org.springframework.data</groupId> </exclusion> <exclusion> <artifactId>log4j-core</artifactId> <groupId>org.apache.logging.log4j</groupId> </exclusion> <exclusion> <groupId>org.elasticsearch.plugin</groupId> <artifactId>transport-netty4-client</artifactId> </exclusion> <exclusion> <groupId>org.elasticsearch.client</groupId> <artifactId>transport</artifactId> </exclusion> </exclusions> <version>2.1.1.RELEASE</version> </dependency> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>6.5.1</version> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpcore</artifactId> <version>4.4.5</version> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.2</version> </dependency>

[elasticmachine]

Pinging @elastic/es-distributed

[elasticmachine]

Pinging @elastic/es-core-features

[javanna]

hi @panchen66 is it possible that the underlying client got closed in the meantime? Do you see anything in the logs of your client application?

[panchen66]

My fault, I found that the service handle exhaustion caused this problem.

[fayaz-rvce]

@panchen66 , I am also facing same issue with ES RestHighLevelClient. Can you explain bit more how did you resolve this? what is service handle exhaustion?

[zaffargachal]

@panchen66 I am facing the same problem when we moved our ES cluster to AWS, Can you explain how did you resolved that issue?

[weizijun]

I also facing the same case，in #42061. Can RestClient add a method to check if client is running? or reset client internal?

[house0128]

@panchen66 @fayaz-rvce @zaffargachal I am facing the same problem when we moved our ES cluster to AWS, Can you explain how did you resolved that issue?

[renweizhukov]

+1

@panchen66 @fayaz-rvce @zaffargachal I am facing the same problem when we moved our ES cluster to AWS, Can you explain how did you resolved that issue?

[pavalanski]

@panchen66 @fayaz-rvce @zaffargachal @renweizhukov I have the same issue. Please help!

[renweizhukov]

After some debugging and configuration tuning, I found that the exception was caused by too many concurrent ElasticSearch clients. After I reduce the number of concurrent clients, the exception is gone.

[pavalanski]

@renweizhukov thanks for the hint! I'm using the client at scheduled method, so every invocation is started in new thread. The client has been injected to the service and never closed. Seems that is the reason of exception in my case. I changed my implementation so a new client is created at every thread when it is needed and closes in the end of method. That fixed the issue.

[renweizhukov]

@renweizhukov thanks for the hint! I'm using the client at scheduled method, so every invocation is started in new thread. The client has been injected to the service and never closed. Seems that is the reason of exception in my case. I changed my implementation so a new client is created at every thread when it is needed and closes in the end of method. That fixed the issue.

Great! Thank you for sharing this!

[Gouravagrawal]

Hey Guys, I followed this thread but not able to resolve this issue I have a rest high level client and Elasticsearch 7.2 I am trying to bulk insert and keep getting this error -"java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED". Could anyone please help in getting me out of this?

[jurimengs]

My fault, I found that the service handle exhaustion caused this problem.

HI , brother, may I ask, why service handle exhaustion caused it?
I saw CloseableHttpAsyncClientBase implements the interface "Closeable", the method "closed" invoked by who?
thanks very much

[jurimengs]

here is some error stack message:
java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46) ~[httpcore-4.4.1.jar:4.4.1]
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123) ~[httpasyncclient-4.1.2.jar:4.1.2]
at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:343) ~[rest-5.4.3.jar:5.4.3]

"Request cannot be executed; I/O reactor status: STOPPED" is from :
protected void ensureRunning() {
final Status currentStatus = this.status.get();
Asserts.check(currentStatus == Status.ACTIVE, "Request cannot be executed; " +
"I/O reactor status: %s", currentStatus);
}

You see, if "currentStatus " is not ACTIVE, it will show us this error..
I mean, it must be closed by someone, so, who did it ?

[jbaiera]

You see, if "currentStatus " is not ACTIVE, it will show us this error..
I mean, it must be closed by someone, so, who did it ?

The Client is built on top of Apache HTTP Components. Inside of the client is an IOReactor object that enables async processing of IO events. Unfortunately, if this reactor thread pool encounters any exceptions it shuts down without any way to reactivate it. Almost every exception that makes it to the IOReactor will shut it down. To make matters worse, these exceptions are logged to an internal audit log on the reactor instead of to any logging appender, and are thus pretty much obfuscated unless client code chooses to collect the errors explicitly.

We have an issue open at the moment (#49124, feel free to add your +1's there) that details how we'd like to instrument the client to detect these outages and report what went wrong. Hopefully we'll have a way forward in the near future.

I want to thank everyone for their patience while we look into these "IO reactor status stopped" exceptions.

[walkman-kuan]

Did you disable cluster shard allocation or disable shard allocation to a particular node accidentally? I'm facing the same issue, and solve it by enabling cluster shard allocation and index-level shard allocation.

[atrophes]

so, is there a way to detect what stopped the I/O reactor ?

[Jigsawk]

@renweizhukov thanks for the hint! I'm using the client at scheduled method, so every invocation is started in new thread. The client has been injected to the service and never closed. Seems that is the reason of exception in my case. I changed my implementation so a new client is created at every thread when it is needed and closes in the end of method. That fixed the issue.

sorry, it's a question. Do we need to close the injected client manually? In my opinion,the client is singleon, each thread invocate the unique client object.

[renweizhukov]

@renweizhukov thanks for the hint! I'm using the client at scheduled method, so every invocation is started in new thread. The client has been injected to the service and never closed. Seems that is the reason of exception in my case. I changed my implementation so a new client is created at every thread when it is needed and closes in the end of method. That fixed the issue.

sorry, it's a question. Do we need to close the injected client manually? In my opinion,the client is singleon, each thread invocate the unique client object.

I don't think the client is singleton. I think you need to close the REST high level client after you are done with using it.

[renweizhukov]

so, is there a way to detect what stopped the I/O reactor ?

What I have done is to catch the exceptions and check if the exception message contains I/O reactor status: STOPPED. If yes, it implies that the client is stuck in an invalid state and won't be able to make any request. I have found similar issues as below:

https://github.com/elastic/elasticsearch/issues/42133
https://github.com/elastic/elasticsearch/issues/45115

To work around this, I simply close the existing client and create a new one.

[TuJianwei]

hi, I meet same problem. My java progrom is Running in docker and the error stack message is

I/O reactor terminated abnormally
java.lang.ArithmeticException: / by zero
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.addChannel(AbstractMultiworkerIOReactor.java:473)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:178)
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:145)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
at java.lang.Thread.run(Thread.java:748)

java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46)
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
at org.elasticsearch.client.RestClient.performRequestAsync(RestClient.java:554)
at org.elasticsearch.client.RestClient.performRequestAsyncNoCatch(RestClient.java:537)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:249)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:568)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:538)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:500)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:427)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:416)

[kaplin]

I might have a fix for this — https://github.com/apache/httpcomponents-core/pull/203/

[xiongmaodeguxiang]

i am also facing same issue with ES RestHighLevenClient using BulkProcessor, but i found that my service is out of memory . it case the connetion closed。