Elasticsearch: Failed to submit a listener notification task. Event loop shut down?

[zzt93]

Elasticsearch version (bin/elasticsearch --version): 5.6.1

Plugins installed: [xpack, analysis-icu, analysis-smartcn, analysis-ik]

JVM version (java -version):1.8.0.45

OS version (uname -a if on a Unix-like system):
Linux ubuntu-100 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

Steps to reproduce:
I am not sure how to reproduce it, noly happen one time in my cluster. Maybe it is because too many shards in my cluster? (9000+ shards, two nodes, when this error happens, we are creating around 4000 shards)

Provide logs (if relevant):
log file

[colings86]

This is almost certainly because you are creating too many shards on your cluster. Why do you have so many shards and why are you trying to create 4000 at one time? The cause of this error is probably due to the thread pool and queue filling up and causing rejected executions.

Without any kinds of steps to reproduce there is not much we can do to diagnose this but regardless the solution here is to not overload your cluster with too many shards and to not try to create 1000's of shards in one go on such a small cluster. If you can reproduce this on a more reasonable cluster size with a more reasonable number of shards please post the steps for reproduction. For now I will close this issue

[jasontedor]

From your log messages:

[2017-11-02T10:35:44,801][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated

and 236ms later:

[2017-11-02T10:35:45,037][INFO ][o.e.n.Node               ] [ubuntu-200] stopped

Your node was in the middle of shutting down, this is expected behavior.

[cawoodm]

In case anyone (like me) lands here with the same error: check that you're not trying to run multiple instances of Logstash on the same port!

[guyfromtheke]

@cawoodm asking, how do you check for multiple running instances of logstash ?

[shihua-guo]

same error. I found the there are two logstash.conf files: I back old conf and rename to logstash.conf.bak ,and create the new logstash.conf file. So there two files in logstash/pipeline:

1. logstash.conf.bak
2. logstash.conf
   Finally, I delete the back up file (logstash.conf.bak) and start success!!This cost my whole day.