Elasticsearch: Failed to register an accepted channel

[DaveCTurner]

A user running 6.5.1 on Ubuntu 18.04 with OpenJDK 1.8.0_191 reports the following exception:

[2019-02-19T16:47:06,524][WARN ][i.n.b.ServerBootstrap    ] [prod3] Failed to register an accepted channel: [id: 0xc5da2913, L:0.0.0.0/0.0.0.0:9400]
java.lang.NullPointerException: null
    at sun.nio.ch.EPollArrayWrapper.isEventsHighKilled(EPollArrayWrapper.java:174) ~[?:?]
    at sun.nio.ch.EPollArrayWrapper.setUpdateEvents(EPollArrayWrapper.java:190) ~[?:?]
    at sun.nio.ch.EPollArrayWrapper.add(EPollArrayWrapper.java:239) ~[?:?]
    at sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:178) ~[?:?]
    at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:132) ~[?:?]
    at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212) ~[?:1.8.0_191]
    at io.netty.channel.nio.AbstractNioChannel.doRegister(AbstractNioChannel.java:386) ~[netty-transport-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:504) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-common-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]

They say:

I'm running into this error message when running extensive queries from a python script and when adding / removing nodes. It seems to occur only when putting elasticsearch "to work". There is no problem when running the normal workload.

[elasticmachine]

Pinging @elastic/es-distributed

[peterhassing]

Hi,
I'm the one who ran into this issue.
If you need any additional info please let me know.

[original-brownbear]

I think this is the likely the result of a known but now fixed JDK issue: https://bugs.openjdk.java.net/browse/JDK-8168500 (though it seems like it's only resolve in JDK 9).

@peterhassing is it possible that a change in FD limits occurred on your machine or that your fd limit is/was exactly 65536?

[ernestas-poskus]

Version: 6.2.3

Have the same issue on logs cluster, an increase of FD's to 131072 did not help

[2019-10-18T16:06:10,778][WARN ][i.n.b.ServerBootstrap    ] Failed to register an accepted channel: [id: 0xba2d8895, L:0.0.0.0/0.0.0.0:9301]
java.lang.NullPointerException: null
        at sun.nio.ch.EPollArrayWrapper.isEventsHighKilled(EPollArrayWrapper.java:174) ~[?:?]
        at sun.nio.ch.EPollArrayWrapper.setUpdateEvents(EPollArrayWrapper.java:190) ~[?:?]
        at sun.nio.ch.EPollArrayWrapper.add(EPollArrayWrapper.java:239) ~[?:?]
        at sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:178) ~[?:?]
        at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:132) ~[?:?]
        at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:212) ~[?:1.8.0_92]
        at io.netty.channel.nio.AbstractNioChannel.doRegister(AbstractNioChannel.java:387) ~[netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:504) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92]