Elasticsearch: Centos 7 and - java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native

[lejeczek]

Elasticsearch version (bin/elasticsearch --version):

$ /usr/share/elasticsearch/bin/elasticsearch --version
Version: 7.7.0, Build: default/rpm/81a1e9eda8e6183f5237786246f6dced26a10eaf/2020-05-12T02:01:37.602180Z, JVM: 14

Plugins installed: []

JVM version (java -version):
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

OS version (uname -a if on a Unix-like system):

3.10.0-1127.8.2.el7.x86_64

Description of the problem including expected versus actual behavior:

$ sudo journalctl -lf -o cat -u elasticsearch.service
Starting Elasticsearch...
fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33)
at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:116)
at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:126)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1510)
at org.elasticsearch.node.Node.start(Node.java:810)
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:317)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:402)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Failed to start Elasticsearch.
Unit elasticsearch.service entered failed state.
elasticsearch.service failed.

Steps to reproduce:
A fresh installation and first systemd start.

Provide logs (if relevant):

[2020-05-20T22:25:41,410][INFO ][o.e.p.PluginsService     ] [whale.priv.xx.xx.priv.xx.xx.x] no plugins loaded
[2020-05-20T22:25:46,363][INFO ][o.e.x.s.a.s.FileRolesStore] [whale.priv.xx.xx.priv.xx.xx.x] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-05-20T22:25:47,054][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [whale.priv.xx.xx.priv.xx.xx.x] [controller/635112] [Main.cc@110] controller (64 bit): Version 7.7.0 (Build a8939d3da43f33) Copyright (c) 2020 Elasticsearch BV
[2020-05-20T22:25:47,892][INFO ][o.e.d.DiscoveryModule    ] [whale.priv.xx.xx.priv.xx.xx.x] using discovery type [zen] and seed hosts providers [settings]
[2020-05-20T22:25:48,913][INFO ][o.e.n.Node               ] [whale.priv.xx.xx.priv.xx.xx.x] initialized
[2020-05-20T22:25:48,914][INFO ][o.e.n.Node               ] [whale.priv.xx.xx.priv.xx.xx.x] starting ...
[2020-05-20T22:25:49,111][INFO ][o.e.t.TransportService   ] [whale.priv.xx.xx.priv.xx.xx.x] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-05-20T22:25:49,342][WARN ][o.e.b.BootstrapChecks    ] [whale.priv.xx.xx.priv.xx.xx.x] system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
[2020-05-20T22:25:49,343][WARN ][o.e.b.BootstrapChecks    ] [whale.priv.xx.xx.priv.xx.xx.x] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-05-20T22:25:49,346][INFO ][o.e.c.c.Coordinator      ] [whale.priv.xx.xx.priv.xx.xx.x] cluster UUID [Sf_kyq4ATb2Doh7uy0ZyUA]
[2020-05-20T22:25:49,367][INFO ][o.e.c.c.ClusterBootstrapService] [whale.priv.xx.xx.priv.xx.xx.x] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-05-20T22:25:49,559][INFO ][o.e.c.s.MasterService    ] [whale.priv.xx.xx.priv.xx.xx.x] elected-as-master ([1] nodes joined)[{whale.priv.xx.xx.priv.xx.xx.x}{QpMP0s3jStqxSElpJUtxPQ}{YNrWU8aoRcaK0gWlol4voA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134992162816, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 4, version: 7, delta: master node changed {previous [], current [{whale.priv.xx.xx.priv.xx.xx.x}{QpMP0s3jStqxSElpJUtxPQ}{YNrWU8aoRcaK0gWlol4voA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134992162816, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2020-05-20T22:25:49,671][INFO ][o.e.c.s.ClusterApplierService] [whale.priv.xx.xx.priv.xx.xx.x] master node changed {previous [], current [{whale.priv.xx.xx.priv.xx.xx.x}{QpMP0s3jStqxSElpJUtxPQ}{YNrWU8aoRcaK0gWlol4voA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134992162816, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 4, version: 7, reason: Publication{term=4, version=7}
[2020-05-20T22:25:49,773][INFO ][o.e.h.AbstractHttpServerTransport] [whale.priv.xx.xx.priv.xx.xx.x] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-05-20T22:25:49,776][INFO ][o.e.n.Node               ] [whale.priv.xx.xx.priv.xx.xx.x] started
[2020-05-20T22:25:49,779][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [whale.priv.xx.xx.priv.xx.xx.x] fatal error in thread [main], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.elasticsearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:34) ~[?:?]
        at java.security.xxcessController.doPrivileged(AccessController.java:312) ~[?:?]
        at org.elasticsearch.systemd.Libsystemd.<clinit>(Libsystemd.java:33) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:116) ~[?:?]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:126) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1510) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:810) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:317) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:402) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.7.0.jar:7.7.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.7.0.jar:7.7.0]

[nik9000]

It'd probably be useful to see your distro version like from /etc/issue.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[rjernst]

The log file provided appears to only be the tail. Can you please provide the rest of the file? JNA failed to load, and it should have an error above the log file snippet here.

Separately, we currently lack graceful degredation for the systemd notifier hook when jna is not available. Need to think about what to do in this case, since we can't dynamically change systemd's reliance on this configuration option we have within our service file.

[tuckson]

Check your tmp directory. Can be set in jvm.options file
-Djava.io.tmpdir=<path> needs to be set to a writable directory. In default ES 7 this is set to a variable. Then of course this variable should be filled.

[lejeczek]

This is a head (it does not seem fixing ES_TMPDIR helps)

[2020-06-09T17:18:05,435][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] unable to load JNA native support library, native methods will be disabled. java.lang.UnsatisfiedLinkError: /tmp/jna--1985354563/jna8471419410690735218.tmp: /tmp/jna--1985354563/jna8471419410690735218.tmp: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?] at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2452) ~[?:?] at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2508) ~[?:?] at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2704) ~[?:?] at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2637) ~[?:?] at java.lang.Runtime.load0(Runtime.java:745) ~[?:?] at java.lang.System.load(System.java:1871) ~[?:?] at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)] at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)] at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)] at java.lang.Class.forName0(Native Method) ~[?:?] at java.lang.Class.forName(Class.java:340) ~[?:?] at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:110) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.7.0.jar:7.7.0] at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.7.0.jar:7.7.0] [2020-06-09T17:18:05,451][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot check if running as root because JNA is not available [2020-06-09T17:18:05,452][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot install system call filter because JNA is not available [2020-06-09T17:18:05,453][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot register console handler because JNA is not available [2020-06-09T17:18:05,455][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot getrlimit RLIMIT_NPROC because JNA is not available [2020-06-09T17:18:05,455][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot getrlimit RLIMIT_AS because JNA is not available [2020-06-09T17:18:05,455][WARN ][o.e.b.Natives ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] cannot getrlimit RLIMIT_FSIZE because JNA is not available [2020-06-09T17:18:05,732][INFO ][o.e.e.NodeEnvironment ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [129.8gb], net total_space [249.9gb], types [rootfs] [2020-06-09T17:18:05,735][INFO ][o.e.e.NodeEnvironment ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] heap size [1gb], compressed ordinary object pointers [true] [2020-06-09T17:18:05,893][INFO ][o.e.n.Node ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] node name [whale.priv.aaaa.ccc.priv.ddd.ee.ff], node ID [QpMP0s3jStqxSElpJUtxPQ], cluster name [elasticsearch] [2020-06-09T17:18:05,898][INFO ][o.e.n.Node ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] version[7.7.0], pid[2051470], build[default/rpm/81a1e9eda8e6183f5237786246f6dced26a10eaf/2020-05-12T02:01:37.602180Z], OS[Linux/3.10.0-1127.8.2.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14/14+36] [2020-06-09T17:18:05,898][INFO ][o.e.n.Node ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] JVM home [/usr/share/elasticsearch/jdk] [2020-06-09T17:18:05,899][INFO ][o.e.n.Node ] [whale.priv.aaaa.ccc.priv.ddd.ee.ff] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rp ...

[rjernst]

We have not experienced this in our own testing. These are kinds of failures to initialize JNA are usually environmental issues, for example having /tmp mounted noexec. See the documentation for what seems like this exact issue: https://www.elastic.co/guide/en/elasticsearch/reference/master/executable-jna-tmpdir.html. As this is not a bug, I hope you don't mind I close this issue.

[sktrinh12]

Hi Sorry to bring this issue up again; I got the same error on a centos machine and then installed openJDK which then gave me the error below. Can anyone please give me some clues on how to resolve the error? I followed that link about noexec for tmpdir but it kinda is over my head. Please be gentle, I am a newbie.

[2020-07-30T13:10:36,809][WARN ][o.e.b.Natives            ]  unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-17213976625080293983/jna18232491376079922055.tmp: /tmp/elasticsearch-17213976625080293983/jna18232491376079922055.tmp: failed to map segment from shared object: Operation not permitted
        at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2452) ~[?:?]
        at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2508) ~[?:?]
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2704) ~[?:?]
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2637) ~[?:?]
        at java.lang.Runtime.load0(Runtime.java:745) ~[?:?]
        at java.lang.System.load(System.java:1871) ~[?:?]
        at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018) ~[jna-5.5.0.jar:5.5.0 (b0)]
        at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) ~[jna-5.5.0.jar:5.5.0 (b0)]
        at com.sun.jna.Native.<clinit>(Native.java:195) ~[jna-5.5.0.jar:5.5.0 (b0)]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:340) ~[?:?]
        at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:110) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.8.1.jar:7.8.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.8.1.jar:7.8.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.8.1.jar:7.8.1]

[vipin12117]

change in /etc/elasticsearch/jvm.options

-Djava.io.tmpdir=${ES_TMPDIR}

-Djava.io.tmpdir=/var/log/elasticsearch

issue will fix.

[elbembo]

change in /etc/elasticsearch/jvm.options

-Djava.io.tmpdir=${ES_TMPDIR}

-Djava.io.tmpdir=/var/log/elasticsearch

issue will fix.

after try every thing from every where you are right

[TheGinge01]

change in /etc/elasticsearch/jvm.options

-Djava.io.tmpdir=${ES_TMPDIR}

-Djava.io.tmpdir=/var/log/elasticsearch

issue will fix.

Yep, that seemed to fix our problem as well.
The only thing I did differently was update /etc/sysconfig/elasticsearch instead of /etc/elasticsearch/jvm.options and added the following:

# Elasticsearch temp directory
ES_TMPDIR=/var/log/elasticsearch

Thanks for the help!

[BBvT]

change in /etc/elasticsearch/jvm.options

-Djava.io.tmpdir=${ES_TMPDIR}

-Djava.io.tmpdir=/var/log/elasticsearch

issue will fix.

Fixed my issue too! Thanks.

[lukeshep]

Cpanel / CENTOS 7.9 kvm [ip-160-153-250-187] v90.0.16
elasticsearch --version
Version: 7.9.2, Build: default/rpm/d34da0ea4a966c4e49417f2da2f244e3e97b4e6e/2020-09-23T00:45:33.626720Z, JVM: 15

Worked for me too thank you so much!

[ddrager]

Ran into this issue, more hosts are providing servers with /tmp as noexec. Updating to exec or changing tmpdir fixed it.