Elasticsearch: Error when start in OpenJDK 9, Ubuntu 16

[steelywing]

Elasticsearch version (bin/elasticsearch --version):
Version: 6.2.2, Build: 10b1edd/2018-02-16T19:01:30.685723Z

Plugins installed: []

JVM version (java -version):
openjdk version "9-internal"
OpenJDK Runtime Environment (build 9-internal+0-2016-04-14-195246.buildd.src)
OpenJDK 64-Bit Server VM (build 9-internal+0-2016-04-14-195246.buildd.src, mixed mode)

OS version (uname -a if on a Unix-like system):
Ubuntu 16.04.3
Linux log 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Error when start

[0.000s][error][logging] Invalid decorator 'utctime'.
Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

[jasontedor]

Please specify the actual output of the commands requested (e.g., java -version), especially as it’s of the utmost relevance here. In this case, I suspect you have a pre-release version of JDK 9 instead of a GA version that supports these features. Please confirm by providing the output of the above command.

[DaveCTurner]

/cc @elastic/es-core-infra

[steelywing]

Thanks, I have updated the Java version.

[rjernst]

As suspected, your java 9 version is a beta, and almost 2 years old. If you want to use java 9, you need to either update to a newer distro release, or install java 9 binaries manually.

[jasontedor]

I understand this to mean that your issue is resolved by upgrading to a GA version and I will close this issue. Please clarify if I’m mistaken.

[steelywing]

I am using Ubuntu repo to install OpenJDK 9, it is the latest LTS version.

If I change to use OpenJDK 9 GA version, it works.

It will be nice if the doc says Ubuntu OpenJDK 9 is not compatible. 😃

[jasontedor]

We do not support pre-release/early access versions of the JDK, ever. The version that the Ubuntu package managers for 16.04 supply is an early access version only, it is not a GA version.

[minfrin]

If Ubuntu is doing something wrong, raise a bug with them and get them to fix it.

I've done it for you: https://bugs.launchpad.net/ubuntu/+source/openjdk-9/+bug/1752300

[southwolf]

Same error here, macOS 10.13.5, Java 10.0.1 and Elastic 6.2.4, both installed with Homebrew.

java --version
java 10.0.1 2018-04-17
Java(TM) SE Runtime Environment 18.3 (build 10.0.1+10)
Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)

/usr/local/Cellar/elasticsearch/6.2.4/libexec/bin

./elasticsearch --version
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[0.002s][error][logging] Error opening log file 'logs/gc.log': No such file or directory
Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.
Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

[anjia0532]

hi, @jasontedor when i run with docker , it throw a warn. OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

how to fix this?

# docker run docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.2 /usr/share/elasticsearch/bin/elasticsearch --version
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Version: 6.3.2, Build: oss/tar/053779d/2018-07-20T05:20:23.451332Z, JVM: 10.0.2


# docker run docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.2 /opt/jdk-10.0.2/bin/java -version
openjdk version "10.0.2" 2018-07-17
OpenJDK Runtime Environment 18.3 (build 10.0.2+13)
OpenJDK 64-Bit Server VM 18.3 (build 10.0.2+13, mixed mode)

[danielmitterdorfer]

@anjia0532:

(a) this warning is harmless for you.
(b) It is produced by the JVM and not by Elasticsearch. It says that the garbage collector that has been configured for Elasticsearch is deprecated since Java 9. This is ok and expected. We advise against changing the garbage collector.

See http://openjdk.java.net/jeps/291 for details on the deprecation in the JVM.

[anjia0532]

ping @danielmitterdorfer , ok , thanks for your reply.

[bodinsamuel]

Sorry to comment on this closed ticket.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. This warning is causing problem on kubernetes, mark all instances as not fully ready.
While I agree it's harmless it's a bit annoying to have this constant warning, plus it break some features of kubernetes. Do you have any roadmap to share about this ? :)
Thanks

[jasontedor]

@bodinsamuel Would you please elaborate why this is a problem on Kubernetes? Because it’s printed to stderr?

Your message is confusing since you say it’s a “causing problem”, and then “a bit annoying”, and then “it break some features”. Please clarify.

[bodinsamuel]

Sure, sorry for being too vague.
Right now, in a kubernetes cluster (in my case in GCP) it's not critical but because of this warning, all ES instances (Pods) are marked as not "clean". The instance works correctly but because of that, autoscaling/autorepairing does not work anymore. It means GCP can not pop an other instance if needed or do a scheduled maintenance.

And it always display these warnings which make you stop looking at logs and thus probably ignoring other warning.

I'm not an expert neither with kubenertes neither with Java, maybe there is a trick (which I would gladly use). I hope it was clearer this time :p

(it's also possible error come from something else I didn't notice)

[jasontedor]

Now that 6.5.0 is released and we support G1GC on JDK 11, you could switch to a 6.5.0 image and configure the G1GC collector. That would remove these warnings. You should of course do validation that G1GC will be compatible with your workload. It is not a panacea.

[hm43]

This is my solution in lubuntu:
https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html