Elasticsearch: Unrecognized VM option 'UseConcMarkSweepGC' on OpenJDK 15

[polyfractal]

Node failed to boot because UseConcMarkSweepGC isn't supported, presumably because it was removed in OpenJDK 15?

https://gradle-enterprise.elastic.co/s/kloq36r7j4e3w

ES_BUILD_JAVA=openjdk13,ES_RUNTIME_JAVA=openjdk15,nodes=general-purpose

OS | openSUSE Leap 15.1, Linux 4.12.14-lp151.28.36-default

Specs | 32 CPUs, 94.4GB RAM
java.version | 13.0.2
java.vm.name | OpenJDK 64-Bit Server VM
java.vm.version | 13.0.2+8
java.runtime.version | 13.0.2+8
java.home | /var/lib/jenkins/.java/openjdk-13.0.2-linux

12:31:34 » Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
12:31:34 »  output:
12:31:34 »  
12:31:34 »  error:
12:31:34 »  Unrecognized VM option 'UseConcMarkSweepGC'
12:31:34 »  Error: Could not create the Java Virtual Machine.
12:31:34 »  Error: A fatal exception has occurred. Program will exit.
12:31:34 »      at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118)
12:31:34 »      at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86)
12:31:34 »      at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
12:31:34 »      at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:92)

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[polyfractal]

Probably related to: https://github.com/elastic/elasticsearch/pull/51335 :)

[jasontedor]

This is because:

• JDK 15 removed support for the CMS-related flags
• this is a build with a JDK 15 runtime Java
• we are trying to start the 7.5 BWC cluster using the same runtime Java (JDK 15)
• not using CMS-related flags (#49123) was not backported to 7.5 since there was no expectation of supporting JDK 14 nor JDK 15 on 7.5

We use to have it in the build that when we started the BWC cluster, we would use a version of the JDK that the version that we were starting was known to be compatible with. I think that when starting older clusters, we should use the bundled JDK overriding the runtime JDK. That's what we need here, and in general, so that we are only starting clusters against versions of the JDK that the version being started supports. Can you investigate making this change @mark-vieira?

[mark-vieira]

I think that when starting older clusters, we should use the bundled JDK overriding the runtime JDK.

Agreed. In fact we have an open issue to use the configured bundled JDK version for all integration testing (except Java matrix tests).

That's what we need here, and in general, so that we are only starting clusters against versions of the JDK that the version being started supports.

We don't have a notion of _maximum_ runtime Java version defined in an easily consumable way that I'm aware of. In any case, when we fetch BWC versions from a archive distribution we don't have any kind of metadata here either. I suppose we can hardcode this, which is probably fine given we only have to go back 1 major version and moving forward we always rely on using the bundled JDK. I'll work up a solution here.

[jasontedor]

That brings us to #43071 so it’s only that we never implemented this for testclusters.

We don't have a notion of maximum runtime Java version defined in an easily consumable way that I'm aware of.

We use to have this, but hard coded into the build as you suggest. It was never pretty to maintain. At some point we moved to using some content out of .ci (since we know the versions we test against there) but I’m having trouble tracking that down on mobile.

[mark-vieira]

Yeah, that PR you mentioned got blown away with the migration to test clusters and it seems we reintroduced the "just use whatever runtime java is" logic. I'm working up a PR for this. I'm not too worried about maintenance. We only have to consider all current active BWC versions < 7.2, which we know to work on Java 13. I don't suspect we'll ever have to change this as going forward we won't introduce any new versions that don't also include a bundled JDK.

[droberts195]

As well as the problem of 7.x being BWC tested against older versions that don't support Java 15, there is also the problem that we appear to be using Java 15 in CI builds for branches older than 7.x. For example, https://gradle-enterprise.elastic.co/s/mp33jqizng4l2 is a 7.5 branch build that tried to use Java 15 and failed because UseConcMarkSweepGC wasn't recognised.

[mark-vieira]

Good catch. Yes, as per Jason's comment we should not be testing 7.5 on Java 14 or 15. I've removed that with 24f026793bd236556dd8cefb0ab795c81e49ff09.

[cbuescher]

@mark-vieira not sure, but this failure from today on 7.6 might be related still https://gradle-enterprise.elastic.co/s/asqniuub2yzlm
https://gradle-enterprise.elastic.co/s/asqniuub2yzlm/console-log#L6326

[cbuescher]

This is also from this morning on 7.6, maybe related https://gradle-enterprise.elastic.co/s/3g43ngv3tvm2w/console-log?task=:qa:verify-version-constants:v7.5.3%23integTest

[mark-vieira]

This has not been backported to 7.x or 7.6. This issue was mistakenly closed when the master PR got merged.

[mark-vieira]

This should be sorted out now that https://github.com/elastic/elasticsearch/pull/51505 has been backported.

[hgshoggins]

Just updated two instances of ES on different machines (with exact same system and OpenJDK versions though). One on 7.9.1, working fine, one on 7.9.2, having this exact issue reporting the unrecognized VM option.

[geerlingguy]

FYI I'm hitting this on OpenJDK 11 now:

Sep 26 04:10:48 instance systemd[1]: Starting Elasticsearch...
Sep 26 04:10:49 instance systemd-entrypoint[4660]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
Sep 26 04:10:49 instance systemd-entrypoint[4660]: output:
Sep 26 04:10:49 instance systemd-entrypoint[4660]: error:
Sep 26 04:10:49 instance systemd-entrypoint[4660]: Unrecognized VM option 'UseConcMarkSweepGC'
Sep 26 04:10:49 instance systemd-entrypoint[4660]: Error: Could not create the Java Virtual Machine.
Sep 26 04:10:49 instance systemd-entrypoint[4660]: Error: A fatal exception has occurred. Program will exit.
Sep 26 04:10:49 instance systemd-entrypoint[4660]:         at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:126)
Sep 26 04:10:49 instance systemd-entrypoint[4660]:         at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
Sep 26 04:10:49 instance systemd-entrypoint[4660]:         at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
Sep 26 04:10:49 instance systemd-entrypoint[4660]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137)
Sep 26 04:10:49 instance systemd-entrypoint[4660]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
Sep 26 04:10:49 instance systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 26 04:10:49 instance systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Everything was working fine on new builds (executed weekly) until yesterday.

Related: https://github.com/elastic/elasticsearch/issues/62716

[ahantoiu]

I commented this GC configuration directives in "/etc/elasticsearch/jvm.options" and it works for me but I am not sure if this is the right thing to do.

GC configuration

-XX:+UseConcMarkSweepGC

-XX:CMSInitiatingOccupancyFraction=75

-XX:+UseCMSInitiatingOccupancyOnly

ES version:

{
"name" : "",
"cluster_name" : "",
"cluster_uuid" : "*",
"version" : {
"number" : "7.9.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "d34da0ea4a966c4e49417f2da2f244e3e97b4e6e",
"build_date" : "2020-09-23T00:45:33.626720Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Java version:

openjdk 11.0.8 2020-07-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.8+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.8+10-LTS, mixed mode, sharing)

[glmrenard]

Did the same it works, but I'm not that confident. Here my java version

openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)

[MrMontesa]

Same here. Updated my CentOS 7 server today and ran into the same issue. Tip from @ahantoiu worked for me as well.
My system:

[root@host elasticsearch]# rpm -qa |grep -E "openjdk|elastic"
java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64
elasticsearch-7.9.2-1.x86_64
java-1.8.0-openjdk-headless-1.8.0.262.b10-0.el7_8.x86_64

[abcqwertz]

I encountered the same problem (CentOS 7, elasticsearch 7.9.2). I made some research and found this elastic-disscusion.
I replaced:

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

with

8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

in my jvm.options file. After that, elastic-service ist starting without problems. In Elastic 7.9.2 the bundled JDK was upgraded to version 15. You can see this in the Pull-Request linked to the discussion link. Hope this helps.

[ksaylor11]

I commented this GC configuration directives in "/etc/elasticsearch/jvm.options" and it works for me but I am not sure if this is the right thing to do.

GC configuration

-XX:+UseConcMarkSweepGC

-XX:CMSInitiatingOccupancyFraction=75

-XX:+UseCMSInitiatingOccupancyOnly

ES version:

{
"name" : "_*", "cluster_name" : "_",
"cluster_uuid" : "",
"version" : {
"number" : "7.9.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "d34da0ea4a966c4e49417f2da2f244e3e97b4e6e",
"build_date" : "2020-09-23T00:45:33.626720Z",
"build_snapshot" : false,
"lucene_version" : "8.6.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

Java version:

openjdk 11.0.8 2020-07-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.8+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.8+10-LTS, mixed mode, sharing)

Just had to do the same, has anyone run into problems yet?

[Atreiide]

Same issue.

CentOS 7

/usr/share/elasticsearch/jdk/bin/java -version :

openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)

I downgraded to 7.9.1 to solve the issue.

Why issue has been closed ?

[CodeNameTheOnlyOne]

also having this issue after doing a yum update
openjdk version "15" 2020-09-15
OpenJDK Runtime Environment AdoptOpenJDK (build 15+36)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15+36, mixed mode, sharing)

commented the lines in /etc/elasticsearch/jvm.options out and it starts now.
not sure why this issue is closed

[jasontedor]

Elasticsearch ships a JVM options file. We might update this between releases. If you bring along your old one and don't merge it properly with the new one we ship, then you might have options in there from an older version of the JVM that are no longer supported on newer versions of the JVM. We haven't shipped with this flag in our default options file since 7.7.0, and it's not in the packages we are shipping now. If it's in your options file, you need to remove it, we can't do it for you.

[davama]

I encountered the same problem (CentOS 7, elasticsearch 7.9.2). I made some research and found this elastic-disscusion.
I replaced:

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

with

8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

in my jvm.options file. After that, elastic-service ist starting without problems. In Elastic 7.9.2 the bundled JDK was upgraded to version 15. You can see this in the Pull-Request linked to the discussion link. Hope this helps.

@abcqwertz
This worked for me!
Thank you!

my environment:

java-1.8.0-openjdk-headless-1.8.0.272.b10-1.el7_9.x86_64
java-1.8.0-openjdk-1.8.0.272.b10-1.el7_9.x86_64
elasticsearch-7.10.0-1.x86_64