Elasticsearch: Default jvm.options specifies GC which has been removed in Enterprise Linux OpenJDK 11 build

[pa-jberanek]

Elasticsearch version (bin/elasticsearch --version):
7.9.3
Plugins installed: []

JVM version (java -version):
openjdk version "11.0.8" 2020-07-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.8+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.8+10-LTS, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):
CentOS 7.8
3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
The jvm.options file in the Enterprise Linux RPM specifies:

8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

However the OpenJDK 11 in Enterprise Linux 7 no longer supports UseConcMarkSweepGC, so if you try to start elasticsearch it fails to start with something like:

systemd-entrypoint[1070]: Unrecognized VM option 'UseConcMarkSweepGC'

Steps to reproduce:

On Enterprise Linux 7 system:

dnf install java-11-openjdk-headless-11.0.8.10-0.el7_8.x86_64

Install elasticsearch 7.9.3 from ES repo

Try to start elasticsearch systemd unit

Provide logs (if relevant):

[jasontedor]

Sorry that you're having trouble. Alas, I am not able to reproduce this, and it would surprise me if the LTS build of JDK 11 removed CMS since the removal was only slated for JDK 14. The implementation of this removal was that CMS was removed in JDK 14, but the flags were ignored, and then in JDK 15 it was an error to use the CMS flags.

I installed the package that you indicated, and configured JAVA_HOME in /etc/sysconfig/elasticsearch to point to this JDK:

centos-7:~$ sudo cat /etc/sysconfig/elasticsearch | grep -i JAVA_HOME
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-11.0.8.10-0.el7_8.x86_64

Then I started Elasticsearch using systemctl, and by hitting the endpoint /_nodes/jvm?pretty=true&&filter_path=**.jvm we can see that it is using JDK 11.0.8 from the package that you indicated.

centos-7:~$ curl -XGET "localhost:9200/_nodes/jvm?pretty=true&&filter_path=**.jvm"
{
  "nodes" : {
    "9LSTad4hQseFxpNNUkw2hg" : {
      "jvm" : {
        "pid" : 16308,
        "version" : "11.0.8",
        "vm_name" : "OpenJDK 64-Bit Server VM",
        "vm_version" : "11.0.8+10-LTS",
        "vm_vendor" : "Oracle Corporation",
        "bundled_jdk" : true,
        "using_bundled_jdk" : false,
        "start_time_in_millis" : 1603827366129,
        "mem" : {
          "heap_init_in_bytes" : 1073741824,
          "heap_max_in_bytes" : 1038876672,
          "non_heap_init_in_bytes" : 7667712,
          "non_heap_max_in_bytes" : 0,
          "direct_max_in_bytes" : 0
        },
        "gc_collectors" : [
          "ParNew",
          "ConcurrentMarkSweep"
        ],
        "memory_pools" : [
          "CodeHeap 'non-nmethods'",
          "Metaspace",
          "CodeHeap 'profiled nmethods'",
          "Compressed Class Space",
          "Par Eden Space",
          "Par Survivor Space",
          "CodeHeap 'non-profiled nmethods'",
          "CMS Old Gen"
        ],
        "using_compressed_ordinary_object_pointers" : "true",
        "input_arguments" : [
          "-Xshare:auto",
          "-Des.networkaddress.cache.ttl=60",
          "-Des.networkaddress.cache.negative.ttl=10",
          "-XX:+AlwaysPreTouch",
          "-Xss1m",
          "-Djava.awt.headless=true",
          "-Dfile.encoding=UTF-8",
          "-Djna.nosys=true",
          "-XX:-OmitStackTraceInFastThrow",
          "-Dio.netty.noUnsafe=true",
          "-Dio.netty.noKeySetOptimization=true",
          "-Dio.netty.recycler.maxCapacityPerThread=0",
          "-Dio.netty.allocator.numDirectArenas=0",
          "-Dlog4j.shutdownHookEnabled=false",
          "-Dlog4j2.disable.jmx=true",
          "-Djava.locale.providers=SPI,COMPAT",
          "-Xms1g",
          "-Xmx1g",
          "-XX:+UseConcMarkSweepGC",
          "-XX:CMSInitiatingOccupancyFraction=75",
          "-XX:+UseCMSInitiatingOccupancyOnly",
          "-Djava.io.tmpdir=/tmp/elasticsearch-1198748067177237691",
          "-XX:+HeapDumpOnOutOfMemoryError",
          "-XX:HeapDumpPath=/var/lib/elasticsearch",
          "-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log",
          "-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m",
          "-XX:MaxDirectMemorySize=536870912",
          "-Des.path.home=/usr/share/elasticsearch",
          "-Des.path.conf=/etc/elasticsearch",
          "-Des.distribution.flavor=default",
          "-Des.distribution.type=rpm",
          "-Des.bundled_jdk=true"
        ]
      }
    }
  }
}

So I do not think the problem here is JDK 11.

[pa-jberanek]

Thank you for trying to reproduce the issue, and I learnt something new today...

The 'elasticsearch' RPM from the Elasticsearch yum repository comes with its own bundled JVM, which with 7.9.3 is Java 15!

The actual issue is that the server in question had been through numerous ES updates, but jvm.options had been modified by us to change the heap size. Due to this modification RPM updates created jvm.options.rpmnew, which we never reconciled back into jvm.options.

What this meant is we ended up with an elasticsearch RPM including JVM 15 but with a jvm.options which read:

## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
# 10-:-XX:-UseConcMarkSweepGC
# 10-:-XX:-UseCMSInitiatingOccupancyOnly
#10-:-XX:+UseG1GC
#10-:-XX:G1ReservePercent=25
#10-:-XX:InitiatingHeapOccupancyPercent=30

I've now read somewhere else that the suggestion is to not modify jvm.options to modify the heap, and instead to add a separate file in jvm.options.d to override the heap. If this is done then subsequent updates of the elasticsearch RPM will happily overwrite the jvm.options file with any new config required for the new JVM included in the RPM.

```

[pa-jberanek]

Looking at the RPM one could also use the ES_JAVA_OPTS variable in /etc/sysconfig/elasticsearch. Both methods are mentioned in the current https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

[pa-jberanek]

Given all the above, I'm happy to close this ticket.

[jasontedor]

The 'elasticsearch' RPM from the Elasticsearch yum repository comes with its own bundled JVM, which with 7.9.3 is Java 15!

That's right, and that's what we encourage users to be using. 😇

Due to this modification RPM updates created jvm.options.rpmnew, which we never reconciled back into jvm.options.

Yeah, it's exactly for this reason that we introduced the jvm.options.d mechanism in #51882.

Looking at the RPM one could also use the ES_JAVA_OPTS variable in /etc/sysconfig/elasticsearch.

Yeah, while this mechanism exists, it is mostly useful for testing and we do not consider it the preferred way to configure the heap size.

I'm glad that you were able to get to the bottom of the issue. Thank you for reporting back.

[jasontedor]

I opened #64256 to clarify the docs.

[pa-jberanek]

Another doc location this is discussed?

https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html

[jasontedor]

@pa-jberanek There is already a comment on that page that the preferred way of configuring JVM options is via jvm.options.d.