Elasticsearch: Elasticsearch systemd configuration values getting replaced on update

[rbw]

Hello,

We're running Elasticsearch 2.2 on RHEL7, with the http://packages.elasticsearch.org Yum repository.

The problem:
The Elasticsearch systemd configuration file seems to be getting overridden / replaced with default values on update. Fixing this requires manual intervention after each version change. Keeping these default values causes our Elasticsearch instance to crash after a while.

We're using the following configuration in /usr/lib/systemd/system/elasticsearch.service:
LimitNOFILE=256000
LimitMEMLOCK=infinity

After an update, these values are replaced with the following:
LimitNOFILE=65535

LimitMEMLOCK=infinity

Repository path:
elasticsearch/distribution/src/main/packaging/systemd/elasticsearch.service

Is it supposed to be like this? Are we doing something wrong?

Thanks,

Regards,
Robert

[dakrone]

I believe we may need to change the way the RPM is built to ensure %config(noreplace) is in the file, @rjernst I pinged you about this, do you know the gradle invocation to do that?

[dakrone]

Related: http://www-uxsup.csx.cam.ac.uk/%7Ejw35/docs/rpm_config.html

[dakrone]

Also related: https://bugzilla.redhat.com/show_bug.cgi?id=1296308

[rbw]

I can help look into this

[rbw]

if you want that is, let me know :)

[dakrone]

@rbw0 that would be great, if I recall @rjernst saying, we need to change the gradle calling of the rpm package to include the noreplace. I think this should be a pretty straightforward change, the hardest part is figuring out where it should go :)

[rjernst]

I have a simple change here (6dffaba) that I have confirmed "works" as far as the attributes are set on deb and rpm. However, I don't want to put up a PR until we can actually test with vagrant. So this is blocked on #16854.

[jasontedor]

@rjernst I picked 6dffaba2483a445845266255cab9c584c3681d28 into #17197 where I included a test.

[Xylakant]

The proper way to solve this is to use systemd's override capabilities which are designed to solve this issue. Do not modify the rpm-provided file, but provide an override snippet in /etc/systemd/system/elasticsearch.service.d/. The file should look somewhat like this

[Service]
LimitNOFILE=256000
LimitMEMLOCK=infinity

Since the override snippet is not part of the package it won't get overwritten on update. See the systemd docs and the excellent digitalocean tutorial at https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

[jasontedor]

The proper way to solve this is to use systemd's override capabilities which are designed to solve this issue.

Yes, that's the plan, we just need to add documentation and tests for this. Thanks for the link, I think that it will be helpful for some.

[jasontedor]

I picked 6dffaba2483a445845266255cab9c584c3681d28 into #17197 where I included a test.

@rjernst I backed this out of #17197; I ran into some discrepancies between RPM 4.8.x on CentOS 6 and RPM 4.11.y on CentOS 7 in how it handles /etc/elasticsearch/scripts and I think it'd be better to more carefully investigate the cause of the discrepancies in a separate pull request.

[seang-es]

java.options is affected by this as well. Will fixes taking advantage of systemd's override capability affect that file?

[jasontedor]

This is covered in the docs now.