Elasticsearch: Path to file elasticsearch.pid not exist after restart

[konstantinkostin28]

During start elasticsearch (version 1.6) service after restart ubuntu machine I have following error:

touch: cannot touch ‘/var/run/elasticsearch/elasticsearch.pid’: No such file or directory

This occurs because elasticsearch.pid move to /elasticsearch directory (PID_DIR="/var/run/elasticsearch", PID_FILE="$PID_DIR/$NAME.pid") and ubuntu remove this after restart. Command "touch" does not create parent directories.

In version 1.5 the .pid file was in /var/run/ directiry (PID_FILE="/var/run/$NAME.pid") and so was no error.

[tlrx]

@konstantinkostin28 thanks for your reporting and you're right, the PID_DIR should be created before touch the file.

[GlenRSmith]

I get the same behavior on an Ubuntu 14.04 VM. Was fine on v1.5.2, broken on 1.6.0.

[yett]

@tlrx ,

1. Should I change the PID_DIR value to `/var/run/' or
2. Change script to create "$PID_DIR" before touch as following?

154   mkdir -p "$LOG_DIR" "$DATA_DIR" "$WORK_DIR" "$PID_DIR" && chown "$ES_USER":"$ES_GROUP" "$LOG_DIR" "$DATA_DIR" "$WORK_DIR" "$PID_DIR"
155   touch "$PID_FILE" && chown "$ES_USER":"$ES_GROUP" "$PID_FILE"

I did the approach 2, and it works. I am just curious about the proper way. Thanks!

[tlrx]

@yett My personal preference goes to /var/run/elasticsearch/elasticsearch.pid (approach 2). The thing is to have a consistent, secured way to read/write this file.

[yett]

Thanks for the tips. Cheers!

[jerrac]

I just upgraded to 1.6 today and ran into this issue. Annoyingly, the /var/run/elasticsearch directory I created before upgrading to 1.6 (in an attempt to avoid this issue on other nodes...) went away after I rebooted (I had non-ES reasons to reboot).

This is on Ubuntu 12.04. I was going from 1.4.5 to 1.6, using the apt repo.

Is there a reason apt doesn't deal with this for us?

[GlenRSmith]

On my Ubuntu 14.04 instance, there is, in fact a /usr/lib/tmpfiles.d/elasticsearch.conf:
d /var/run/elasticsearch 0755 root root - -
This doesn't seem to get handled. In fact, I have other tmpfile entries that don't seem to be processed.

[uschindler]

Same issue here. I fixed this by putting PID_DIR="/var/run" into /etc/default/elasticsearch.
In my opinion, on Ubuntu/Debian all pid files are directly in this folder, so PID_DIR by default should just be /var/run

[DASAR]

Please fix this issue in official Elasticsearch repository (ppa) for Ubuntu.

[GlenRSmith]

@uschindler Can you elaborate on "on Ubuntu/Debian all pid files are directly in this folder"? On my Ubuntu 14.04 LTS install with only Elasticsearch as an additional install, I've got a number of services whose PID files are in their on subdirectories in /var/run (CUPS, for example).

[uschindler]

Daemons generally only create a sub-dir if they create more than just a PID as run file. But generally, the pid file should be placed in /var/run and have ending ".pid".

I have to first lookup the packaging rules in Debian, but there is something about that in it!

[tlrx]

Sorry for the annoyance. I pushed a fix in #11674, it basically creates & chown the PID_DIR in init.d scripts and had been tested on Ubuntu 12.04, 14.04, Debian 7.8, CentOS6.6 and few others.

The Debian Guidelines are pretty open concerning PID files as long as it is stored in /var/run (source: https://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.txt):

Programs may have a subdirectory of /var/run; this is encouraged for programs that use more than
one run-time file.

For defaults, we'd like to keep the current /var/run/elasticsearch directory. It was already the case on RPM based distributions and keeping things consistent across platforms make scripts easier to maintain. It also helps when multiple instances runs on the same machine and make things simplier when the JVM Security Manager is enabled. Note that it can been overridden using the PID_DIR environment variable as @uschindler already mentioned.

[arbourd]

@tlrx Thanks for your work on this. :)

[uschindler]

@tlrx Thanks. Patch looks good.

In any case the packaging manuals already say what I mentioned in my first post:

/var/run : Run-time variable data

Purpose

This directory contains system information data describing the system since it
was booted. Files under this directory must be cleared (removed or truncated as
appropriate) at the beginning of the boot process. Programs may have a
subdirectory of /var/run; this is encouraged for programs that use more than
one run-time file. [42] Process identifier (PID) files, which were originally
placed in /etc, must be placed in /var/run. The naming convention for PID files
is .pid. For example, the crond PID file is named /var/run/
crond.pid.

Your text was just a bit clipped. Basically it says: If there is only the PID file and nothing more of runtime data it should be placed directly in /var/run. Otherwise a subdirectory with the name of the program should be created ("this is encouraged for programs that use more than one run-time file").

But I am also fine with your pull request. Thanks in any case!

[GlenRSmith]

@tlrx Thanks!

[bskern]

When will this get released? I noticed this version here does not contain these changes

[tlrx]

@bskern the issue has been reported for 1.6.0, and the package you indicate is 1.6.0 too so it does not contain the fix.

The fix will will be released with 1.6.1, hopefully soon.

[baelter]

Also an issue on manjaro

[yashvit]

Linux Version: Ubuntu 16.04
Elasticsearch Version: 1.7.3

I have recently added some new nodes to my existing cluster, and randomly one server would not start elasticsearch on restart. When manually trying to start it gives the following error:

sudo systemctl status elasticsearch

* elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2017-06-16 12:32:28 UTC; 3s ago
     Docs: http://www.elastic.co
  Process: 1437 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=$PID_DIR/elasticsearch.pid -Des.default.path.home=$ES_HOME -Des.default.p
 Main PID: 1437 (code=exited, status=3)

Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]: java.io.FileNotFoundException: /var/run/elasticsearch/elasticsearch.pid (No such file or directory)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at java.io.FileOutputStream.open0(Native Method)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at java.io.FileOutputStream.open(FileOutputStream.java:270)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:194)
Jun 16 12:32:28 els02.jobsoid.net elasticsearch[1437]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Jun 16 12:32:28 els02.jobsoid.net systemd[1]: elasticsearch.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jun 16 12:32:28 els02.jobsoid.net systemd[1]: elasticsearch.service: Unit entered failed state.
Jun 16 12:32:28 els02.jobsoid.net systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

If I create a folder /var/run/elasticsearch and change ownership to elasticsearch:elasticsearch then elasticsearch starts. On restart again the same issue.

I understand this above issue is related to version 1.6 but still I tried changing the PID_DIR folder to /var/run in the service file at /usr/lib/systemd/system/elasticsearch.service

with that it gives an error: java.io.FileNotFoundException: /var/run/elasticsearch.pid (Permission denied) instead of "no such file or directory"

One other thing i noticed, when I manually create the dir in /var/run and force elasticsearch to start, it seems to work fine but the GET /_nodes does not return any stats of the OS or Filesystem.

Makes me think its some permission issue?

[clintongormley]

@yashvit it's really time to upgrade :)

[ghanshyamca]

Resolved issue for "Elasticsearch.service: Can’t open PID file /run/elasticsearch/elasticsearch.pid No such file or directory"

By running the below command:-

sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch