Elasticsearch: Starts and stops a single elasticsearch instance on this system (code=exited, status=3)

Created on 20 Apr 2015  ·  24Comments  ·  Source: elastic/elasticsearch

$ sudo service elasticsearch start
$ sudo service elasticsearch status

 ● elasticsearch.service - Starts and stops a single elasticsearch instance on this system
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Пн. 2015-04-20 14:15:37 YEKT; 2s ago
     Docs: http://www.elasticsearch.org
  Process: 2478 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.default.config=$CONF_FILE -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR (code=exited, status=3)
 Main PID: 2478 (code=exited, status=3)

апр. 20 14:15:37 MG392 elasticsearch[2478]: ... 4 more
апр. 20 14:15:37 MG392 elasticsearch[2478]: log4j:WARN No appenders could be found for logger (node).
апр. 20 14:15:37 MG392 elasticsearch[2478]: log4j:WARN Please initialize the log4j system properly.
апр. 20 14:15:37 MG392 elasticsearch[2478]: log4j:WARN See http://logging.apache.org/log4j/1.2/...fo.
апр. 20 14:15:37 MG392 elasticsearch[2478]: {1.5.1}: Initialization Failed ...
апр. 20 14:15:37 MG392 elasticsearch[2478]: - ElasticsearchIllegalStateException[Failed to crea...nt]
апр. 20 14:15:37 MG392 elasticsearch[2478]: AccessDeniedException[/usr/share/elasticsearch/data]
апр. 20 14:15:37 MG392 systemd[1]: elasticsearch.service: main process exited, code=exited, st...NTED
апр. 20 14:15:37 MG392 systemd[1]: Unit elasticsearch.service entered failed state.
апр. 20 14:15:37 MG392 systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

Linux MG392 3.19.0-14-generic # 14-Ubuntu SMP Mon Apr 13 22:18:24 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

/usr/share/elasticsearch/bin$ ./elasticsearch -v
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
Version: 1.5.1, Build: 5e38401/2015-04-09T13:41:35Z, JVM: 1.8.0_45

:DeliverPackaging >bug Delivery
Source
picture Fudoshiki

Most helpful comment

I encountered the same issue and finally figured it out that in the process of upgrading elasticsearch on centos (linux) my elasticsearch user and group was recreated and had a new user and group id. The logs and data directories and files within them were still owned by the old user and thus elasticsearch kept failing without giving a clue. Here is the command line that fixed it:

sudo chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/

picture imsaar on 4 Aug 2015
👍23 ❤11 🎉5 😄3

All 24 comments

@Fudoshiki thanks for reporting.

Can you please provide:

  • the complete error log (use journalctl -u elasticsearch.service)
  • the paths declared in your /etc/default/elasticsearch file
tlrx picture tlrx on 20 Apr 2015
👍1 
апр. 20 14:22:00 MG392 systemd[1]: elasticsearch.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
апр. 20 14:22:00 MG392 systemd[1]: Unit elasticsearch.service entered failed state.
апр. 20 14:22:00 MG392 systemd[1]: elasticsearch.service failed.
апр. 20 14:31:10 MG392 systemd[1]: Started Starts and stops a single elasticsearch instance on this system.
апр. 20 14:31:10 MG392 systemd[1]: Starting Starts and stops a single elasticsearch instance on this system...
апр. 20 14:31:10 MG392 elasticsearch[7352]: Failed to configure logging...
апр. 20 14:31:10 MG392 elasticsearch[7352]: org.elasticsearch.ElasticsearchException: Failed to load logging configuration
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:139)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:89)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.bootstrap.Bootstrap.setupLogging(Bootstrap.java:100)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:184)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
апр. 20 14:31:10 MG392 elasticsearch[7352]: Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at java.nio.file.Files.readAttributes(Files.java:1686)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:109)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at java.nio.file.Files.walkFileTree(Files.java:2602)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:123)
апр. 20 14:31:10 MG392 elasticsearch[7352]: ... 4 more
апр. 20 14:31:10 MG392 elasticsearch[7352]: log4j:WARN No appenders could be found for logger (node).
апр. 20 14:31:10 MG392 elasticsearch[7352]: log4j:WARN Please initialize the log4j system properly.
апр. 20 14:31:10 MG392 elasticsearch[7352]: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
апр. 20 14:31:10 MG392 elasticsearch[7352]: {1.5.1}: Initialization Failed ...
апр. 20 14:31:10 MG392 elasticsearch[7352]: - ElasticsearchIllegalStateException[Failed to created node environment]
апр. 20 14:31:10 MG392 elasticsearch[7352]: AccessDeniedException[/usr/share/elasticsearch/data]
апр. 20 14:31:10 MG392 elasticsearch[7352]: org.elasticsearch.ElasticsearchIllegalStateException: Failed to created node environment
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:162)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:213)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
апр. 20 14:31:10 MG392 elasticsearch[7352]: Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:383)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at java.nio.file.Files.createDirectory(Files.java:630)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at java.nio.file.Files.createAndCheckIsDirectory(Files.java:734)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at java.nio.file.Files.createDirectories(Files.java:720)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:105)
апр. 20 14:31:10 MG392 elasticsearch[7352]: at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:160)
апр. 20 14:31:10 MG392 elasticsearch[7352]: ... 4 more
Fudoshiki picture Fudoshiki on 20 Apr 2015 
# Run Elasticsearch as this user ID and group ID
#ES_USER=elasticsearch
#ES_GROUP=elasticsearch

# Heap Size (defaults to 256m min, 1g max)
#ES_HEAP_SIZE=2g

# Heap new generation
#ES_HEAP_NEWSIZE=

# max direct memory
#ES_DIRECT_SIZE=

# Maximum number of open files, defaults to 65535.
#MAX_OPEN_FILES=65535

# Maximum locked memory size. Set to "unlimited" if you use the
# bootstrap.mlockall option in elasticsearch.yml. You must also set
# ES_HEAP_SIZE.
#MAX_LOCKED_MEMORY=unlimited

# Maximum number of VMA (Virtual Memory Areas) a process can own
#MAX_MAP_COUNT=262144

# Elasticsearch log directory
#LOG_DIR=/var/log/elasticsearch

# Elasticsearch data directory
#DATA_DIR=/var/lib/elasticsearch

# Elasticsearch work directory
#WORK_DIR=/tmp/elasticsearch

# Elasticsearch configuration directory
#CONF_DIR=/etc/elasticsearch

# Elasticsearch configuration file (elasticsearch.yml)
#CONF_FILE=/etc/elasticsearch/elasticsearch.yml

# Additional Java OPTS
#ES_JAVA_OPTS=

# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true
Fudoshiki picture Fudoshiki on 20 Apr 2015

Which version of Ubuntu are you using?

tlrx picture tlrx on 20 Apr 2015

15.04

Fudoshiki picture Fudoshiki on 20 Apr 2015

/usr/share/elasticsearch/bin$ ./elasticsearch -v
Version: 1.4.4, Build: c88f77f/2015-02-19T13:05:36Z, JVM: 1.8.0_45

● elasticsearch.service - LSB: Starts elasticsearch
   Loaded: loaded (/etc/init.d/elasticsearch)
   Active: active (running) since Пн. 2015-04-20 14:52:37 YEKT; 1min 15s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11316 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
 Main PID: 7352 (code=exited, status=3)
   CGroup: /system.slice/elasticsearch.service
           └─11335 /usr/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -...

апр. 20 14:52:37 MG392 systemd[1]: Starting LSB: Starts elasticsearch...
апр. 20 14:52:37 MG392 elasticsearch[11316]: * Starting Elasticsearch Server
апр. 20 14:52:37 MG392 elasticsearch[11316]: ...done.
апр. 20 14:52:37 MG392 systemd[1]: Started LSB: Starts elasticsearch.

working

1.5.0 too.

Fudoshiki picture Fudoshiki on 20 Apr 2015

@Fudoshiki Elasticsearch version 1.4.4 does not install a SystemD file for Debian. In this case the SysV init.d script is used.

tlrx picture tlrx on 20 Apr 2015

On mint 17.1 elasticsearch 1.5.1 works too, I think this ubuntu bug

Fudoshiki picture Fudoshiki on 20 Apr 2015

You are hitting the same bug as #10322. As a workaround you can uncomment the paths in the /etc/default/elasticsearch file and it should work.

tlrx picture tlrx on 20 Apr 2015
👍2

thank you, when will available new package in repo?

Fudoshiki picture Fudoshiki on 20 Apr 2015

@Fudoshiki New packages (1.5.3, 1.6.0) have been released and integrate the fix.

This issue is resolved so I'm closing it, feel free to reopen if needed.

tlrx picture tlrx on 17 Jun 2015

Hi,
I think this ticket needs to be re-opened.

I've installed ES 1.7.0 using apt-get.
The system is Debian8 with openJDK 1.7.0.79.

After doing everything in this page, mlock.all still didn't work, max_file_descriptors was still 65K (instead of 262144).

After digging in /etc/default/elasticsearch, I found in the comments a pointer to _/usr/lib/systemd/system/elasticsearch.service_.
After setting LimitNOFILE=262144 and LimitMEMLOCK=infinity, restarting ES, mlock.all was finally true and max_file_descriptors was set to 262144.

I'll be happy to provide additional information as needed.

Thanks,
Yarden

ayashjorden picture ayashjorden on 22 Jul 2015

@ayashjorden thanks for your feedback. You've done the right thing (uncommenting LimitMEMLOCK). I think we should make the documentation clearer about this, but I'm not sure why we should re open this issue. Can you elaborate please?

tlrx picture tlrx on 22 Jul 2015

@tlrx, maybe I posted too fast.
Anyway, I've opened a new issue, so we can leave this one closed and if needed, I can remove my post from here.

Thanks,
Yarden

ayashjorden picture ayashjorden on 22 Jul 2015

I encountered the same issue and finally figured it out that in the process of upgrading elasticsearch on centos (linux) my elasticsearch user and group was recreated and had a new user and group id. The logs and data directories and files within them were still owned by the old user and thus elasticsearch kept failing without giving a clue. Here is the command line that fixed it:

sudo chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/

imsaar picture imsaar on 4 Aug 2015
👍23 ❤11 🎉5 😄3

Thx imsaar ! It worked for me.
(full new installation with apt-get on debian 8, ES 1.6.2, openJDK 1.7.0.79)

will-iam picture will-iam on 5 Aug 2015

@imsaar was this upgrade via the rpm or deb packages, or did you do it manually with the zip or tar.gz packages? If the former, please open a new issue about this.

clintongormley picture clintongormley on 5 Aug 2015

I did it with the .deb package

will-iam picture will-iam on 6 Aug 2015

I opened a new issue here https://github.com/elastic/elasticsearch/issues/12688

clintongormley picture clintongormley on 6 Aug 2015

On Ubuntu I completely removed elasticsearch with these commands:

apt-get purge elasticsearch

rm /usr/lib/systemd/system/elasticsearch.service
rm /var/lib/dpkg/info/elasticsearch.list /var/lib/dpkg/info/elasticsearch.postrm
rm /etc/init.d/elasticsearch
rm /etc/default/elasticsearch
rm -rf /etc/elasticsearch

userdel elasticsearch

And then reinstalled and it worked

gitaarik picture gitaarik on 7 Aug 2015
👍5

+1 @imsaar it worked on openSUSE 13.2

hrahal picture hrahal on 12 Nov 2015

Thank you, @imsaar . I got same problem on Debian 8 with ES 1.6.2 started with openjdk "1.8.0_102"

KIVagant picture KIVagant on 30 Aug 2016

-- Logs begin at Sat 2016-12-10 07:51:47 CST, end at Sat 2016-12-10 08:29:34 CST
Dec 10 07:52:06 Khalili-Mint systemd[1]: Starting LSB: Starts elasticsearch...
Dec 10 07:52:08 Khalili-Mint systemd[1]: Started LSB: Starts elasticsearch.
Dec 10 07:56:20 Khalili-Mint systemd[1]: Started LSB: Starts elasticsearch.
Dec 10 08:13:19 Khalili-Mint systemd[1]: Started LSB: Starts elasticsearch.
Dec 10 08:21:21 Khalili-Mint systemd[1]: Started Elasticsearch.
lines 1-6/6 (END)

ervice elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendo
Active: active (exited) since Sat 2016-12-10 07:52:08 CST; 41min ago
Docs: http://www.elastic.co

Dec 10 07:52:06 Khalili-Mint systemd[1]: Starting LSB: Starts elasticsearch...
Dec 10 07:52:08 Khalili-Mint systemd[1]: Started LSB: Starts elasticsearch.
Dec 10 07:56:20 Khalili-Mint systemd[1]: Started LSB: Starts elasticsearch.
Dec 10 08:13:19 Khalili-Mint systemd[1]: Started LSB: Starts elasticsearch.
Dec 10 08:21:21 Khalili-Mint systemd[1]: Started Elasticsearch.
l

ghoroubi picture ghoroubi on 10 Dec 2016

@imsaar life saver! thank you so much from centos7 , java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64

ohoroyoi picture ohoroyoi on 24 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

malpani picture malpani  ·  3Comments
abrahamduran picture abrahamduran  ·  3Comments
DhairyashilBhosale picture DhairyashilBhosale  ·  3Comments
ppf2 picture ppf2  ·  3Comments
rbayliss picture rbayliss  ·  3Comments