Elasticsearch: Elasticsearch stops itself randomly.. No error visible

Created on 17 Nov 2012 · 14Comments · Source: elastic/elasticsearch

I installed the Debian package (0.19.11) on Ubuntu 12.04.1 LTS. Running with Oracle Java 7 had this same issue happening on OpenJDK 7

java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)

Basically after running for about 20 minutes elasticsearch will just stop running. I can't find a crash in the error log or even a shutdown sequence it's just not running anymore.

I am a little unsure what queries are being run. I am running it with Graylog2. But I don't think that is the issue as I turned off the Graylog daemons and even with nothing connected elasticsearch still terminated itself.

Graylog2 Server - 0.9.6p1 - Current Stable
Graylog2 Web Interface - 0.9.6p1 - Current Stable

Source

KellyLSB

All 14 comments

Check syslog and kern.log around the time when elasticsearch disappeared, do you see anything about oom-killer there?

imotov on 17 Nov 2012

Yes there is. I am running this on an Amazon EC2 Micro instance. This server only serves analytical and logging data and receives minimal web traffic.

[809124.687157] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[809124.687165] [  246]     0   246     4308       46   0       0             0 upstart-udev-br
[809124.687169] [  250]     0   250     5367       97   0     -17         -1000 udevd
[809124.687172] [  309]     0   309     5366       98   0     -17         -1000 udevd
[809124.687176] [  313]     0   313     5366       98   0     -17         -1000 udevd
[809124.687179] [  377]     0   377     1816      124   0       0             0 dhclient3
[809124.687183] [  382]     0   382     3797       47   0       0             0 upstart-socket-
[809124.687187] [  595]     0   595    12489      151   0     -17         -1000 sshd
[809124.687190] [  604]   102   604     5982       80   0       0             0 dbus-daemon
[809124.687194] [  621]   101   621    63429      103   0       0             0 rsyslogd
[809124.687197] [  674]     0   674     3626       42   0       0             0 getty
[809124.687201] [  683]     0   683     3626       40   0       0             0 getty
[809124.687204] [  685]     0   685   140793    31479   0       0             0 ruby
[809124.687208] [  689]     0   689     3626       42   0       0             0 getty
[809124.687211] [  691]     0   691     3626       42   0       0             0 getty
[809124.687215] [  693]     0   693     3626       41   0       0             0 getty
[809124.687218] [  702]   107   702    87952     6706   0       0             0 mongod
[809124.687222] [  704]     0   704     1082       37   0       0             0 acpid
[809124.687225] [  705]     0   705     4778       56   0       0             0 cron
[809124.687228] [  706]     0   706     4227       40   0       0             0 atd
[809124.687232] [  724]   103   724    46897      305   0       0             0 whoopsie
[809124.687235] [  742]     0   742    15706      256   0       0             0 nginx
[809124.687239] [  744]    33   744    15785      330   0       0             0 nginx
[809124.687243] [  745]    33   745    15785      330   0       0             0 nginx
[809124.687246] [  746]    33   746    15861      371   0       0             0 nginx
[809124.687250] [  747]    33   747    15785      330   0       0             0 nginx
[809124.687254] [  759]     0   759    14767      641   0       0             0 php5-fpm
[809124.687257] [  760]    33   760    14767      635   0       0             0 php5-fpm
[809124.687261] [  761]    33   761    14767      635   0       0             0 php5-fpm
[809124.687264] [  762]    33   762    14767      635   0       0             0 php5-fpm
[809124.687268] [  763]    33   763    14767      635   0       0             0 php5-fpm
[809124.687271] [  795]     0   795     3626       41   0       0             0 getty
[809124.687275] [ 1016]  1000  1016     6650      203   0       0             0 tmux
[809124.687279] [ 1017]  1000  1017     1100       25   0       0             0 sh
[809124.687282] [ 1021]  1000  1021     5429      639   0       0             0 bash
[809124.687285] [ 2267]   106  2267   474148    58629   0       0             0 java
[809124.687289] [ 3036]     0  3036   223097    38592   0       0             0 java
[809124.687292] Out of memory: Kill process 2267 (java) score 389 or sacrifice child
[809124.687308] Killed process 2267 (java) total-vm:1896592kB, anon-rss:234516kB, file-rss:0kB

KellyLSB on 17 Nov 2012

Basically, what happens here is system runs out of physical memory and kills the es out of self preservation. There are only two things you can do - reduce memory consumed by es or add more memory to the server (move to a larger instance in your case). Considering that micro instance has only 613M of memory, I would suggest the latter approach unless you have really small index.

imotov on 17 Nov 2012

👍1

yah i just went ahead and boosted the memory. thanks.

KellyLSB on 17 Nov 2012

I have same problem in VPS with 2 gb ram memory, please help

johnfelipe on 15 May 2015

I'm having the same problem on an r3.2xlarge instance with 60 GB of RAM.

OS: Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-48-generic x86_64).
I increased the heap size to 40GB, so that is definitely not the problem. The only message I see in syslog is:

Jun 25 17:53:57 ip-172-31-14-128 kernel: [42761.897554] init: elasticsearch main process (3697) terminated with status 143
Jun 25 17:53:57 ip-172-31-14-128 kernel: [42761.897566] init: elasticsearch main process ended, respawning
Jun 25 17:55:21 ip-172-31-14-128 kernel: [42846.452398] init: elasticsearch main process (4003) terminated with status 143

brianvoss on 25 Jun 2015

That's elasticsearch being killed by a sigterm signal - the same one sent by kill <pid>

You may want to check if it was killed by the oomkiller - I don't remember if it tries a sigterm before the sigkill but it might and that'd look like this.

If it is the oomkiller this isn't an Elasticsearch bug - its you giving the process more memory than the machine can let it have. If.

References:
http://tldp.org/LDP/abs/html/exitcodes.html#EXITCODESREF
http://unixhelp.ed.ac.uk/CGI/man-cgi?signal+7

nik9000 on 25 Jun 2015

👍1

This might be old, but you might get an error such as this:

-bash: fork: Cannot allocate memory

...if you try starting a new instance.
Bottom like as stated, it's usually a memory issue when it randomly terminated.

cnvo on 14 Jun 2016

i am having the same issue, i have 4gb ram and a 2 core cpu running elasticsearch 2.4.0 on ubuntu 14.04.
Its getting stopped without any error visible in the logs.
Can someone help me in this

keshav-marketingmindz on 28 Sep 2016

👍1

@keshav-marketingmindz Did you solve your issue? I am in the same boat with you.

nextofsearch on 30 Oct 2016

@nextofsearch what do you see in syslog for the time when elasticsearch died?

imotov on 31 Oct 2016

@nextofsearch Yes i resolved it, i used two different servers both are centOS. One server for elasticsearch only and one for other such as php mysql and our website. This worked for me. Hope will work for you too.
Thanks

keshav-marketingmindz on 2 Nov 2016