Elasticsearch: sysctl: permission denied on key 'vm.max_map_count' - OpenVZ Elasticsearch 0.90.9 compatibility issue

[bline79]

Hello,

Elasticsearch 0.90.9 and higher do not work with openvz (tested with debian wheezy i386) with the following error showing on startup:

Starting ElasticSearch Server:sysctl: permission denied on key 'vm.max_map_count'

[spinscale]

Hey,

can you please check, if the process started anyway? It should have been. The error is just a permission problem due to your VM, but should not prevent elasticsearch from starting usually.

[dmikhaylov]

I've got the same issue, and on my VM it's not starting.

[spinscale]

can you add set -x to the elasticsearch init script and paste the output from /etc/init.d/elasticsearch start here?

[dmikhaylov]

do you mean /etc/init.d/elasticsearch start set -x?

[spinscale]

sorry for not being clear. You can do the following: Open /etc/init.d/elasticsearch in your favourite editor and change the current setup

#!/bin/sh
#

to

#!/bin/sh
set -x
#

Then run /etc/init.d/elasticsearch start and copy paste all that output into this ticket. You might want to comment out or remove that line again after that, so you do not get his verbose output all the time.

Thanks a lot for helping!

[dmikhaylov]

• PATH=/bin:/usr/bin:/sbin:/usr/sbin
• NAME=elasticsearch
• DESC='ElasticSearch Server'
• DEFAULT=/etc/default/elasticsearch
  ++ id -u
• '[' 0 -ne 0 ']'
• . /lib/lsb/init-functions
  ++ FANCYTTY=
  ++ '[' -e /etc/lsb-base-logging.sh ']'
  ++ . /etc/lsb-base-logging.sh
  +++ LOG_DAEMON_MSG=
• '[' -r /etc/default/rcS ']'
• . /etc/default/rcS
  ++ TMPTIME=0
  ++ SULOGIN=no
  ++ DELAYLOGIN=no
  ++ UTC=yes
  ++ VERBOSE=no
  ++ FSCKFIX=no
• ES_USER=elasticsearch
• ES_GROUP=elasticsearch
• JDK_DIRS='/usr/lib/jvm/java-7-oracle /usr/lib/jvm/java-7-openjdk /usr/lib/jvm/java-7-openjdk-amd64/ /usr/lib/jvm/java-7-openjdk-armhf /usr/lib/jvm/java-7-openjdk-i386/ /usr/lib/jvm/java-6-sun /usr/lib/jvm/java-6-openjdk /usr/lib/jvm/java-6-openjdk-amd64 /usr/lib/jvm/java-6-openjdk-armhf /usr/lib/jvm/java-6-openjdk-i386 /usr/lib/jvm/default-java'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-7-oracle/bin/java -a -z '' ']'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-7-openjdk/bin/java -a -z '' ']'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-7-openjdk-amd64//bin/java -a -z '' ']'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-7-openjdk-armhf/bin/java -a -z '' ']'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-7-openjdk-i386//bin/java -a -z '' ']'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-6-sun/bin/java -a -z '' ']'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-6-openjdk/bin/java -a -z '' ']'
• JAVA_HOME=/usr/lib/jvm/java-6-openjdk
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-6-openjdk-amd64/bin/java -a -z /usr/lib/jvm/java-6-openjdk ']'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-6-openjdk-armhf/bin/java -a -z /usr/lib/jvm/java-6-openjdk ']'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/java-6-openjdk-i386/bin/java -a -z /usr/lib/jvm/java-6-openjdk ']'
• for jdir in '$JDK_DIRS'
• '[' -r /usr/lib/jvm/default-java/bin/java -a -z /usr/lib/jvm/java-6-openjdk ']'
• export JAVA_HOME
• ES_HOME=/usr/share/elasticsearch
• MAX_OPEN_FILES=65535
• LOG_DIR=/var/log/elasticsearch
• DATA_DIR=/var/lib/elasticsearch
• WORK_DIR=/tmp/elasticsearch
• CONF_DIR=/etc/elasticsearch
• CONF_FILE=/etc/elasticsearch/elasticsearch.yml
• MAX_MAP_COUNT=65535
• '[' -f /etc/default/elasticsearch ']'
• . /etc/default/elasticsearch
• PID_FILE=/var/run/elasticsearch.pid
• DAEMON=/usr/share/elasticsearch/bin/elasticsearch
• DAEMON_OPTS='-p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch'
• export ES_HEAP_SIZE
• export ES_HEAP_NEWSIZE
• export ES_DIRECT_SIZE
• export ES_JAVA_OPTS
• test -x /usr/share/elasticsearch/bin/elasticsearch
• case "$1" in
• checkJava
• '[' -x /usr/lib/jvm/java-6-openjdk/bin/java ']'
• JAVA=/usr/lib/jvm/java-6-openjdk/bin/java
• '[' '!' -x /usr/lib/jvm/java-6-openjdk/bin/java ']'
• '[' -n '' -a -z '' ']'
• log_daemon_msg 'Starting ElasticSearch Server'
• '[' -z 'Starting ElasticSearch Server' ']'
• log_use_fancy_output
• TPUT=/usr/bin/tput
• EXPR=/usr/bin/expr
• '[' -t 1 ']'
• '[' xxterm '!=' x ']'
• '[' xxterm '!=' xdumb ']'
• '[' -x /usr/bin/tput ']'
• '[' -x /usr/bin/expr ']'
• /usr/bin/tput hpa 60
• /usr/bin/tput setaf 1
• '[' -z ']'
• FANCYTTY=1
• case "$FANCYTTY" in
• true
• /usr/bin/tput xenl
  ++ /usr/bin/tput cols
• COLS=80
• '[' 80 ']'
• '[' 80 -gt 6 ']'
  ++ /usr/bin/expr 80 - 7
• COL=73
• log_use_plymouth
• '[' n = y ']'
• plymouth --ping
• printf ' * Starting ElasticSearch Server '
  
  
  
  • Starting ElasticSearch Server ++ /usr/bin/expr 80 - 1
  
  
• /usr/bin/tput hpa 79
  + printf ' '
  ++ pidofproc -p /var/run/elasticsearch.pid elasticsearch
  ++ local pidfile line i pids= status specified pid
  ++ pidfile=
  ++ specified=
  ++ OPTIND=1
  ++ getopts p: opt
  ++ case "$opt" in
  ++ pidfile=/var/run/elasticsearch.pid
  ++ specified=1
  ++ getopts p: opt
  ++ shift 2
  ++ base=elasticsearch
  ++ '[' '!' 1 ']'
  ++ '[' -n /var/run/elasticsearch.pid -a -r /var/run/elasticsearch.pid ']'
  ++ read pid
  ++ '[' -n 16987 ']'
  +++ kill -0 16987
  ++ ps 16987
  ++ return 1
• pid=
• '[' -n '' ']'
• mkdir -p /var/log/elasticsearch /var/lib/elasticsearch /tmp/elasticsearch
• chown elasticsearch:elasticsearch /var/log/elasticsearch /var/lib/elasticsearch /tmp/elasticsearch
• touch /var/run/elasticsearch.pid
• chown elasticsearch:elasticsearch /var/run/elasticsearch.pid
• '[' -n 65535 ']'
• ulimit -n 65535
• '[' -n '' ']'
• '[' -n 65535 ']'
• sysctl -q -w vm.max_map_count=65535
  error: permission denied on key 'vm.max_map_count'
• start-stop-daemon --start -b --user elasticsearch -c elasticsearch --pidfile /var/run/elasticsearch.pid --exec /usr/share/elasticsearch/bin/elasticsearch -- -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch
• log_end_msg 0
• '[' -z 0 ']'
• '[' 73 ']'
• '[' -x /usr/bin/tput ']'
• log_use_plymouth
• '[' n = y ']'
• plymouth --ping
• printf '\r'
• /usr/bin/tput hpa 73
  + '[' 0 -eq 0 ']'
• echo '[ OK ]'
  [ OK ]
• return 0
• exit 0

[spinscale]

can you run ps p $(cat /var/run/elasticsearch.pid) - this actually looks as if elasticsearch was started...

[dmikhaylov]

Returns just header, i've tried service elasticsearch status and it says * elasticsearch is not running.

[spinscale]

Can you make sure elasticsearch does not run, and try this on the commandline (as root!):

touch /var/run/elasticsearch.pid
chown elasticsearch:elasticsearch /var/run/elasticsearch.pid
start-stop-daemon -v --start --user elasticsearch -c elasticsearch --pidfile /var/run/elasticsearch.pid --exec /usr/share/elasticsearch/bin/elasticsearch -- -p /var/run/elasticsearch.pid -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch

and paste the output here?

[dmikhaylov]

{0.90.8}: Initialization Failed ...

• ElasticSearchIllegalStateException[Failed to obtain node lock, is the following location writable?: [/var/data/elasticsearch/hostname]]
  IOException[failed to obtain lock on /var/data/elasticsearch/hostname/nodes/49]
  IOException[Cannot create directory: /var/data/elasticsearch/hostname/nodes/49]
  Turns out I just forgot to create /var/data directory. Now everything works. Thank you, @spinscale.

[spinscale]

@empirik One last question: Did this message also occur in the log file at /var/log/elasticsearch or was it just printed to stdout? Would be awesome if you could check it!

[dmikhaylov]

@spinscale I don't see this message in logs.

[derEremit]

Just for the record. Getting same error on startup in openvz, but elasticsearch runs without problems for now. A bit irritating as normaly a service start error means the service won't run. As opposed to a warning.
I expect without correct vm.max_map_count I should expect "only" performace problems?

[spinscale]

I dont know openvz enough, but not setting this setting can also result in lucene exceptions and indexing problems, it is not only about performance here - I dont how openvz is handling this. Can you get us any insight here @derEremit?

[13h3r]

Got the same on openvz

[kritik]

I see the same message on big index but after allocating more memmory elasticsearch starts at least :) Using OpenVZ too and elasticsearch 1.1.1.

[idanov]

Got the same on OpenVZ and elasticsearch 1.2.0.

[r0mdau]

Got the same on OpenVZ and elasticsearch 1.2.0
Template : debian-7.0-x86_64.tar.gz
Host Server : Proxmox 3.1-21

[derEremit]

Basically openvz does not allow modifications of kernel parameters as these would affect the host and every other guest machine.
I think that setting should be removed from init scripts and put in a README.
The init script could check for the correct setting and output a warning but not set these kernel parameters itself!

[joke2k]

:+1:

[scamianbas]

Hi all,
kinda newbie here.
I'm using Proxmox, Ubuntu OpenVZ container, Elasticsearch 1.2.1.
I had the same service starting issues and I fixed that by removing the "-d" option in DAEMON_OPTS (a directory path shoud have been provided here)
Also I changed the line
if [ -n "$MAX_MAP_COUNT" ]
with
if [ -n "$MAX_MAP_COUNT" ] && [ ! -f /proc/sys/vm/max_map_count ]
to address the unwanted sysctl issue.
Hope it helps.

[adeleglise]

Hi all,

Thanks to scamianbas for his tips, I tested this on proxmox 3.2-4, debian7-x64 template, works like a charm !

Thanks to the elasticsearch guys too for this great piece of software ;)

[andykillen]

FYI: fresh install of Unbuntu 12.4 with elastic search 1.3.1 over the top, following this Gist (easy way) for install https://gist.github.com/wingdspur/2026107

I get the error

• Starting Elasticsearch Server
  error: permission denied on key 'vm.max_map_count'

when I do a ps check, I get

ps p $(cat /var/run/elasticsearch.pid)
PID TTY STAT TIME COMMAND
8246 ? Sl 0:08 /usr/lib/jvm/java-7-openjdk-i386//bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX

and finally when doing "service elasticsearch status"

it says it is running.

So the error has a slightly different prefix (error: and not sysctl:) but as with other information above it is not stopping the server. If needed I have the screen output with set -x in the elasticsearch config file.

[grepwood]

I'm having the same issue on CentOS 6.5 with elasticsearch 1.3.2 in OpenVZ.
ps p $(cat /var/run/elasticsearch/elasticsearch.pid)
PID TTY STAT TIME COMMAND
1132 ? Sl 0:05 /usr/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Delastics
Just like andykillen, the service is running after all.

[deric]

Same issue with LXC and Ubuntu 14.04

sysctl: permission denied on key 'vm.max_map_count'

though elasticsearch is running, is it a huge problem for production environment?

[karmux]

I also get this error on VPS running Ubuntu 14.04 when I (re)start Elasticsearch but is running at least.

sysctl: permission denied on key 'vm.max_map_count'

[sts]

Can someone please merge this? This works! :+1: :shipit:

if [ -n "$MAX_MAP_COUNT" ]
with
if [ -n "$MAX_MAP_COUNT" ] && [ ! -f /proc/sys/vm/max_map_count ]

[spinscale]

@sts, shouldnt the check be vice versa and only done if the file exists and thus without the negation? I am confused or maybe misunderstanding the intent

[alexgarel]

@sts to avoid the warning you can also edit /etc/default/elasticsearch to set MAX_MAP_COUNT with an empty value, no need to fix the startup script.

[grepwood]

@alexgarel I think it'd be very nice of ES to construct /etc/default/elasticsearch that has MAX_MAP_COUNT set to a null value when an offending environment like OpenVZ is detected as the host of ES, or otherwise, whatever it is meant to be set to.

[marfago]

I'm still experiencing the problem with es 1.4.2. In my openvz guest (vps) the proc file already exists:

cat /proc/sys/vm/max_map_count
65530

but I get the same error

sysctl: permission denied on key 'vm.max_map_count'

[settermjd]

FWIW - I'm on Centos 6.5 and also get this error. As suggested, I did a quick check and the daemon does appear to be running; though I've not attempted to send any data to it yet. But it does seem to be running.

[nvucinic]

Hi I have the same problem, (t's a setup of OpenVZ containers runing CentOS and Elasticsearch-1.3.9-1, 5 nodes), after starting elasticsearch i got "error: permission denied on key 'vm.max_map_count'" error but elasticsearch is working, well for some time. At random times nodes just "die", one by one, with status "elasticsearch dead but pid file exists". There is nothing helpfull in syslog//elastic node log.
It just happnes and they all die one by one.

[Yashin-s]

hi nvucinic, have you been able to find a solution ? I am getting exactly the same issue. I cannot set the max_map_count, and I get an error during elasticsearch start. It starts well but then dies and I also get the error message "elasticsearch dead but pid file exists"

[chuntley]

Exact same issue has nvucinic. I just implemented the empty ES_MAP_SIZE fix, and updated to ElasticSearch 1.5. I'll let you know if it happens again.

[dwerder]

Hi @Yashin-s and @chuntley ,
we have the same problem with Nodes just randomly dieing.
There is also a discussion in the OpenVZ Bugzilla.

Java (Elasticsearch 1.4.x) reproducibly crashes with random segmentation faults
https://bugzilla.openvz.org/show_bug.cgi?id=3187

[mrqwer88]

Not only 1.4.x - I saw this problem whith 1.5 too.

[dwerder]

The problem affects nearly all versions of elasticsearch. And it looks like problems with OpenVZ ploop and Kernels greater then 042stab10x . A workaround is to downgrade to version 042stab09x.
Or to use simfs as filesystem.

[mrqwer88]

Whith simfs it not reproduse on new 042stab10x kernels?
Thanks, we try only whith ploop. And try it for test in chroot from node(whith mount ploop) - in chroot problem not reproduce.

[radev]

Yes, our instances are stable with recent kernel and simfs. With ploop they are crashing on every combination of JVM 1.7.0.x, 1.8.0.x and ES 1.3.x, 1.4.x

[dwerder]

same problem is discussed here: https://github.com/elastic/elasticsearch/issues/9582

[TomArrow]

has this problem been solved? i am stuck with a vserver from german strato, running debian 7, getting the 'vm.max_map_count' error. i doubt i can change the file system.

any idea what the problem is?

solution or path to solution would be much appreciated.

[spinscale]

please read #9582, that should help you, with newer elasticsearch releases you can set bootstrap.sigar: false. 2.0 wont have this problem at all due to sigar being removed

[TomArrow]

added bootstrap.sigar: false to /etc/elasticsearch/elasticsearch.yml.
unfortunately, that does not help. here is my long output. is this the same issue or should i open up / follow another request?

+ PATH=/bin:/usr/bin:/sbin:/usr/sbin

• NAME=elasticsearch
• DESC=Elasticsearch Server
• DEFAULT=/etc/default/elasticsearch
• id -u
• [ 0 -ne 0 ]
• . /lib/lsb/init-functions
• run-parts --lsbsysinit --list /lib/lsb/init-functions.d
• [ -r /lib/lsb/init-functions.d/20-left-info-blocks ]
• . /lib/lsb/init-functions.d/20-left-info-blocks
• FANCYTTY=
• [ -e /etc/lsb-base-logging.sh ]
• true
• [ -r /etc/default/rcS ]
• . /etc/default/rcS
• ES_USER=elasticsearch
• ES_GROUP=elasticsearch
• JDK_DIRS=/usr/lib/jvm/java-8-oracle/ /usr/lib/jvm/j2sdk1.8-oracle/ /usr/lib/jvm/jdk-7-oracle-x64 /usr/lib/jvm/java-7-oracle /usr/lib/jvm/j2sdk1.7-oracle/ /usr/lib/jvm/java-7-openjdk /usr/lib/jvm/java-7-openjdk-amd64/ /usr/lib/jvm/java-7-openjdk-armhf /usr/lib/jvm/java-7-openjdk-i386/ >/usr/lib/jvm/default-java
• [ -r /usr/lib/jvm/java-8-oracle//bin/java -a -z ]
• [ -r /usr/lib/jvm/j2sdk1.8-oracle//bin/java -a -z ]
• [ -r /usr/lib/jvm/jdk-7-oracle-x64/bin/java -a -z ]
• [ -r /usr/lib/jvm/java-7-oracle/bin/java -a -z ]
• [ -r /usr/lib/jvm/j2sdk1.7-oracle//bin/java -a -z ]
• [ -r /usr/lib/jvm/java-7-openjdk/bin/java -a -z ]
• [ -r /usr/lib/jvm/java-7-openjdk-amd64//bin/java -a -z ]
• [ -r /usr/lib/jvm/java-7-openjdk-armhf/bin/java -a -z ]
• [ -r /usr/lib/jvm/java-7-openjdk-i386//bin/java -a -z ]
• [ -r /usr/lib/jvm/default-java/bin/java -a -z ]
• JAVA_HOME=/usr/lib/jvm/default-java
• export JAVA_HOME
• ES_HOME=/usr/share/elasticsearch
• MAX_OPEN_FILES=65535
• LOG_DIR=/var/log/elasticsearch
• DATA_DIR=/var/lib/elasticsearch
• WORK_DIR=/tmp/elasticsearch
• CONF_DIR=/etc/elasticsearch
• CONF_FILE=/etc/elasticsearch/elasticsearch.yml
• MAX_MAP_COUNT=262144
• PID_DIR=/var/run/elasticsearch
• [ -f /etc/default/elasticsearch ]
• . /etc/default/elasticsearch
• PID_FILE=/var/run/elasticsearch/elasticsearch.pid
• DAEMON=/usr/share/elasticsearch/bin/elasticsearch
• DAEMON_OPTS=-d -p /var/run/elasticsearch/elasticsearch.pid --default.config=/etc/elasticsearch/elasticsearch.yml --default.path.home=/usr/share/elasticsearch --default.path.logs=/var/log/elasticsearch --default.path.data=/var/lib/elasticsearch --default.path.work=/tmp/elasticsearch --default.path.conf=/etc/elasticsearch
• export ES_HEAP_SIZE
• export ES_HEAP_NEWSIZE
• export ES_DIRECT_SIZE
• export ES_JAVA_OPTS
• test -x /usr/share/elasticsearch/bin/elasticsearch
• checkJava
• [ -x /usr/lib/jvm/default-java/bin/java ]
• JAVA=/usr/lib/jvm/default-java/bin/java
• [ ! -x /usr/lib/jvm/default-java/bin/java ]
• [ -n -a -z ]
• log_daemon_msg Starting Elasticsearch Server
• [ -z Starting Elasticsearch Server ]
• log_daemon_msg_pre Starting Elasticsearch Server
• log_use_fancy_output
• TPUT=/usr/bin/tput
• EXPR=/usr/bin/expr
• [ -t 1 ]
• [ xxterm-256color != x ]
• [ xxterm-256color != xdumb ]
• [ -x /usr/bin/tput ]
• [ -x /usr/bin/expr ]
• /usr/bin/tput hpa 60
• /usr/bin/tput setaf 1
• [ -z ]
• FANCYTTY=1
• true
• /bin/echo -n [....]
  [....] + [ -z ]
• /bin/echo -n Starting Elasticsearch Server:
  Starting Elasticsearch Server:+ return
• pidofproc -p /var/run/elasticsearch/elasticsearch.pid elasticsearch
• local pidfile base status specified pid OPTIND
• pidfile=
• specified=
• OPTIND=1
• getopts p: opt
• pidfile=/var/run/elasticsearch/elasticsearch.pid
• specified=specified
• getopts p: opt
• shift 2
• base=elasticsearch
• [ ! specified ]
• [ -n /var/run/elasticsearch/elasticsearch.pid ]
• [ -e /var/run/elasticsearch/elasticsearch.pid ]
• [ -r /var/run/elasticsearch/elasticsearch.pid ]
• read pid
• [ -n ]
• [ specified ]
• return 3
• pid=
• [ -n ]
• mkdir -p /var/log/elasticsearch /var/lib/elasticsearch /tmp/elasticsearch
• chown elasticsearch:elasticsearch /var/log/elasticsearch /var/lib/elasticsearch /tmp/elasticsearch
• [ -n /var/run/elasticsearch ]
• [ ! -e /var/run/elasticsearch ]
• [ -n /var/run/elasticsearch/elasticsearch.pid ]
• [ ! -e /var/run/elasticsearch/elasticsearch.pid ]
• [ -n 65535 ]
• ulimit -n 65535
• [ -n ]
• [ -n 262144 -a -f /proc/sys/vm/max_map_count ]
• sysctl -q -w vm.max_map_count=262144
  sysctl: permission denied on key 'vm.max_map_count'
• start-stop-daemon --start -b --user elasticsearch -c elasticsearch --pidfile /var/run/elasticsearch/elasticsearch.pid --exec /usr/share/elasticsearch/bin/elasticsearch -- -d -p /var/run/elasticsearch/elasticsearch.pid --default.config=/etc/elasticsearch/elasticsearch.yml --default.path.home=/usr/share/elasticsearch --default.path.logs=/var/log/elasticsearch --default.path.data=/var/lib/elasticsearch --default.path.work=/tmp/elasticsearch --default.path.conf=/etc/elasticsearch
• return=0
• [ 0 -eq 0 ]
• i=0
• timeout=10
• sleep 1
• i=1
• [ 1 -gt 10 ]
• sleep 1
• i=2
• [ 2 -gt 10 ]
• sleep 1
• i=3
• [ 3 -gt 10 ]
• sleep 1
• i=4
• [ 4 -gt 10 ]
• sleep 1
• i=5
• [ 5 -gt 10 ]
• sleep 1
• i=6
• [ 6 -gt 10 ]
• sleep 1
• i=7
• [ 7 -gt 10 ]
• sleep 1
• i=8
• [ 8 -gt 10 ]
• sleep 1
• i=9
• [ 9 -gt 10 ]
• sleep 1
• i=10
• [ 10 -gt 10 ]
• sleep 1
• i=11
• [ 11 -gt 10 ]
• log_end_msg 1
• [ -z 1 ]
• local retval
• retval=1
• log_end_msg_pre 1
• log_use_fancy_output
• TPUT=/usr/bin/tput
• EXPR=/usr/bin/expr
• [ -t 1 ]
• [ xxterm-256color != x ]
• [ xxterm-256color != xdumb ]
• [ -x /usr/bin/tput ]
• [ -x /usr/bin/expr ]
• /usr/bin/tput hpa 60
• /usr/bin/tput setaf 1
• [ -z 1 ]
• true
• true
• /usr/bin/tput setaf 1
• RED=
• /usr/bin/tput setaf 2
• GREEN=
• /usr/bin/tput setaf 3
• YELLOW=
• /usr/bin/tput op
• NORMAL=
• /usr/bin/tput civis
• /usr/bin/tput sc
• /usr/bin/tput hpa 0
• [ 1 -eq 0 ]
• [ 1 -eq 255 ]
• /bin/echo -ne [FAIL
  [FAIL+ /usr/bin/tput rc
• /usr/bin/tput cnorm
• log_use_fancy_output
• TPUT=/usr/bin/tput
• EXPR=/usr/bin/expr
• [ -t 1 ]
• [ xxterm-256color != x ]
• [ xxterm-256color != xdumb ]
• [ -x /usr/bin/tput ]
• [ -x /usr/bin/expr ]
• /usr/bin/tput hpa 60
• /usr/bin/tput setaf 1
• [ -z 1 ]
• true
• true
• /usr/bin/tput setaf 1
• RED=
• /usr/bin/tput setaf 3
• YELLOW=
• /usr/bin/tput op
• NORMAL=
• [ 1 -eq 0 ]
• [ 1 -eq 255 ]
• /bin/echo -e failed!
  failed!
• log_end_msg_post 1
• :
• return 1
• exit 1

[Yashin-s]

Hello, just for your information, I was unable to solve the problem on the original server. Under the recommendation of someone, I migrated to another server. The current server has much less RAM and is running on Ubuntu. I started the elasticsearch service several months ago, and it's still doing good...

[TomArrow]

thanks for the note, Yashin-s. unfortunately that is not an option at this moment. i also do not feel like randomly trying out various servers without ever knowing what the problem was in the first place.

[TomArrow]

i tried to start the daemon proces directly, this is what came out:

root@h922069:/var/log/elasticsearch# /usr/share/elasticsearch/bin/elasticsearch -d -p /var/run/elasticsearch/elasticsearch.pid --default.config=/etc/elasticsearch/elasticsearch.yml --default.path.home=/usr/share/elasticsearch --default.path.logs=/var/log/elasticsearch --default.path.data=/var/lib/elasticsearch --default.path.work=/tmp/elasticsearch --default.path.conf=/etc/elasticsearchroot@h922069:/var/log/elasticsearch# Exception in thread "main" java.lang.UnsupportedClassVersionError: org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor version 51.0
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.elasticsearch.bootstrap.Elasticsearch. Program will exit.

[jasontedor]

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor version 51.0

This means that you're trying to run classes targeting JDK 7 on a JVM that doesn't support JDK 7 class files.

[TomArrow]

perfect, solved. thank you.

[ccfontes]

Also getting:

* Starting Elasticsearch Server                                                                                                                                                  
sysctl: permission denied on key 'vm.max_map_count'

on OpenVZ, but already running Elastic 1.7.1. Shouldn't this issue be solved already?

[joshuajonah]

This is insane. I guess I'm going to change over to a KVM VPS.

[TomArrow]

You may want to try to execute the command from the init script. To me, it revealed that it really was all just a problem with the wrong JDK version. The vm.max_map_count error was basically just a demonic distraction from the true problem.

[jdkcn]

the Elasticsearch 2.0 has the same problem.

but the server started up.

root@es0:~# /etc/init.d/elasticsearch start
[....] Starting Elasticsearch Server:sysctl: permission denied on key 'vm.max_map_count'
. ok 

[spinscale]

This has nothing to do with sigar. It is just the init script trying to access a file in the proc filesystem... And falling due to not being able to access it

[grepwood]

Since this is exclusively a problem with OpenVZ, shouldn't Elasticsearch not attempt to access that file if OpenVZ is detected?

[alexgarel]

This is not only true for OpenVZ, this is also true for LXC or Docker, every time you have a read only access to /sys or /proc.
But in some way, this is a warning, not a show stopper.
What we need here is pedagogy. If the script can't write on /sys, just explain this could degrade performance. In the case of a container, the user knows if he sets the /sys parameter to its right value on the host.
If some of the commiter is reading, I suggest to change the script to catch sysctl error and emit a warning message, telling which parameter it was not able to change, to which value.

[clintongormley]

the startup script now has this:

if [ -n "$MAX_MAP_COUNT" -a -f /proc/sys/vm/max_map_count ]; then

should we change that to check if /proc/sys/vm/max_map_count is writable instead of just -f?

[alexgarel]

yes this checking its writable is a good idea. Maybe emitting a warning if it exists but is not writable.

[dagatsoin]

Update on this? Could this still cause problem with Lucen? I have this error too:

[root@vps1569 init.d]$ service elasticsearch restart
Stopping elasticsearch:                                    [  OK  ]
error: permission denied on key 'vm.max_map_count'
Starting elasticsearch: log4j:WARN No appenders could be found for logger (common).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
                                                           [  OK  ]

[jogaco]

Upgrading from 1.7 to 2.1 I got the same error:

 * Starting Elasticsearch Server
sysctl: permission denied on key 'vm.max_map_count'

Using Ubuntu 14.04.1 LTS,
The service is working luckily. Consequences of this error, anyone?

[grepwood]

@jogaco Absolutely no practical consequences other than having this message in the logs

[karmi]

@spinscale, this issue is still linked from the TravisCI docs, maybe we can look into it?

[spinscale]

@karmi it is trivial to add another check if the proc file is writable and silence out the message, I just tested that locally on non OpenVZ and everything works as expected.

What I dont know is, if just adding another check, that checks if the file in the proc file system is writable, actually changes anything for OpenVZ users. Maybe one OpenVZ user can shed some light on this? If we check for being able to write /proc/sys/vm/max_map_count instead that it is being a file (what we do now), would the sysctl call not be done on OpenVZ?

[Altareq]

same problem on ES 2.3.0 Ubuntu 14.04

[consistime]

there is no solutions for this problem? fresh install of ubuntu server will be fix them?

[consistime]

this is caused on ubuntu 14 on proc/sys/vm permitted

[consistime]

now i know the problem is on server host is use openvz or any virtualization that doesnt support editing

[consistime]

then it is posible on how to make elasticsearch and java not without editing this request

[consistime]

omg help mee

[grepwood]

@rectoid The bug isn't causing any actual issues. Your Elasticsearch will work fine. It's just annoying to see in the logs.

I can't believe it's been over 2 years.

[adamgontarz]

on /etc/init.d/elasticsearch init script comment lines:

#       if [ -n "$MAX_MAP_COUNT" ]; then
#               sysctl -q -w vm.max_map_count=$MAX_MAP_COUNT
#       fi

[talaikis]

Same on 2.4.1, Ubuntu 14.04.

[arypurnomoz]

steps that i took to make es 5 run in an environment when i can't change vm.max_map_count

1. make es bind to localhost, (when binding to localhost es will start development mode, which wont do the check)
2. use nginx to forward http traffic to es, or you can use iptables to route the tcp request from outside network to localhost

but remember, es will run on DEVELOPMENT mode

here is my workaround for making it run inside docker container
https://gist.github.com/arypurnomoz/8d24155f024f8d4b6bee6b053c6e47d9

[florian-asche]

You should not have to. My is running fine, except that error/warning message...

[clintongormley]

As of 5.0, Elasticsearch will not start in production mode if vm.max_map_count is not high enough. Silencing the warning when we are not able to apply the MAX_MAP_COUNT setting on openvz will just make debugging the issue harder, as it will not be obvious why the setting is not being applied. Instead, if you are running on a system where you cannot set vm.max_map_count, but it is set to be high enough for Elasticsearch's bootstrap checks, then you can silence the warning by removing the MAX_MAP_COUNT setting. If the value on your system is NOT high enough, then your cluster is going to crash and burn at some stage and you will lose data. Instead of trying to work around it with hacks like https://github.com/elastic/elasticsearch/issues/4978#issuecomment-257076124, you should either speak to your sysadmin to configure vm.max_map_count correctly, or move to a platform where you can set it.

[jasontedor]

@arypurnomoz Your workaround is overly complicated if all you want to do is expose a node via HTTP on an interface other than localhost. You can set http.host independently of transport.host (network.host sets both); the bootstrap checks are only enforced if a node is able to form a cluster with other instances not on localhost.

[vanthome]

you should provide a flag to prevent this check or better do not conduct it in development mode. this currently prevents me from running tests on travis.ci.

[JESWINKNINAN]

I have this issue before and solved by running my docker container in privileged mode, The actual reason behind this error is quite simple, Elasticsearch needs to change the "vm.max_map_count" to run, so changing via sysctl by Elasticseach itself and action is denied due to lack of permission. The issue will occur even the it is also a root user(Docker).

Simply run the container in privileged mode and this will fix the similar type of issues.

root@Zion:/home/jeswin/dockerproject# docker run -i -t --privileged ecc4953c8296 "/bin/bash"

[x-itec]

This problem really drives me mad with docker !!!!!!!! --privileged does not work anymore, nothing works anymore!!!! memory is set, but not checked correctly by ES5.x so it tries to write for no reason! result: ERROR

[davron112]

My version: elasticsearch: 5.6.4 (but not working 6.2.3)
I added elasticsearch.yml : network.bind_host: "0.0.0.0"
To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf.
vm.max_map_count=262144
To verify after rebooting, runsysctl vm.max_map_count.

docker-compose build elasticsearch
docker-compose up -d elasticsearch
docker-compose ps (double, because first worked, but second not worked)

result:

laradock_elasticsearch /bin/bash bin/es- Up 0.0.0.0:9200->9200/tcp