Elasticsearch: Elasticsearch init scripts set max open files to 65535, but expects 65536

[joehillen]

Elasticsearch version: 5.0.0-alpha1 78ab6c5b7ff82da7f7d3c059b4a43d80bad188fb

JVM version: OpenJDK Runtime Environment (build 1.8.0_72-internal-b15)

OS version: Ubuntu 14.04

Description of the problem including expected versus actual behavior:

Elasticsearch expects max open file descriptors to be set to 65536, but init scripts set it to 65535.

Steps to reproduce:

1. Launch elasticsearch using /etc/init.d/elasticsearch

Provide logs (if relevant):

[2016-03-30 23:12:05,787][ERROR][bootstrap                ] Exception
java.lang.RuntimeException: max file descriptors [65535] for elasticsearch process likely too low, increase to at least [65536]
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:79)
    at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:60)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:188)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:264)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
    at org.elasticsearch.cli.Command.main(Command.java:53)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)

[ajaybhatnagar]

I had the setting in /etc/security/limits.conf with alpha3 release and starting ES still throws the error
elasticsearch - nofile 65536

and init file contained following lines:

Run Elasticsearch as this user ID and group ID

ES_USER=elasticsearch
ES_GROUP=elasticsearch

Maximum number of open files

MAX_OPEN_FILES=65536

Error logs:

[2016-06-02 20:05:44,637][ERROR][bootstrap ] [es-tst-m01] Exception
java.lang.RuntimeException: bootstrap checks failed
max file descriptors [65535] for elasticsearch process likely too low, increase to at least [65536]
at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:125)
at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:85)

[jasontedor]

@ajaybhatnagar How are you starting Elasticsearch? What are the limits for the root user? Note that the limits for the root user must be at least as high as the elasticsearch user.

[ajaybhatnagar]

Root and es user settings for nofile:
root - nofile 100000
elasticsearch - nofile 65536

ulimit -n output for ES user:
65536

Starting ES with service elasticsearch start

[jasontedor]

@ajaybhatnagar I'm going to make some assumptions here, please correct if any of them are wrong.

Starting ES with service elasticsearch start

I'm assuming that you deliberately left sudo off here. From this, I'm assuming that you are thus logged in as the root user. I'm further assuming that you made the above changes to /etc/security/limits.conf and did not logout and log back in.

If these assumptions are correct, the solution to the problem is for you to just log out and log back in.

[ajaybhatnagar]

Rebooted the node and still the same error.

Init script has the line below, yet startup is finding max file descriptors below the configured value. Hardcoded value picked from somewhere else or overwritten ?

Maximum number of open files

MAX_OPEN_FILES=65536
Check on the node:
root@es-tst-m01:/es1/logs/elasticsearch# ulimit -n
100000
root@es-tst-m01:/es1/logs/elasticsearch# su - elasticsearch
elasticsearch@es-tst-m01:~$ ulimit -n
65536
[2016-06-03 13:35:51,324][ERROR][bootstrap ] [es-tst-m01] Exception
java.lang.RuntimeException: bootstrap checks failed
max file descriptors [65535] for elasticsearch process likely too low, increase to at least [65536]

[jasontedor]

You have a configuration error somewhere, it's just a matter of finding where. Can you check cat /proc/sys/fs/file-max? If it's too low, execute sysctl -w fs.file-max=65536 or some other higher value to raise it. Consider putting this value in /etc/sysctl.conf so that it persists across system reboots. You will have to logout and log back in if you changed this value.

[dajoen]

I ran into the same problem just now. And had to change the nofiles in /usr/lib/systemd/system/elasticsearch.service. After changing this to the same value as fs.file-max the message disappeared.

[ajaybhatnagar]

Caused by setting in /etc/deafult/elasticsearch. Closed.
thx

[X-Mars]

hi
i have the same error
but i don't have /etc/deafult/elasticsearch
can u help me
thx

[jasontedor]

@X-Mars This issue is closed, and was specific to how the defaults that Elasticsearch shipped with were inconsistent with a warning log message that it would produce. Please open a new post on the Elastic Discourse forum with details about your setup and debugging steps that you have already gone through.

[and1990]

I want to run elasticsearch in docker, but it shows the error when i am running with the command 'docker-compose up'. what should i do with this?

My docker-compose.yml:

version: '2.0'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.1.0
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.1.0
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local

networks:
  esnet:

[jasontedor]

@and1990 Please look at the content in the comment immediately above yours.

[and1990]

@jasontedor Thanks. I have solved the problem. Those guys who use elasticsearch in docker have a look at this document https://blog.docker.com/2015/04/docker-release-1-6, the Ulimits section.

[micookie]

and reboot system