Elasticsearch: [DOC] Docker recommendation should be moved to Docker getting started section

[matt-isett]

Elasticsearch version (bin/elasticsearch --version):
Current

OS version (uname -a if on a Unix-like system):
Docker

Description of the problem including expected versus actual behavior:
A Docker recommendation related to MEMLOCK was mentioned in a low level issue comment. This setting can have improved performance for running docker and should be added to the Notes section on the Install Elasticsearch with Docker.

@dliappis

The general recommendation would be that Elasticsearch in Docker would also have the same defaults as recommended for running Elasticsearch with systemd so setting LimitMEMLOCK=infinity for the Docker systemd unit as you yourself have suggested in https://github.com/elastic/elasticsearch-docker/issues/152#issuecomment-372903395 makes sense.

Do note however, that while this recommendation makes sense if the only thing Docker is running is Elasticsearch, it may not make sense in a mixed environment where other containers are running that may take advantage of infinity to do crazy things.

[jasontedor]

@matt-isett I am confused what is being asked for here that is not already in the docs? From the notes section that you link to:

. Swapping needs to be disabled for performance and node stability. This can be
achieved through any of the methods mentioned in the
<<setup-configuration-memory,{es} docs>>. If you opt for the
`bootstrap.memory_lock: true` approach, apart from defining it through any of
the <<docker-configuration-methods,configuration methods>>, you will
additionally need the `memlock: true` ulimit, either defined in the
https://docs.docker.com/engine/reference/commandline/dockerd/#default-ulimits[Docker Daemon]
or specifically set for the container. This is demonstrated above in the
<<docker-prod-cluster-composefile,docker-compose.yml>>. If using `docker run`:
+
--
  -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1
--

This mentions memlock, this links to the disable swapping section of the Elasticsearch docs, and this includes Docker-specific instructions. What specifically do you think needs to be different here?

[dliappis]

@jasontedor @matt-isett is asking a different thing here. Our existing documentation describes how to set memlock to unlimited inside the container, however, this will inherit the defaults that the Docker (systemd) service got started with. The docker systemd unit file does not set LimitMEMLOCK, so it inherits whatever is the systemd default (or things like [email protected]) which _may_ even differ per distro:

$ lsb_release -d && systemctl show | grep -i memlock
Description:    Ubuntu 16.04.3 LTS
DefaultLimitMEMLOCK=65536
DefaultLimitMEMLOCKSoft=65536

$ lsb_release -d && systemctl show | grep -i memlock
Description:    CentOS Linux release 7.4.1708 (Core) 
DefaultLimitMEMLOCK=65536

lsb_release -d && systemctl show | grep -i memlock
Description:    openSUSE Leap 42.3
DefaultLimitMEMLOCK=65536

lsb_release -d && systemctl show | grep -i memlock
Description:    SUSE Linux Enterprise Server 12 SP3
DefaultLimitMEMLOCK=65536

$ lsb_release -d && systemctl show | grep -i memlock
Description:    Fedora release 27 (Twenty Seven)
DefaultLimitMEMLOCK=65536
DefaultLimitMEMLOCKSoft=65536

@jarpy explains how he was able to override this in Ubuntu (Xenial I think) by editing the Docker unit file: https://github.com/elastic/elasticsearch-docker/issues/152#issuecomment-372903395

[jasontedor]

Thanks for clarifying @dliappis. A few comments:

• we should not be recommending editing unit files, but instead adding an unit override (e.g., in /etc/systemd/system/docker.service.d/).
• as the disable swapping that I link to mention, memory locking is not our preferred option, so we should be careful to not over-highlight this particular situation; as the docs that I link to note:

The preferred option is to completely disable swap. If this is not an option, whether or not to prefer minimizing swappiness versus memory locking is dependent on your environment.

This is consistent with your comment:

Do note however, that while this recommendation makes sense if the only thing Docker is running is Elasticsearch, it may not make sense in a mixed environment where other containers are running that may take advantage of infinity to do crazy things.

[elasticmachine]

Pinging @elastic/es-core-infra

[dliappis]

Agreed @jasontedor . My personal sentiment is that setting LimitMEMLOCK=infinity as the default in Docker sounds like opening the floodgates (not to mention that changing this in k8s maybe an more complicated thing). I will however defer to @jarpy 's opinion on the matter.

One minor comment:

we should not be recommending editing unit files, but instead adding an unit override (e.g., in /etc/systemd/system/docker.service.d/).

Using systemctl edit will actually automatically create the unit override (e.g. as in the command in https://github.com/elastic/elasticsearch-docker/issues/152#issuecomment-372903395).

[jasontedor]

My personal sentiment is that setting LimitMEMLOCK=infinity as the default in Docker sounds like opening the floodgates (not to mention that changing this in k8s maybe an more complicated thing).

Agree. And if we document them all it ends up looking like we are highlighting this as important when it's not even the approach we recommend.

Using systemctl edit will actually automatically create the unit override (e.g. as in the command in https://github.com/elastic/elasticsearch-docker/issues/152#issuecomment-372903395).

Right; I'm trying to avoid us sowing confusion from:

@jarpy explains how he was able to override this in Ubuntu (Xenial I think) by editing the Docker unit file: elastic/elasticsearch-docker#152 (comment)

The phrase that I think can lead to confusion here is "the Docker unit file" (emphasis added).

[dliappis]

Ah I see how what I said earlier could cause confusion. Apologies :)

[jasontedor]

No additional feedback, closing.