Microk8s: Elasticsearch stable Helm charts max file descriptors

[novakov-alexey]

Stable Helm Elasticsearch charts https://github.com/helm/charts/tree/master/stable/elasticsearch
fail on Elasticsearch nodes startup:

[2018-12-18T22:11:25,113][INFO ][o.e.b.BootstrapChecks    ] [es-elasticsearch-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

microk8s.inspect output:

Inspecting services
  Service snap.microk8s.daemon-docker is running
  Service snap.microk8s.daemon-apiserver is running
  Service snap.microk8s.daemon-proxy is running
  Service snap.microk8s.daemon-kubelet is running
  Service snap.microk8s.daemon-scheduler is running
  Service snap.microk8s.daemon-controller-manager is running
  Service snap.microk8s.daemon-etcd is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system info
  Copy network configuration to the final report tarball
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Inspect kubernetes cluster

 WARNING:  IPtables FORWARD policy is DROP. Consider enabling traffic forwarding with: sudo iptables -P FORWARD ACCEPT
Building the report tarball
  Report tarball is at /var/snap/microk8s/340/inspection-report-20181218_231630.tar.gz

On host:

cat /etc/sysctl.conf
vm.max_map_count = 262144
fs.file-max = 1634038

[ktsakalozos]

Hi @novakov-alexey

Are you able to set an environment variable discovery.type to single-node as suggested https://stackoverflow.com/questions/42300463/elasticsearch-5-x-bootstrap-checks-failing and we do in this PR https://github.com/ubuntu/microk8s/pull/217/files#diff-a74758cc33ad9bbd62f6a3d900aa3ee1R95

Thank you for trying out MicroK8s

[novakov-alexey]

Hi @ktsakalozos

Thanks for quick reply.

Setting discovery.type to single-node is more or less workaround. I want to start more than one Elasticsearch node on the Microk8s for testing purposes. When using single-node config, Elasticsearch won't build proper cluster 😞

However, I have tried to change the max file descriptors on the host and using Deployment spec via initContainers. It does't help. 😕

The same Helm charts work for me on Minikube with VM. In VM case, I only set proper host OS settings inside the Minikube VM.

Is there particular issue to propagate fs.file-max to Microk8s runtime?

[ktsakalozos]

@novakov-alexey I cannot give you a good solution for this now. Our friends in the snapcraft forum may know more about this. https://forum.snapcraft.io

[novakov-alexey]

@ktsakalozos ok, I will ask there as well, thank you.

[ktsakalozos]

@novakov-alexey, a friend suggested the following:

{
  "insecure-registries" : ["localhost:32000"],
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 65536,
      "Soft": 65536
     }
  }
}

for /var/snap/microk8s/current/args/docker-daemon.json.

[novakov-alexey]

@ktsakalozos thanks a lot. It seems this helps (I also restarted the microk8s)

[ephill]

@ktsakalozos Any pointers for doing the same thing using microk8s 1.14? Since the move to containerd these settings which were working for us are now being ignored obviously. Having trouble finding an equivalent setting in containerd.

[shenkejie15]

I run into the same problem as ephill. Since microk8s 1.14 no longer has docker-daemon.json.

[kzhang7]

Same problem here. For now I have to rollback to 1.13 to get default-ulimits working again.

[irrgit]

@ktsakalozos Any pointers for doing the same thing using microk8s 1.14? Since the move to containerd these settings which were working for us are now being ignored obviously. Having trouble finding an equivalent setting in containerd.

Same issue here @ephill @shenkejie15

[ktsakalozos]

There is no such config in containerd (confirmed here https://github.com/containerd/containerd/issues/3150). I guess we need to look at the runc arguments?

[ktsakalozos]

Ok here is a workaround that seems to be working (for me at least).

Edit /var/snap/microk8s/current/args/containerd-env and add this line at the end:

ulimit -n 65536

Then restart MicroK8s with microk8s.stop and microk8s.start. Your containerd should now have the limit set to 65536. Let me know it this worked.

[elolivier]

@ktsakalozos it worked for me, thanks a lot!

[shenkejie15]

@ktsakalozos It works, thanks for your help.