Helm-charts: [elasticsearch] Cannot run a single node cluster

[ebuildy]

Chart version:

7.4.0

Kubernetes version:

1.14.0

Kubernetes provider:

on-premise

Helm Version:

2.X

Describe the bug:

As a developper, I want to run a simple elasticsearch cluster, with a single node.

Steps to reproduce:

As indicated here https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html, values.yml:

esConfig:
  elasticsearch.yml: |
    discovery:
      type: single-node
      seed_hosts: ""

Expected behavior:

A single pod should run.

Provide logs and/or server output (if relevant):

This chart is hardcoding the setting cluster.initial_master_nodes which throw the exception

stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: setting [cluster.initial_master_nodes] is not allowed when [discovery.type] is set to [single-node]"

[Crazybus]

One of our tested examples is a single node cluster.

The only configuration you need to add is replicas: 1 and it will work.

[ebuildy]

I run:

helm install --set replicas=1 --set imageTag=7.4.0 --name single elastic/elasticsearch

This output

NAME:   single
LAST DEPLOYED: Thu Oct  3 19:26:11 2019
NAMESPACE: dev-steam
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                    READY  STATUS    RESTARTS  AGE
elasticsearch-master-0  0/1    Init:0/1  0         3s

==> v1/Service
NAME                           TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)            AGE
elasticsearch-master           ClusterIP  10.233.45.179  <none>       9200/TCP,9300/TCP  2s
elasticsearch-master-headless  ClusterIP  None           <none>       9200/TCP,9300/TCP  2s

==> v1/StatefulSet
NAME                  READY  AGE
elasticsearch-master  0/1    2s

==> v1beta1/PodDisruptionBudget
NAME                      MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
elasticsearch-master-pdb  N/A            1                0                    2s


NOTES:
1. Watch all cluster members come up.
  $ kubectl get pods --namespace=dev-steam -l app=elasticsearch-master -w
2. Test cluster health using Helm test.
  $ helm test single

But the elasticsearch node is broken:

{"type": "server", "timestamp": "2019-10-03T17:29:06,466Z", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "no known master node, scheduling a retry" }

"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:214) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]"] }

[Crazybus]

I run:

Running this commands results in a working cluster for me on GKE.

1. Can you provide the output of helm get single.
2. Is there anything special about your on premise cluster that might cause issues? E.g. no persistent storage available?
3. Does deploying a cluster with the defaults work (3 nodes cluster)?

[Crazybus]

Another thing worth checking. Can you show the output of kubectl get pv. If you previously had a 3 node cluster and didn't delete the volumes after them being used the one node cluster will start up expecting to find 2 more nodes.

[ebuildy]

Ho ! You are right ! I am deeply sorry. I didnt notice that helm delete --purge is not removing the PV(c).

Thanks you for the support.

[demisx]

I am deploying this from scratch to AWS k8s 1.14.6 cluster created by kops and my kubectl get pv output is empty (I've deleted all pvc from a previous 3 node install):

helm install --set replicas=1 --set imageTag=7.4.0 --name single elastic/elasticsearch

However, I am still seeing this error in the log. Any ideas are appreciated as I am pretty stomped at this point. I get this error deploying 1 or 3 default ones:

{"type": "server", "timestamp": "2019-10-22T18:37:52,360Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{elasticsearch-master-0}{60-eI8yESdyLkVHLzLpdkg}{8eKBeNsPQRabxtlwfZidTA}{100.96.5.7}{100.96.5.7:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{60-eI8yESdyLkVHLzLpdkg}{8eKBeNsPQRabxtlwfZidTA}{100.96.5.7}{100.96.5.7:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-10-22T18:38:00,347Z", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "no known master node, scheduling a retry" }
{"type": "server", "timestamp": "2019-10-22T18:38:01,347Z", "level": "DEBUG", "component": "o.e.a.a.c.h.TransportClusterHealthAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "timed out while retrying [cluster:monitor/health] after failure (timeout [1s])" }
{"type": "server", "timestamp": "2019-10-22T18:38:01,348Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}", 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:214) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.4.0.jar:7.4.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]"] }

[demisx]

The error above was due to a bug in helm 2.15.0. It worked as expected after downgrading to 2.14.3.

[kfirfer]

Have the same problem, replicas 1 doesnt helped
chart version: 7.7.1
helm: 3

I had manually disable "cluster.initial_master_nodes" env var, in order to success deploy

[psychohamster]

I had manually disable "cluster.initial_master_nodes" env var, in order to success deploy

How did you 'disable' it? I tried just overriding it in the values.yaml but that breaks helm upgrade as in this issue: https://github.com/helm/helm/issues/6002

[ebuildy]

@demisx did you configure ES as a single node cluster ?

esConfig:
  elasticsearch.yml: |
    discovery:
      type: single-node
      seed_hosts: ""

[Dexolite]

In the statefulset.yaml, line#295 in spec.template.spec.containers.env

      - name: cluster.initial_master_nodes
        value: "{{ template "elasticsearch.endpoints" . }}"

Using template function to set the cluster.initial_master_nodes as env variable, which conflicts discovery.type: single-node.

Basically in this form the helmchart is unable to receive a 'discovery.type: single-node' (helm v3.3.3)
I've deleted the env var in statefulset.yaml to workaround this.

[ebuildy]

I am reading everywhere the solution to fix clusering issue is to remove data and restart ... I understand this is ok for dev and test, but on production, this can happen, is it a clean way to fix clustering issue?

[SubhamPramanik]

We're also facing the same issue, we recently upgraded our ES from 6.8 to 7.7. We're getting a similar error as mentioned in https://github.com/elastic/helm-charts/issues/312#issuecomment-545098138.

We're using replicas:1 and tried to use discovery single-node config but it started crash looping as mentioned by @Dexolite.

Unfortunately, deleting PVs didn't help either. Is there anything else that can be done?

[DmitriySafronov]

In the statefulset.yaml, line#295 in spec.template.spec.containers.env

      - name: cluster.initial_master_nodes
        value: "{{ template "elasticsearch.endpoints" . }}"

Using template function to set the cluster.initial_master_nodes as env variable, which conflicts discovery.type: single-node.

Basically in this form the helmchart is unable to receive a 'discovery.type: single-node' (helm v3.3.3)
I've deleted the env var in statefulset.yaml to workaround this.

You can set this variable to empty value like in snippet below:

extraEnvs:
  - name: discovery.type
    value: single-node
  - name: cluster.initial_master_nodes
    value: ""

P.S.: sorry for my bad English

[gilperme]

@SubhamPramanik commenting out the cluster.initial_master_nodes (in my case the configuration yaml) took care of this issue for me. Hopefully you've resolved. FYI, setting it to "" as @DmitriySafronov stated did not work in my case.

[evtimmy]

@gilperme @DaimyGovaert for me, setting the value to null did the trick:

  extraEnvs:
  - name: discovery.type
    value: single-node
  - name: cluster.initial_master_nodes
    value: null