Helm-charts: Elasticsearch pods fail to startup

Created on 12 Sep 2019  ·  16Comments  ·  Source: elastic/helm-charts

Chart Version
7.3.0
Kubernetes Version
1.13
Kubernetes provider
AWS (EKS cluster)
Helm Version
2.11.0
Everytime I run helm install --name elasticsearch elastic/elasticsearch --namespace elasticsearch --set resources.requests.memory=1.5Gi --set resources.limits.memory=1.5Gi --tiller-namespace spinnaker it appears to be deploying the master pods but they don't start.

NAME:   elasticsearch
LAST DEPLOYED: Wed Sep 11 17:49:30 2019
NAMESPACE: elasticsearch
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                    READY  STATUS       RESTARTS  AGE
elasticsearch-master-0  0/1    Init:0/1     0         1s
elasticsearch-master-1  0/1    Terminating  0         5m46s
elasticsearch-master-2  0/1    Terminating  4         5m46s

==> v1/Service
NAME                           TYPE       CLUSTER-IP    EXTERNAL-IP  PORT(S)            AGE
elasticsearch-master           ClusterIP  172.20.72.18  <none>       9200/TCP,9300/TCP  1s
elasticsearch-master-headless  ClusterIP  None          <none>       9200/TCP,9300/TCP  1s

==> v1beta1/PodDisruptionBudget
NAME                      MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
elasticsearch-master-pdb  N/A            1                0                    1s

==> v1beta1/StatefulSet
NAME                  READY  AGE
elasticsearch-master  0/3    1s


NOTES:
1. Watch all cluster members come up.
  $ kubectl get pods --namespace=elasticsearch -l app=elasticsearch-master -w
2. Test cluster health using Helm test.
  $ helm test elasticsearch

Here is the output of running helm get elasticsearch --tiller-namespace spinnaker

REVISION: 1
RELEASED: Wed Sep 11 17:49:30 2019
CHART: elasticsearch-7.3.0
USER-SUPPLIED VALUES:
resources:
  limits:
    memory: 1.5Gi
  requests:
    memory: 1.5Gi

COMPUTED VALUES:
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
clusterHealthCheckParams: wait_for_status=green&timeout=1s
clusterName: elasticsearch
esConfig: {}
esJavaOpts: -Xmx1g -Xms1g
esMajorVersion: ""
extraEnvs: []
extraInitContainers: []
extraVolumeMounts: []
extraVolumes: []
fsGroup: ""
fullnameOverride: ""
httpPort: 9200
image: docker.elastic.co/elasticsearch/elasticsearch
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.3.0
ingress:
  annotations: {}
  enabled: false
  hosts:
  - chart-example.local
  path: /
  tls: []
initResources: {}
labels: {}
lifecycle: {}
masterService: ""
masterTerminationFix: false
maxUnavailable: 1
minimumMasterNodes: 2
nameOverride: ""
networkHost: 0.0.0.0
nodeAffinity: {}
nodeGroup: master
nodeSelector: {}
persistence:
  annotations: {}
  enabled: true
podAnnotations: {}
podManagementPolicy: Parallel
podSecurityContext:
  fsGroup: 1000
priorityClassName: ""
protocol: http
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 3
resources:
  limits:
    cpu: 1000m
    memory: 1.5Gi
  requests:
    cpu: 100m
    memory: 1.5Gi
roles:
  data: "true"
  ingest: "true"
  master: "true"
schedulerName: ""
secretMounts: []
securityContext:
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000
service:
  annotations: {}
  nodePort: null
  type: ClusterIP
sidecarResources: {}
sysctlInitContainer:
  enabled: true
sysctlVmMaxMapCount: 262144
terminationGracePeriod: 120
tolerations: []
transportPort: 9300
updateStrategy: RollingUpdate
volumeClaimTemplate:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

HOOKS:
---
# elasticsearch-sjzcy-test
apiVersion: v1
kind: Pod
metadata:
  name: "elasticsearch-sjzcy-test"
  annotations:
    "helm.sh/hook": test-success
spec:
  containers:
  - name: "elasticsearch-sphns-test"
    image: "docker.elastic.co/elasticsearch/elasticsearch:7.3.0"
    command:
      - "sh"
      - "-c"
      - |
        #!/usr/bin/env bash -e
        curl -XGET --fail 'elasticsearch-master:9200/_cluster/health?wait_for_status=green&timeout=1s'
  restartPolicy: Never
MANIFEST:

---
# Source: elasticsearch/templates/poddisruptionbudget.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: "elasticsearch-master-pdb"
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: "elasticsearch-master"
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-master
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch-7.3.0"
    app: "elasticsearch-master"
  annotations:
    {}

spec:
  type: ClusterIP
  selector:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch-7.3.0"
    app: "elasticsearch-master"
  ports:
  - name: http
    protocol: TCP
    port: 9200
  - name: transport
    protocol: TCP
    port: 9300
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-master-headless
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch-7.3.0"
    app: "elasticsearch-master"
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  clusterIP: None # This is needed for statefulset hostnames like elasticsearch-0 to resolve
  # Create endpoints also if the related pod isn't ready
  publishNotReadyAddresses: true
  selector:
    app: "elasticsearch-master"
  ports:
  - name: http
    port: 9200
  - name: transport
    port: 9300
---
# Source: elasticsearch/templates/statefulset.yaml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: elasticsearch-master
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch-7.3.0"
    app: "elasticsearch-master"
  annotations:
    esMajorVersion: "7"
spec:
  serviceName: elasticsearch-master-headless
  selector:
    matchLabels:
      app: "elasticsearch-master"
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-master
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 30Gi

  template:
    metadata:
      name: "elasticsearch-master"
      labels:
        heritage: "Tiller"
        release: "elasticsearch"
        chart: "elasticsearch-7.3.0"
        app: "elasticsearch-master"
      annotations:

    spec:
      securityContext:
        fsGroup: 1000

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - "elasticsearch-master"
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 120
      volumes:
      initContainers:
      - name: configure-sysctl
        securityContext:
          runAsUser: 0
          privileged: true
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.3.0"
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        resources:
          {}

      containers:
      - name: "elasticsearch"
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000

        image: "docker.elastic.co/elasticsearch/elasticsearch:7.3.0"
        imagePullPolicy: "IfNotPresent"
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5

          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                # If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
                # Once it has started only check that the node itself is responding
                START_FILE=/tmp/.es_start_file

                http () {
                    local path="${1}"
                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    else
                      BASIC_AUTH=''
                    fi
                    curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
                }

                if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy'
                    http "/"
                else
                    echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                        touch ${START_FILE}
                        exit 0
                    else
                        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                        exit 1
                    fi
                fi
        ports:
        - name: http
          containerPort: 9200
        - name: transport
          containerPort: 9300
        resources:
          limits:
            cpu: 1000m
            memory: 1.5Gi
          requests:
            cpu: 100m
            memory: 1.5Gi

        env:
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: cluster.initial_master_nodes
            value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,"
          - name: discovery.seed_hosts
            value: "elasticsearch-master-headless"
          - name: cluster.name
            value: "elasticsearch"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS
            value: "-Xmx1g -Xms1g"
          - name: node.data
            value: "true"
          - name: node.ingest
            value: "true"
          - name: node.master
            value: "true"
        volumeMounts:
          - name: "elasticsearch-master"
            mountPath: /usr/share/elasticsearch/data

And the ooutput of running kubectl get pods --namespace=elasticsearch -l app=elasticsearch-master

NAME                     READY     STATUS             RESTARTS   AGE
elasticsearch-master-0   0/1       CrashLoopBackOff   4          4m
elasticsearch-master-1   0/1       Running            0          4m
elasticsearch-master-2   0/1       Error              4          3m
bug triagstale
Source
picture CallmeTorre
👍1

Most helpful comment

Found the problem: antiAffinity was set to hard and but I didn't have enough nodes so only one pod was being scheduled... since that pod couldn't go green (not finding enough masters) the deploy hanged.

picture ashantyk on 6 Feb 2020
👍3

All 16 comments

If you reduce the amount of memory you will also need to adjust the jvm memory size:

https://github.com/elastic/helm-charts/blob/8502b6f5224e51b228c13e21681770c03e337fb8/elasticsearch/values.yaml#

That could potentially be causing issues although I think that the cluster should still start up properly.

Could you also give me the log output from each of the containers? It's interesting that elasticsearch-master-1 does seem to be running but the others are crashing.

Crazybus picture Crazybus on 18 Sep 2019

I have the same issue. Everything is running (3 nodes) but readinessProbess are failing and causes the pods to restart.
If I try to check the health, this is what I get:

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

(I forwarded port with kubectl port-forward svc/elasticsearch-master 9200 -n logging and then curl 'http://localhost:9200/_cluster/health?wait_for_status=green&timeout=1s'

Logs:
| elasticsearch {"type": "server", "timestamp": "2019-10-10T15:33:46,214Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}", │ │ elasticsearch "stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null", │ │ elasticsearch "at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:214) [elasticsearch-7.4.0.jar:7.4.0]", │ │ elasticsearch "at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.4.0.jar:7.4.0]", │ │ elasticsearch "at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.4.0.jar:7.4.0]", │ │ elasticsearch "at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.4.0.jar:7.4.0]", │ │ elasticsearch "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.4.0.jar:7.4.0]", │ │ elasticsearch "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]", │ │ elasticsearch "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]", │ │ elasticsearch "at java.lang.Thread.run(Thread.java:830) [?:?]"] } │ │ elasticsearch {"type": "server", "timestamp": "2019-10-10T15:33:53,510Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "master not discovered or elected yet, an election requires at least 2 nodes with ids from [jfp510YSTUWui7MHQ_kcNg, wsK1Hg0P │ │ mKHlnCoU0hwZg, -F4nt28_Q7ORXOrpl2_KPQ[], have discovered [{elasticsearch-master-2}{jfp510YSTUWui7MHQ_kcNg}{LtX9Z5psRVKcNtfb3UxNQQ}{10.10.18.183}{10.10.18.183:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] which is not a quorum; discovery will continue using [10.10.1.193:9300, 10.10.34.124:9300] from hosts provid │ │ ers and [{elasticsearch-master-2}{jfp510YSTUWui7MHQ_kcNg}{LtX9Z5psRVKcNtfb3UxNQQ}{10.10.18.183}{10.10.18.183:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 9, last-accepted version 280 in term 9" } │ │

RobertoUa picture RobertoUa on 10 Oct 2019

to reproduce, change replicas to 2, in already deployed ES. elasticsearch-master-1 will try to restart but the probes will fail so it will never be ready (CrashLoopBackOff) and elasticsearch-master-0 will be still running but lots of exceptions in the logs:

│ elasticsearch {"type": "server", "timestamp": "2019-10-10T15:57:46,346Z", "level": "DEBUG", "component": "o.e.a.a.c.n.i.TransportNodesInfoAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to execute on node [wsK1Hg0PTmKHlnCoU0hwZg]", "cluster.uuid": "mvthtCX-QSiRJlptaxKARg", "node.id": "-F4nt28_Q7ORXO │
│ rpl2_KPQ" ,                                                                                                                                                                                                                                                                                                                                                      │
│ elasticsearch "stacktrace": ["org.elasticsearch.transport.NodeNotConnectedException: [elasticsearch-master-1][10.10.0.99:9300] Node not connected",                                                                                                                                                                                                              │
│ elasticsearch "at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:222) ~[elasticsearch-7.4.0.jar:7.4.0]",                                                                                                                                                                                                                     │
│ elasticsearch "at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:617) ~[elasticsearch-7.4.0.jar:7.4.0]",                                                                                                                                                                                                                       │
│ elasticsearch "at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:589) [elasticsearch-7.4.0.jar:7.4.0]",                                                                                                                                                                                                                          │
│ elasticsearch "at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:182) [elasticsearch-7.4.0.jar:7.4.0]",
RobertoUa picture RobertoUa on 10 Oct 2019

@RobertoUa another gotcha that I've encountered is that even if you delete the app that had 3 replicas and install again with 2 replicas but forget to delete persistent volume claims (which don't get removed by Helm when you delete the app), it still fails with MasterNotDiscoveredException.

dolsem picture dolsem on 1 Nov 2019

@dolsem got the same problem on Digital Ocean managed K8s. After removing PVC manually that didn't fix anything :(

holms picture holms on 6 Nov 2019

@dolsem got the same problem on Digital Ocean managed K8s. After removing PVC manually that didn't fix anything :(

@holms Would you mind opening a new bug report with the requested details so we can investigate further.

fatmcgav picture fatmcgav on 6 Nov 2019

@fatmcgav

After increasing readiness probe to 200 it worked:

readinessProbe:
    initialDelaySeconds: 200

And I think it might be due to this exit 1 in readinessProbe that you have in here:
https://github.com/elastic/helm-charts/blob/e463e7a1f4f719687e8b4294bc3223bfbd021915/elasticsearch/templates/statefulset.yaml#L223

After readinessProbe timeout pod always exits. This is not the best practice to do it this way as far as I'm aware

holms picture holms on 6 Nov 2019

@fatmcgav

After increasing readiness probe to 200 it worked:

readinessProbe:
    initialDelaySeconds: 200

And I think it might be due to this exit 1 in readinessProbe that you have in here:
https://github.com/elastic/helm-charts/blob/e463e7a1f4f719687e8b4294bc3223bfbd021915/elasticsearch/templates/statefulset.yaml#L223

After readinessProbe timeout pod always exits. This is not the best practice to do it this way as far as I'm aware

@holms Cheers for the update.

Bumping the initialDelaySeconds value would indicate that, for whatever reason, the pods are taking a long time to start up and become ready for you.
By default, Elasticsearch within the pod is expected to return a 'green' cluster health within 40 seconds (10 second initial delay plus 3 retries at 10 seconds each)...

Maybe it's worth checking the pod logs to see _why_ it's taking soo long to become ready...

fatmcgav picture fatmcgav on 6 Nov 2019

@fatmcgav opened new issue #361 also with logs attached. But really this happens even on my home server. So IDK what is going on, but best practises states you never do exit 1 in readiness, you might end up in such situation anytime in different environment.

holms picture holms on 11 Nov 2019

@holms Cool, cheers. Let's continue this discussion on #361.

fatmcgav picture fatmcgav on 11 Nov 2019

@CallmeTorre Is this still an issue for you?

fatmcgav picture fatmcgav on 13 Nov 2019

@RobertoUa another gotcha that I've encountered is that even if you delete the app that had 3 replicas and install again with 2 replicas but forget to delete persistent volume claims (which don't get removed by Helm when you delete the app), it still fails with MasterNotDiscoveredException.

Indeed. In my case, it was some leftover pvc's from a removed es cluster, causing the NodeNotConnectedException issue. Kind of expected. I deleted the related pvc's and redeployed the cluster and the issue is gone. The number of replicas was the same on both cases though.

christidis picture christidis on 22 Jan 2020

Hi guys,

I'm also receiving master_not_discovered_exception / MasterNotDiscoveredException and I can't figure out why.

Kubernetes: v1.17.0
CIDR: flannel
Chart version: 7.5.2

Manifest:

---
# Source: elasticsearch/templates/poddisruptionbudget.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: "elasticsearch-master-pdb"
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: "elasticsearch-master"
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-master
  labels:
    release: "es"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  annotations:
    {}
spec:
  type: ClusterIP
  selector:
    release: "es"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  ports:
  - name: http
    protocol: TCP
    port: 9200
  - name: transport
    protocol: TCP
    port: 9300
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-master-headless
  labels:
    release: "es"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  clusterIP: None # This is needed for statefulset hostnames like elasticsearch-0 to resolve
  # Create endpoints also if the related pod isn't ready
  publishNotReadyAddresses: true
  selector:
    app: "elasticsearch-master"
  ports:
  - name: http
    port: 9200
  - name: transport
    port: 9300
---
# Source: elasticsearch/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-master
  labels:
    release: "es"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  annotations:
    esMajorVersion: "7"
spec:
  serviceName: elasticsearch-master-headless
  selector:
    matchLabels:
      app: "elasticsearch-master"
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-master
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
  template:
    metadata:
      name: "elasticsearch-master"
      labels:
        release: "es"
        chart: "elasticsearch"
        app: "elasticsearch-master"
      annotations:

    spec:
      securityContext:
        fsGroup: 1000
        runAsUser: 1000
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - "elasticsearch-master"
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 120
      volumes:
      initContainers:
      - name: configure-sysctl
        securityContext:
          runAsUser: 0
          privileged: true
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.5.2"
        imagePullPolicy: "IfNotPresent"
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        resources:
          {}

      containers:
      - name: "elasticsearch"
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000
        image: "docker.elastic.co/elasticsearch/elasticsearch:7.5.2"
        imagePullPolicy: "IfNotPresent"
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 50
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5
          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                # If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
                # Once it has started only check that the node itself is responding
                START_FILE=/tmp/.es_start_file

                http () {
                    local path="${1}"
                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    else
                      BASIC_AUTH=''
                    fi
                    curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
                }

                if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy and there are master nodes available'
                    http "/_cluster/health?timeout=0s"
                else
                    echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                        touch ${START_FILE}
                        exit 0
                    else
                        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                        exit 1
                    fi
                fi
        ports:
        - name: http
          containerPort: 9200
        - name: transport
          containerPort: 9300
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 2Gi
        env:
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: cluster.initial_master_nodes
            value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,"
          - name: discovery.seed_hosts
            value: "elasticsearch-master-headless"
          - name: cluster.name
            value: "elasticsearch"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS
            value: "-Xmx1g -Xms1g"
          - name: node.data
            value: "true"
          - name: node.ingest
            value: "true"
          - name: node.master
            value: "true"
        volumeMounts:
          - name: "elasticsearch-master"
            mountPath: /usr/share/elasticsearch/data
---
# Source: elasticsearch/templates/test/test-elasticsearch-health.yaml
apiVersion: v1
kind: Pod
metadata:
  name: "es-khddf-test"
  annotations:
    "helm.sh/hook": test-success
spec:
  containers:
  - name: "es-xsqvw-test"
    image: "docker.elastic.co/elasticsearch/elasticsearch:7.5.2"
    command:
      - "sh"
      - "-c"
      - |
        #!/usr/bin/env bash -e
        curl -XGET --fail 'elasticsearch-master:9200/_cluster/health?wait_for_status=green&timeout=1s'
  restartPolicy: Never

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0   0/1     Running   0          14m
elasticsearch-master-1   0/1     Pending   0          14m
elasticsearch-master-2   0/1     Pending   0          14m
es-khddf-test            0/1     Error     0          14m

and logs for the Running pod keeps repeating:

{"type": "server", "timestamp": "2020-02-04T15:31:04,041Z", "level": "DEBUG", "component": "o.e.a.s.m.TransportMasterNodeAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "timed out while retrying [cluster:monitor/health] after failure (timeout [1s])" }
{"type": "server", "timestamp": "2020-02-04T15:31:04,042Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "path: /_cluster/health, params: {wait_for_status=green, timeout=1s}", 
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:218) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.5.2.jar:7.5.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.2.jar:7.5.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]"] }
{"type": "server", "timestamp": "2020-02-04T15:31:05,590Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [elasticsearch-master-0, elasticsearch-master-1, elasticsearch-master-2] to bootstrap a cluster: have discovered [{elasticsearch-master-0}{jcDINcndSQSZ5fPBuhiyQw}{VOTh1xvtSLy9ygU_R-IVFw}{192.168.0.117}{192.168.0.117:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [] from hosts providers and [{elasticsearch-master-0}{jcDINcndSQSZ5fPBuhiyQw}{VOTh1xvtSLy9ygU_R-IVFw}{192.168.0.117}{192.168.0.117:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2020-02-04T15:31:13,039Z", "level": "DEBUG", "component": "o.e.a.s.m.TransportMasterNodeAction", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "no known master node, scheduling a retry" }

SSH-ing into the pod and running curl on elasticsearch-master-0:9200 and elasticsearch-master-headless:9200 resolves the names and returns:

{
  "name" : "elasticsearch-master-0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "7.5.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
    "build_date" : "2020-01-15T12:11:52.313576Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

but doing /_cluster/health gives:

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

At this point I don't have any ideas on what to check further. Please help :(

ashantyk picture ashantyk on 4 Feb 2020

Found the problem: antiAffinity was set to hard and but I didn't have enough nodes so only one pod was being scheduled... since that pod couldn't go green (not finding enough masters) the deploy hanged.

ashantyk picture ashantyk on 6 Feb 2020
👍3

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

botelastic[bot] picture botelastic[bot] on 6 May 2020

This issue has been automatically closed because it has not had recent activity since being marked as stale.

botelastic[bot] picture botelastic[bot] on 5 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ilhaan picture ilhaan  ·  7Comments
ppawel picture ppawel  ·  4Comments
dperetti picture dperetti  ·  4Comments
kawat55 picture kawat55  ·  3Comments
naseemkullah picture naseemkullah  ·  4Comments