Helm-charts: error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/]

[melissajenner22]

Chart version:
7.7.1
Kubernetes version:
1.16
Kubernetes provider: E.g. GKE (Google Kubernetes Engine)
eks
Helm Version:

2.16.10
helm get release output

e.g. helm get elasticsearch (replace elasticsearch with the name of your helm release)

Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting.

If you find some secrets in plain text in helm get release output you should use Kubernetes Secrets to managed them is a secure way (see Security Example).

Output of helm get release

$ helm get logstash
REVISION: 1
RELEASED: Thu Aug 13 20:48:50 2020
CHART: logstash-7.7.1
USER-SUPPLIED VALUES:
{}

COMPUTED VALUES:
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
envFrom: []
extraContainers: ""
extraEnvs: []
extraInitContainers: ""
extraPorts: []
extraVolumeMounts: ""
extraVolumes: ""
fullnameOverride: ""
httpPort: 9600
image: docker.elastic.co/logstash/logstash
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.7.1
labels: {}
lifecycle: {}
livenessProbe:
  failureThreshold: 3
  httpGet:
    path: /
    port: http
  initialDelaySeconds: 300
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 5
logstashConfig: {}
logstashJavaOpts: -Xmx1g -Xms1g
logstashPipeline: {}
maxUnavailable: 1
nameOverride: ""
nodeAffinity: {}
nodeSelector: {}
persistence:
  annotations: {}
  enabled: false
podAnnotations: {}
podManagementPolicy: Parallel
podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000
podSecurityPolicy:
  create: false
  name: ""
  spec:
    fsGroup:
      rule: RunAsAny
    privileged: true
    runAsUser:
      rule: RunAsAny
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    volumes:
    - secret
    - configMap
    - persistentVolumeClaim
priorityClassName: ""
rbac:
  create: false
  serviceAccountName: ""
readinessProbe:
  failureThreshold: 3
  httpGet:
    path: /
    port: http
  initialDelaySeconds: 60
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 1
resources:
  limits:
    cpu: 1000m
    memory: 1536Mi
  requests:
    cpu: 100m
    memory: 1536Mi
schedulerName: ""
secretMounts: []
securityContext:
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000
service: {}
terminationGracePeriod: 120
tolerations: []
updateStrategy: RollingUpdate
volumeClaimTemplate:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

HOOKS:
MANIFEST:

---
# Source: logstash/templates/poddisruptionbudget.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: "logstash-logstash-pdb"
  labels:
    app: "logstash-logstash"
    chart: "logstash"
    heritage: "Tiller"
    release: "logstash"
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: "logstash-logstash"
---
# Source: logstash/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: logstash-logstash
  labels:
    app: "logstash-logstash"
    chart: "logstash"
    heritage: "Tiller"
    release: "logstash"
spec:
  selector:
    matchLabels:
      app: "logstash-logstash"
      release: "logstash"
  replicas: 1
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: "logstash-logstash"
      labels:
        app: "logstash-logstash"
        chart: "logstash"
        heritage: "Tiller"
        release: "logstash"
      annotations:


    spec:
      securityContext:
        fsGroup: 1000
        runAsUser: 1000

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - "logstash-logstash"
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 120
      volumes:
      containers:
      - name: "logstash"
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000

        image: "docker.elastic.co/logstash/logstash:7.7.1"
        imagePullPolicy: "IfNotPresent"
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: http
          initialDelaySeconds: 300
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5

        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: http
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5

        ports:
        - name: http
          containerPort: 9600
        resources:
          limits:
            cpu: 1000m
            memory: 1536Mi
          requests:
            cpu: 100m
            memory: 1536Mi

        env:
          - name: LS_JAVA_OPTS
            value: "-Xmx1g -Xms1g"
        volumeMounts:

Describe the bug:

$ kubectl logs logstash-logstash-0

[2020-08-14T03:51:17,479][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2020-08-14T03:51:17,569][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}

**Steps to reproduce:**

1.
helm install --name elasticsearch ./elasticsearch --namespace elk
2.
$ helm install --name kibana ./kibana --namespace elk
3.
$ helm install --name logstash ./logstash --namespace elk
**Expected behavior:**

**Provide logs and/or server output (if relevant):**

*Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting*

```

Any additional context:

[fatmcgav]

@melissajenner22 this looks like a config issue.

Is there an elasticsearch deployment in the same namespace?

Can you provide the output of kubectl get pods and kubectl get services?

[melissajenner22]

$ kubectl get pods -n elk
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          14h
kibana-kibana-7458f597b7-hhj95   1/1     Running   0          13h
logstash-logstash-0              1/1     Running   0          12h

$ kubectl get services -n elk
NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
elasticsearch-master            ClusterIP   122.20.129.47    <none>        9200/TCP,9300/TCP   14h
elasticsearch-master-headless   ClusterIP   None             <none>        9200/TCP,9300/TCP   14h
kibana-kibana                   ClusterIP   122.20.228.118   <none>        5601/TCP            13h

A side question. How to enable EXTERNAL-IP (loadBalancer)?

[kudrew]

A side question. How to enable EXTERNAL-IP (loadBalancer)?

On GKE, you can create a service of type LoadBalancer or change the value of service in Chart.yaml from ClusterIP to LoadBalancer. (This will open up your FW unless you give it the proper annotations to make the LB internal and specify a Source Range and expose the service with an Ingress)

[es1o]

I have the same problem. I use standalone logstash helm chart and it is trying to connect elasticsearch:9200 because of license. How can I disable this check?

[jmlrt]

Hi,
You should be able to disable Logstash monitoring to Elasticsearch feature by adding this line to your values:

https://github.com/elastic/helm-charts/blob/6cfbe0078b178d678940b10057243f4e1e1c353d/logstash/examples/elasticsearch/values.yaml#L7

logstashConfig:
  logstash.yml: |
    xpack.monitoring.enabled: false

[abbbhardwaj]

Hi

I am experiencing the same issue, I use helm charts and want to change the hostname of elasticsearch url from http://elasticsearch:9200/ to http://elasticsearch-master:9200/ in logstash helm chart values file. how can I do it?

[jmlrt]

Hi @abbbhardwaj, if you want to use Logstash monitoring with a different Elasticsearch URL, you can override it with the following setting:

logstashConfig:
  logstash.yml: |
    monitoring.elasticsearch.hosts: http://elasticsearch:9200

source: https://www.elastic.co/guide/en/logstash/current/docker-config.html#_docker_defaults

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jmlrt]

closing as stale