Elasticsearch-js: Can't connect to Elasticsearch with Node.Js on Kubernetes (socket hang up)

[flolu]

I have a NodeJs application running inside a Kubernetes cluster. I've also followed the official steps to setup Elasticsearch on Kubernetes.

I am not sure if this has actually something to do with elasticsearch-js or if I am doing something wrong here.

Issue

I am unable to connect to the Elasticsearch cluster. I get this error:

ConnectionError: socket hang up

This is a code snippet of my connection:

const client = new elasticsearch.Client({
  node: process.env.elasticsearch_node, // http://elasticsearch-es-http.default.svc.cluster.local:9200
});

Minimal reproduction

I've created a minimal reproduction of this issue here: https://github.com/flolu/elasticsearch-k8s-connection. (Setup instructions are in the README)

Basically, everything works fine when running Elasticsearch inside Docker compose, but I can't connect when running inside Kubernetes.

[delvedor]

Hello! It looks like you need to use https, be sure to use the self-signed certificates provided by ECK. Finally, you also need to add the basic authentication details provided by ECK :)

[flolu]

@delvedor okay, that makes sense.

But obviously just switching to https won't let me connect to Elastic, either. I get this error:

ConnectionError: self signed certificate in certificate chain

So... is there any documentation about using the certificates? I mean do I need to configure the certificates inside my NodeJs application or at the cluster level?

[delvedor]

Given that you are using a self-signed certificate, you should also set rejectUnauthorized to false:

const { Client } = require('@elastic/elasticsearch')
const client = new Client({
  node: 'https://localhost:9200',
  auth: {
    username: 'elastic',
    password: 'changeme'
  },
  ssl: {
    ca: fs.readFileSync('./cacert.pem'),
    rejectUnauthorized: false
  }
})

You can find the SSL documentation here.

[flolu]

@delvedor Ahh okay. That's great!

On Kubernetes, Elasticsearch provides a number of different certificates. Which one should I choose when running Elasticsearch on Kubernetes in production?

default-token-kwmnv                       kubernetes.io/service-account-token   3      12h
elasticsearch-es-default-es-config        Opaque                                1      12h
elasticsearch-es-elastic-user             Opaque                                1      12h
elasticsearch-es-http-ca-internal         Opaque                                2      12h
elasticsearch-es-http-certs-internal      Opaque                                3      12h
elasticsearch-es-http-certs-public        Opaque                                2      12h
elasticsearch-es-internal-users           Opaque                                2      12h
elasticsearch-es-remote-ca                Opaque                                1      12h
elasticsearch-es-transport-ca-internal    Opaque                                2      12h
elasticsearch-es-transport-certificates   Opaque                                7      12h
elasticsearch-es-transport-certs-public   Opaque                                1      12h
elasticsearch-es-xpack-file-realm         Opaque                                3      12h

Those secrets have the following data:

Data
====
ca.crt:   1176 bytes
tls.crt:  2551 bytes
tls.key:  1675 bytes

Thus I would use ca.crt for the ssl.ca field in the NodeJs client, right?

[flolu]

@delvedor Thank you so much! I've made it work!

I am now passing the certificate and the elastic user password into my NodeJs service via environment variables like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: search-deployment
spec:
  selector:
    matchLabels:
      app: search
  replicas: 1
  template:
    metadata:
      labels:
        app: search
    spec:
      containers:
        - name: search
          image: search:placeholder_name
          imagePullPolicy: Always
          env:
            - name: elasticsearch_node
              value: https://elasticsearch-es-http.default.svc.cluster.local:9200
            - name: elasticsearch_certificate
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-es-http-ca-internal
                  key: tls.crt
            - name: elasticsearch_password
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-es-elastic-user
                  key: elastic

And then inside the NodeJs application I connect like this:

const client = new elasticsearch.Client({
  node: process.env.elasticsearch_node,
  auth: {
    username: "elastic",
    password: process.env.elasticsearch_password || "changeme",
  },
  ssl: {
    ca: process.env.elasticsearch_certificate,
    rejectUnauthorized: false,
  },
});

I am just wondering if it is recommended to set rejectUnauthorized to false in production?
If I set ti to true I get this error:
```
ConnectionError: Hostname/IP does not match certificate's altnames: Host: elasticsearch-es-http.default.svc.cluster.local. is not in the cert's altnames: DNS:elasticsearch-es-http.default.es.local, DNS:elasticsearch-es-http, DNS:elasticsearch-es-http.default.svc, DNS:elasticsearch-es-http.default
at ClientRequest. (node_modules/@elastic/elasticsearch/lib/Connection.js:107:18)
at ClientRequest.emit (events.js:210:5)
at TLSSocket.socketErrorListener (_http_client.js:406:9)
at TLSSocket.emit (events.js:210:5)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
````

[delvedor]

Glad you were able to sort it out!
Yeah, it's fine to use it as long as you trust that certificate. By default, Node.js rejects certificates that didn't use an official CA authority, but given that you have created that cert via Elasticsearch, it's safe to use.

Closing this, feel free to reopen if you have other questions! :)