Cloud-on-k8s: Using self-signed certificates for filebeat

[SebastianCaceresUltra]

Bug Report

I'm trying to deploy filebeat with the ECK cluster I deployed using the quickstart.

What did you do?

I created the filebeat deployment along with its necessary service-account and cluster-role-binding.
I'm using the $CLUSTER_NAME-es-http-certs-internal, and $CLUSTER_NAME-es-http-ca-internal, as the filebeat configureation requires setting the output.elasticsearch.ssl.certificate_authorities, the output.elasticsearch.ssl.certificate and the output.elasticsearch.ssl.key variables.
What did you expect to see?

I would have expected filebeat to be able to access the elasticsearch cluster.

What did you see instead? Under which circumstances?

I saw error messages on the filebeat logs, and on the elasticsearch logs. I think it might be a
certificate issue.

Environment

• ECK version:

  0.9.0

• Kubernetes information:

  • Cloud: GKE

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"archive", BuildDate:"2019-08-29T18:43:18Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.6-gke.13", GitCommit:"fcbc1d20b6bca1936c0317743055ac75aef608ce", GitTreeState:"clean", BuildDate:"2019-06-19T20:50:07Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}

• Resource definition:

# configmap
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    app: filebeat
data:
  filebeat.yml: |-
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl:
        certificate_authorities: /etc/ssl/certs/ca/tls.crt
        certificate: /etc/ssl/certs/client/tls.crt
        key: /etc/ssl/certs/client/tls.key

# DaemonSet
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  labels:
    app: filebeat
spec:
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.3.2
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch-logging-es-http
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: elasticsearch-logging-es-elastic-user
              key: elastic
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: certificates
          mountPath: /etc/ssl/certs/client
          readOnly: true
        - name: ca
          mountPath: /etc/ssl/certs/ca
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
      - name: certificates
        secret:
          secretName: elasticsearch-logging-es-http-certs-internal
      - name: ca
        secret:
          secretName: elasticsearch-logging-es-http-ca-internal

• Logs:

# Filebeat

2019-09-23T13:54:39.486Z    ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(http://elasticsearch-logging-es-http:9200)): Get http://elasticsearch-logging-es-http:9200: EOF
2019-09-23T13:54:39.486Z    INFO    [publisher] pipeline/retry.go:189   retryer: send unwait-signal to consumer
2019-09-23T13:54:39.486Z    INFO    [publisher] pipeline/retry.go:191     done
2019-09-23T13:54:39.487Z    INFO    [publisher] pipeline/retry.go:166   retryer: send wait signal to consumer
2019-09-23T13:54:39.487Z    INFO    pipeline/output.go:93   Attempting to reconnect to backoff(elasticsearch(http://elasticsearch-logging-es-http:9200)) with 27 reconnect attempt(s)
2019-09-23T13:54:39.487Z    INFO    [publisher] pipeline/retry.go:168     done

# Elasticsearch

{"type": "server", "timestamp": "2019-09-23T13:53:48,744+0000", "level": "WARN", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "elasticsearch-logging", "node.name": "elasticsearch-logging-es-bxcgm4l89z", "cluster.uuid": "xxwxDHc0TJme8Fx1hZok1Q", "node.id": "kKTx9zoPQfu2xxKHZBYglg",  "message": "caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/10.132.15.214:37448}" , 
"stacktrace": ["io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a20656c61737469637365617263682d6c6f6767696e672d65732d687474703a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a417574686f72697a6174696f6e3a204261736963205a57786863335270597a6f324e445a79656a56344e32707764476479626d527a64484233596d746e4f58633d0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.35.Final.jar:4.1.35.Final]",
"at java.lang.Thread.run(Thread.java:835) [?:?]",
"Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a20656c61737469637365617263682d6c6f6767696e672d65732d687474703a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a417574686f72697a6174696f6e3a204261736963205a57786863335270597a6f324e445a79656a56344e32707764476479626d527a64484233596d746e4f58633d0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1206) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.35.Final.jar:4.1.35.Final]",
"... 16 more"] }

[barkbay]

Hi,

not an SSL/TLS record

I guess the problem comes from the fact that filebeat is using HTTP by default: https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#protocol-option

Also see this documentation about the secrets to use when accessing Elasticsearch: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-accessing-elastic-services.html#k8s-default-self-signed-certificate

[pebrc]

I have an example of a filebeat configuration with ECK here https://github.com/pebrc/cloud-on-k8s/blob/d3eae6cb20b4bba6c77debc0075f93d405733089/config/samples/filebeat/filebeat-kubernetes.yaml Please note that the Elasticsearch cluster name is hard-coded in this example as are the names of the secrets (this was for a demo only)

In short as Michael has pointed out:

• you need to use https to talk to Elasticsearch
  also:
• ssl.certificate and ssl.key are not necessary unless you want to present a client certificate
• you should not use the secrets with internal in the name, they are meant to be used by ECK only but the one suffixed with -public

[SebastianCaceresUltra]

Thank you both for your answers.

The protocol configuration was necessary for it to work, and it did the trick.

I was also using the wrong certificates (too many certificates actually), so I fixed that too.

Thanks again!