Fluent-bit: http output to AWS elasticsearch

[caulagi]

I am trying to send container logs from my kubernetes cluster to elasticsearch cluster. I use the hosted elasticsearch service on AWS. I used the configurations from https://github.com/fluent/fluent-bit-kubernetes-logging/tree/0.13-dev based on recommendation in https://github.com/fluent/fluent-bit/issues/468, with this difference -

$ git diff
diff --git a/output/elasticsearch/fluent-bit-configmap.yaml b/output/elasticsearch/fluent-bit-configmap.yaml
index e56224c..fb9ebc0 100644
--- a/output/elasticsearch/fluent-bit-configmap.yaml
+++ b/output/elasticsearch/fluent-bit-configmap.yaml
@@ -43,12 +43,17 @@ data:

   output-elasticsearch.conf: |
     [OUTPUT]
-        Name            es
-        Match           *
-        Host            ${FLUENT_ELASTICSEARCH_HOST}
-        Port            ${FLUENT_ELASTICSEARCH_PORT}
-        Logstash_Format On
-        Retry_Limit     False
+        Name  http
+        Match *
+        Host  vpc-<redacted>-v6bo2foroydhur6ewg7ja7bxia.eu-central-1.es.amazonaws.com
+        Port  443
+        tls On
+        tls.verify On
+        tls.debug On
+        Index k8s
+        Retry_Limit 3
+        Format json
+        Retry_Limit False

I get the following errors -

Elasticsearch 6.2 - [2018/03/25 17:23:39] [error] [out_http] HTTP STATUS=405
Elasticsearch 5.5 - [2018/03/25 17:14:20] [error] [out_http] HTTP STATUS=400

My test container in kubernetes generating the logs was a simple nginx setup created by helm create.

I am able to able to create documents in the elasticsearch cluster in AWS using - curl -XPOST elasticsearch_domain_endpoint/movies/movie -d '{"title": "Spirited Away"}' -H 'Content-Type: application/json.

[caulagi]

The logs look like this -

root@fluent-bit-dxphn:/var/log/containers# tail -f fashionable-moose-foobar-799c59777-f7qft_logging_foobar-c474567d56b23e244371e7de08862a06a92b3a2d4ba90c62c82092b9d438175c.log
{"log":"100.96.4.1 - - [25/Mar/2018:17:43:36 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"kube-probe/1.8\" \"-\"\n","stream":"stdout","time":"2018-03-25T17:43:36.682453507Z"}
{"log":"100.96.4.1 - - [25/Mar/2018:17:43:46 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"kube-probe/1.8\" \"-\"\n","stream":"stdout","time":"2018-03-25T17:43:46.007681252Z"}
{"log":"100.96.4.1 - - [25/Mar/2018:17:43:46 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"kube-probe/1.8\" \"-\"\n","stream":"stdout","time":"2018-03-25T17:43:46.682401599Z"}
{"log":"100.96.4.1 - - [25/Mar/2018:17:43:56 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"kube-probe/1.8\" \"-\"\n","stream":"stdout","time":"2018-03-25T17:43:56.007616317Z"}

[edsiper]

@caulagi

why you don't use our native Elasticsearch output plugin ?

[caulagi]

@edsiper, the hosted instance of ES on AWS allows access only over https and I am not able to connect to the cluster on 9200.

[edsiper]

so you can use the following configuration:

[OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            443
        Logstash_Format On
        Retry_Limit     False
        TLS             On

[caulagi]

Ah, I didn't know that was possible. Quite neat! Thanks, works now.

[discordianfish]

fluent-bit doesn't support authentication/requests signining for the aws managed service, right?
I'm using a proxy for that right now, but native support would be more reliable and secure.

[jason-riddle]

I’m guessing the best compromise would be to use an aws signing proxy in combination with the elasticsearch plugin? Are there any instructions on how to do this?

[thadamski]

@jason-riddle This article has instructions for using a proxy service to sigv4 sign your elasticsearch output. It references this repo, which is basically a merging of https://github.com/fluent/fluent-bit-kubernetes-logging and https://github.com/abutaha/aws-es-proxy

I personally have been using https://github.com/abutaha/aws-es-proxy inside of a service, and point my fluentbit to the proxy service instead of directly at AWS ES

[daparthi001]

My hosted aws vpc elastic search is not receiving logs from k8s cluster same vpc.

[PettitWesley]

@daparthi001 What is your auth setting in Amazon ES?

Check out #1807 and #1804