Containers-roadmap: [AWS for Fluent Bit] [request]: Support sending logs direct to managed AWS ElasticSearch

Created on 15 Jan 2020 · 7Comments · Source: aws/containers-roadmap

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

Newer versions of Amazon ElasticSearch Service require Sigv4 auth with valid AWS credentials. Fluent Bit has an output plugin for ES, but it does not support Sigv4 auth.

Many customers have asked for the ability to send logs directly to Amazon ES. This applies to FireLens users and EKS users who run Fluent Bit as a daemonset.

EKS FireLens OSS Work in Progress

Source

PettitWesley

👍80

Most helpful comment

@antoine-choimet-cbp This will be released as part of Fluent Bit 1.5, tentatively planned for some time this month.

PettitWesley on 2 Jun 2020

👍4

All 7 comments

Also tracked in fluent/fluent-bit#1807 and fluent/fluent-bit#1804

PettitWesley on 15 Jan 2020

The first step is Sigv4 and AWS Credential support in Fluent Bit; this work is almost done:

PettitWesley on 15 Jan 2020

https://github.com/fluent/fluent-bit/pull/2011

PettitWesley on 8 Mar 2020

Hello, have you any news about this request?

antoine-choimet-cbp on 2 Jun 2020

@antoine-choimet-cbp This will be released as part of Fluent Bit 1.5, tentatively planned for some time this month.

PettitWesley on 2 Jun 2020

👍4

This was shipped a while ago, closing.

PettitWesley on 17 Sep 2020

I realise this is closed-as-shipped, but it might be nice to mention this Fluent-bit integration as an ingestion option in the ELK feature page, since it's talking about other non-Logstash options.