Containers-roadmap: [FireLens] [request]: FireLens/Fluent Bit as a unified observability solution

[PettitWesley]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
At the moment, users often run 2 - 3 side cars or daemons for observability. Fluent Bit/FireLens takes care of logs, but then you need a metrics agent and a tracing agent. This proliferation of agents adds management overhead.

We are evaluating turning Fluent Bit into a more general observability solution. The community is already working on a Statsd input support.

This idea is very early-stage; please give us your thoughts and let us know if you think it would be valuable.

Which service(s) is this request for?
Fargate, EKS, ECS

[binarylogic]

Hey @PettitWesley, I can confirm this is a valuable idea, but would it be possible to consider vector.dev as an _additional_ option? This was part of our original design and Vector currently solves this problem. You can see a list of metrics integrations here and log integrations here. Information on our data model is here.

We'd _love_ the opportunity to work with you and AWS. We're big fans of AWS, Firelens, ECS, and EKS. I believe Vector would be a fantastic option for your users. We've been seeing a lot of success and happy users (over 100K downloads per day), and I'd be happy to refer to you a number of happy Vector users. Let me know!

[PettitWesley]

Update: This initiative will possibly not use Fluent Bit; instead, we are evaluating the incubating CNCF project OpenTelemetry. The project has contribution and involvement from a large set of monitoring providers; the OpenTelemetry Collector is evolving into a unified telemetry router.

Nothing is set in stone; this initiative is still very much "Under consideration".

[singhkays]

@PettitWesley I'm looking into FluentBit as a metrics collector as well and came across this. Can you share if there were reasons other than the emergence of OpenTelemetry that led to dropping this idea?

[PettitWesley]

@singhkays The emergence of OpenTelemetry was the main factor- I think its important to have a robust community focused on solving the same problems. Fluent Bit/Fluentd are great for logging because that's what the community and maintainers are focused on. OpenTelemetry is full of experts who care about metrics/tracing.

Besides that, the OpenTelemetry collector is also written in Golang. That's a huge advantage that puts it ahead of other projects in C (Fluent Bit) and Rust. I think Go sits at a very good point on all tradeoffs for a language- easy to learn, large number of devs who already know it, easy to write code in, but still performant enough.