Fluent-bit: Add Loki output plugin

[redbaron]

Describe the solution you'd like

Grafanalabs announced Loki log management system, it would be really nice to have output plugin in Fluentbit for it.

Describe alternatives you've considered

Loki comes with their own promtail log collector, nobody really needs yet another log collector.

Additional context

Loki API is dead simple and documented in https://github.com/grafana/loki/blob/master/docs/api.md

[cosmo0920]

Hi, I've created grafana loki plugin which uses Golang interface: https://github.com/cosmo0920/fluent-bit-go-loki

[cosmo0920]

I’ve sent a PR to create Grafana/Loki fluent-bit-go plugin and merged in https://github.com/grafana/loki/pull/847

[dawidmalina]

Would be good if this plugin could be moved from grafana and promoted as official plugin under fluent-bit umbrella same as plugin for es or splunk.

[TPX01]

Any update on this? I've currently created a hybrid fluent-bit (using the image provided by grafana and config from fluent bit itself). 🚀

With minimal tweaks I was able to get it working on Openshift. Both promtail and fluent-bit loki plugin (from Grafana) are proving to be very hard to get up and running. I can share my helm config if needed..

My C experience is very rusty so maybe not the best idea for me at this point 😅

[edsiper]

Yes, there is a plan.

One feature that we are working on Fluent Bit core is the ability to send chunks of data "in order".

Fluent Bit was designed for fast performance and data delivery, meaning we send data in parallel. This could lead to having data chunks arriving at different intervals on the destination, for most of the services this is not a problem, but Loki requires that all data arrives ordered by timestamp which is something we cannot guarantee at the moment, consider this case:

• Chunk A -> Failed to deliver (network issue or whatever)
• Chunk A -> will be retried in N seconds
• Chunk B -> Delivered
• Chunk A -> Cannot deliver because timestamps are earlier than Chunk B timestamps

the solution is to implement a logic to deliver data in order without multiplexing and in order. Of course, this hurt performance (meaning: deliver data at a slow rate) but solves the problem. This part is "work in process"

[TPX01]

@edsiper thank you for elaborating this! Really appreciated 👌

[dojci]

I've created buffering inside loki output plugin https://github.com/grafana/loki/pull/2142 as an alternative to the future in order flb core buffering

[lanphan]

Hi @edsiper ,
Is there any update on this? We're evaluating fluent-bit as a very excellent log collector, but we need to send logs to Loki.
Looking forward to your response.
Thanks.

[edsiper]

@lanphan it will be implemented but other previous tasks are required like custom buffering (work in process)

For now you can use the golang plugin available for Fluent Bit

[lanphan]

Thanks for your quick response, @edsiper .
As I'm new to Fluent-bit, I searched around and found that:

1. https://docs.fluentbit.io/manual/pipeline/outputs : there is no output for Loki here
2. https://github.com/grafana/loki/commit/b27f92e86bba86841d0947d416636cbe8d4eaf74 : fluent-bit plugin in Loki repo

What you mentioned is #2 I found above?

[cosmo0920]

2\. [grafana/loki@b27f92e](https://github.com/grafana/loki/commit/b27f92e86bba86841d0947d416636cbe8d4eaf74) : fluent-bit plugin in Loki repo

Yeah, you can use official golang plugin for loki.

[lanphan]

Thank @cosmo0920 .
However, just a bit curious, as @edsiper explained above about the data order issue fluent-bit got, how can fluent-bit plugin of loki solved that problem?

[cosmo0920]

fluent-bit golang plugin of Loki uses dque to implement FIFO queue which is described here:
https://github.com/grafana/loki/pull/2142/files#diff-447765ac3e6f9a153bbe056c0855b73bR84-R101

[edsiper]

Fixed: https://docs.fluentbit.io/manual/pipeline/outputs/loki