Fluent-bit: out_forward: support client side loadbalancing / failover

[f0]

Hi,

it would be nice if fluentbit supports multiple targets and loadbalancing between them, at least failover should be supported

regards f0

[nEosAg007]

Thanks f0 for reporting this enhancement. I would also like to have this feature.

Regards,

[fxposter]

I personally don't think this feature is a must have. I believe you can have a TCP load balancer in front of your multiple Fluentd-s and it will do the magic for you (including healthchecks/etc). As @edsiper said - currently fluent-bit opens one TCP connection per chunk of data on "flush" and sends it to receiving end. With this kind of approach load balancer will spread the load and make it work just fine.
I'd just like to have this setup to be documented/recommended somewhere in the docs.

[edsiper]

Actually there many users who needs this because they cannot use a third-party balancer. So it's a feature that will be implemented shortly.

Having a balancer in the middle between Fluent Bit and Fluentd is something that works too, I will make sure to mention that in the documentation but cannot recommend X solution since it's out of Fluent scope.

[katrielt]

Would like to see this implemented as well, so I can replace fluentd with fluent-bit in clients. We use multiple outputs to the same load balancer (different ports) to better utilize our collector nodes. This setus allows running 4 or 5 fluentd instances per collector node.

[katrielt]

Any plans to implement this in the near future?

[edsiper]

FYI: To be implemented, no ETA yet due to other priorities.

[cdj03]

any updates on this issue ? Would like to see fail over capability similar to the multiple servers config option in fluentd now.

[edsiper]

Load balancing design is underway, it will be part of 0.14 release, no fixed date yet.

[cdj03]

Thanks for the update. Will look forward to the 0.14 release......:)

[wpalmeri]

@edsiper While waiting on this to come in 0.14, I verified that fluentbit will balance traffic and properly fail over if I point a Forward target at a kubernetes cluster DNS name that resolves with multiple A records. Question: under the hood does fluentbit pin forwarding from individual Inputs (or record tags) to a single connection? Or does this introduce the possibility that logs from a single Input (eg a single docker container) could now be sent to different upstream targets?

[edsiper]

FYI: Feature merged in GIT Master :)

more details here: https://github.com/fluent/fluent-bit/commit/dcd72451b9707333ae62b9dd3faf26f7575c10d6

[edsiper]

Balancing mode (round-robin for now) has been implemented. In order to get started read about the Upstream Servers feature and it config here:

https://docs.fluentbit.io/manual/configuration/upstream_servers

[stonith]

@edsiper any plans on adding failover support as well?

[musaprg]

@edsiper Hi, any updates or plans for this feature?