Fluent-bit: Networking with KeepAlive support

[edsiper]

In the current design of Fluent Bit, when plugins perform I/O, the backend interface _upstream_ and _io_ always creates a new TCP connection. This used to be enough for previous use cases where users used to flush data after long periods of aggregation.

Nowadays Fluent Bit is used heavily on data transmission and users are flushing data on average every second, in order to optimize data transmission for this scenario and reduce CPU load when the connection involves TLS/SSL handshakes, we need to provide __optional keepalive support__.

The following options are being proposed from a design perspective:

• keepalive: boolean, False by default.
• keepalive_timeout: maximum period of time in seconds where a keepalive connection can be _alive_ (despite the number of recycles).

The interface will be agnostic for plugins and handled directly by the engine. The upstream API will care of connection recycles if required.

[floriankammermann]

Is this issue still open, since I see that keepalive decisions are in the code?

Got the answer myself, will be in release 1.4.0:

We are implementing Keepalive support for 1.4 release, you can track it progress here.

https://github.com/fluent/fluent-bit/issues/573

[alkar]

Is this implemented or is there more work to be done?

I can see in the release notes that it has and from experimenting with 1.4.0 it does seem to have some effect when enabled but the change is not reflected in docs and this ticket is still open.

[edsiper]

1.4 -> implemented through "keepalive" option. It has not been promoted too much since we realized that we needed other areas of improvements like timeouts, so on 1.5 we are implementing new networking options with different names.

You can use keepalive true in 1.4

If you are interested in to know what's coming on 1.5 about this check the dev-docs here:

https://docs.fluentbit.io/manual/v/master/administration/networking

[edsiper]

This is already implemented and merged on 1.5.

Closing.