Fluent-bit: Flushing buffered data at shutdown

[musaprg]

Is your feature request related to a problem? Please describe.
I'm trying to replace fluentd to fluent-bit for reducing container size, but fluent-bit seem not to flush data in buffer when shutting down. Is there any feature like flush_at_shutdown in fluent-bit?

Describe the solution you'd like
Describe alternatives you've considered
Additional context

[edsiper]

are you using the storage file system type ?, please share your full config

[musaprg]

No. I just use the default buffering configuration (on memory).

My configuration file is below. I try to receive log data from fluent-logger by fluent-bit sidecar container and forward it to fluentd container.

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    info

[INPUT]
    Name    forward
    Listen  0.0.0.0
    Port    24224
    Buffer_Chunk_Size 32KB
    Buffer_Max_Size   64KB

[OUTPUT]
    Name  forward
    Match *
    Host  fluentd

FYI: my docker-compose.yml is below

version: '3'
services:
  app:
    build: .
  fluent-bit:
    image: fluent/fluent-bit:1.2
    volumes:
    - .:/fluent-bit/etc
  fluentd:
    image: fluent/fluentd:v1.6
    volumes:
    - .:/fluentd/etc

[edsiper]

Fluent Bit does a graceful shutdown upon SIGTERM, e.g:

kill `pidof fluent-bit`

The grace period by default is 5 seconds, since you are using the default and your flush time is the same likely something is pending, I would suggest to set Grace 10 in your Service section.

[musaprg]

Is there any guarantee to flush all data in the buffer, and any new data won't come into the buffer after starting graceful shutdown? I think fluent-bit should flush data in buffer immediately and block new input after it receives shutdown signals.

[edsiper]

@musaprg hmm looking at the code found that input plugins (ingestion) will not stop until the engine shutdowns completely. So the ideal behavior will be: upon shutdown request, stop ingestion, and flush buffers during the grace period.

Something to fix.

[musaprg]

Yes, I think so too.

[musaprg]

@edsiper I'm sorry, I checked codes again and I found that fluent-bit terminated input plugin when it caught SIGTERM signals.

https://github.com/fluent/fluent-bit/commit/ea09c764ccce76cf7ce4d117f9dc0e50d505b335

[musaprg]

but I think it is better to flush current buffer forcibly than wait until next flushing timing.

[edsiper]

Fixed in f303cdb5c5c9d5a7d98d824878551d916441a95a.

Binary now uses library mode so it can do graceful shutdown (in the grace period)