Fluent-bit: Fluentbit tail missing some big-ish log line even with Buffer_Max_Size set to high value

Created on 23 Jan 2020 · 4Comments · Source: fluent/fluent-bit

Bug Report

Describe the bug
One of our app is logging to stdout and fluentbit is able to capture the stdout log from (docker) log path successfully and send onto Splunk which we can see fine.

However, we have noticed 1 type of log line is always missing from splunk, even though we can see if in the pod log.

The log line in question is a big-ish 1 line which made me look at the tuning of:

[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Parser           docker
    Tag              kube.*
    Docker_Mode      On
    Docker_Mode_Flush 4
    Refresh_Interval 5
    Mem_Buf_Limit    32MB
    Buffer_Max_Size  2048KB
    Skip_Long_Lines  On

And the log line is question, has this stats:

wc -l long-line.log
       1 long-line.log

wc -c long-line.log
   20904 long-line.log

du -sk long-line.log
24  long-line.log

so not particulary big in size so i assumed the Skip_Long_Lines On with Buffer_Max_Size set to 2048KB was fine.

I can't check if fluentbit is not capturing it or not sending it but would be awesome if i can find this out as i am pretty sure Splunk can handle this log line fine.

fluentbit log shows nothing but i guess thats because its not in debug which i can set to.

Source

kiich

👍1

Most helpful comment

Linking related issues as it now seems the issue is not the big log line but the fact the log file in question is both:

1) symlink to another file
2) That file gets rotated

And when that log file gets filled up super quickly, it seems to then cause this issue of lost log in splunk.

https://github.com/fluent/fluent-bit/issues/1712
https://github.com/fluent/fluent-bit/issues/1108
https://github.com/fluent/fluent-bit/issues/375
https://github.com/fluent/fluent-bit/issues/1118

kiich on 29 Jan 2020

🚀1 🎉1 👍1

All 4 comments

Also to add - once this problematic log line appears in the log, we don't see any subsequent log lines neither in splunk - which seems to match the behaviour of not setting Skip_Long_Lines to true - but we are in this case?

kiich on 23 Jan 2020

Using Fluent Bit v1.2.2