Fluent-bit: the ignore_older of tail config is of no effect

[yubobo]

but，in fact，add all files to scan the queue whenever the file is generated.

[yubobo]

[yubobo]

[yubobo]

I change the value of ignore_older to 1m ,the problem is the same.

[edsiper]

The ignore_older option is only available when a Parser is specificied and that parser matched a line, so when the parser lookup the time of that specific line/record it checks if is older than ignore_older or not. It does not apply to the file age, it's record based.

[acarsercan]

Someone be kind enough to paste an example config with ignore_older & parser

Thanks

[avdhoot]

@acarsercan Might late. Set parser=myparser in input plug-in tail. tail plugin will use Time_Key & Time_Format find out correct time to compare.

[PARSER]                                                                                                                                                                
    Name         myparser                                                                                                                                                 
    Format       json                                                                                                                                                   
    Time_Key     time                                                                                                                                                   
    Time_Format  %Y-%m-%dT%H:%M:%S.%L                                                                                                                                   
    Time_Keep    On
[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Tag              containers.*
    Refresh_Interval 1
    Rotate_Wait      60
    Mem_Buf_Limit    5MB
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state-sumo.db
    DB.Sync          Normal
    Ignore_Older 1m
    Parser  myparser