Fluentd: [fluentd 0.14.17] plugin:Parser ignores "suppress_parse_error_log true"

[zette]

Using fluetnd:0.14.17 in docker container based on Alpine Linux.

<system>                                                                                                                                                                                                   
  log_level info                                                                                                   
</system> 

<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

<filter **>                                                                 
    @type parser                                                              
    format /^<.*>(?<syslog_time>\w+\s*\d+\s*\S+) (?<host>\S+) (?<process>\S+\[\d+\]): (?<c_ip>[\w\.]+):(?<c_port>\d
    key_name log                                                                                                   
    reserve_data true                                                                                              
    suppress_parse_error_log true                                                                                  
    ignore_key_not_exist true                                                                                      
    time_parse false                                                                                               
    @label @DONE                                                                                                   
</filter>

<label @DONE>                                                                                                      
  <match dev.docker.**>                                                                                            
        @type elasticsearch                                                                                        
        [...]
  </match>
</label>

Even the suppress_parse_error_log is set to true, I get a WARN in log each time filter does not hit pattern.

2017-07-05 10:05:44 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not match with data '\e[0m\e[0m2017-07-05 12:05:27,874 |sched-worker-7| INFO [xxx] - Synchronization job started beanName = [singleSendOrdersJobRunner]!\n'" tag="qa.docker.383a0473b962745961d2830159338c0b364f9c13ce65b72fb16b1b283d0f5344.383a0473b962745961d2830159338c0b364f9c13ce65b72fb16b1b283d0f5344-json.log" time=1499249127 record={"log"=>"\e[0m\e[0m2017-07-05 12:05:27,874 |sched-worker-7| INFO [xxx] - Synchronization job started beanName = [singleSendOrdersJobRunner]!\n", "stream"=>"stdout", "time"=>"2017-07-05T10:05:27.875178118Z"}

Shouldn't it just ignore not matched lines, and not put everything to fluent log file?

[repeatedly]

With v0.14, filter_parser uses only @ERROR label instead of log message.
So if you want to disable it, please set emit_invalid_record_to_error false in filter configuration.

We are not sure v0.14's parser filter should support v0.12 log message and same parameters.

[zette]

That is fine.
But working with stream of multi format logs from one source (ie. docker containers forwarded file logs) will bring complexity to config file.
I could handle it with multiple parse filters (as one shown) easily, if only I could disable emiting warnings for each false-match.
In fact, the following example, works as I've described - trying to match one by one filer, only raising unnecessary WARN entries in log:

<system>                                                                                                                                                                                                   
  log_level info                                                                                                   
</system> 

<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

<filter **>                                                                 
    @type parser                                                              
    format /first_regexp/
    key_name log                                                                                                   
    reserve_data true                                                                                              
    suppress_parse_error_log true                                                                                  
    ignore_key_not_exist true                                                                                      
    time_parse false                                                                                               
    @label @DONE                                                                                                   
</filter>


<filter **>                                                                 
    @type parser                                                              
    format /second_regexp/
    key_name log                                                                                                   
    reserve_data true                                                                                              
    suppress_parse_error_log true                                                                                  
    ignore_key_not_exist true                                                                                      
    time_parse false                                                                                               
    @label @DONE                                                                                                   
</filter>

...

<filter **>                                                                 
    @type parser                                                              
    format none #final catch group for all unparsed
    key_name log                                                                                                   
    reserve_data true                                                                                              
    suppress_parse_error_log                                                                                   
    ignore_key_not_exist true                                                                                      
    time_parse false                                                                                               
    @label @OTHER                                                  
</filter>

<label @DONE>                                                                                                      
  <match dev.docker.**>                                                                                            
        @type elasticsearch                                                                                        
        [...]
  </match>
</label>

It would help if WARN logging could be disabled selectively for filters.

[nlundbo]

I second this!

[teamcitytest3]

are there some workarounds or solutions?

[repeatedly]

@teamcitytest3 Set emit_invalid_record_to_error false is not fit for your case?

# v0.12
emit_invalid_record_to_error false
suppress_parse_error_log true                                                                                  
ignore_key_not_exist true   

is same as

# v1.0
emit_invalid_record_to_error false

[vishnuvg1984]

With regard to above comment by "Massahiro " dated May 21st ,

Unifying the three keys from v012 to just one "emit_invalid_record_to_error false" cause a lot of problems. Now we are not able to stop the error logging for "invalid_record_parse_error".

In newer version if set the above parameter to true, that is causing lengthy logs, and if there are too many invalid parse records ,within a few days the log folder will be out of space. There is no flexibility in newer version to stop logging the lenghty "parse error" logs, while keeping the emit invalid records as it is.

We need a solution, that we want to emit_invalid_records_to_error=true, but same time logging parse_error should be "false". Right now its not possible with latest version. Is there any alternative solutions. ?

[brendanly]

 # Quiet down the excessive parser warnings
 <system>
   log_level error
 </system>

This worked for me. Doing the global approach to avoid chatty fluentd logs, but similar can work on a per-plugin config.

Source: https://docs.fluentd.org/deployment/logging

[ganmacs]

seems to be solved. closed

[Evreke]

Seems that i face the same problem. I send my logs after parse filter to ELK but there is a lot of logs that don't meet parsers regexp. I want to store only sucessfuly parsed logs in ELK but not that not match my regexp.