Fluentd: suppressed same stacktrace

[Raghu-rajanna]

I am getting suppressed the Same stack race unable to parse error in logs. I am using fluentd to transfer ossec logs that are in JSON format to Influxdb.

This is my configuration file td-agent.conf

<source>
 @type tail
 <parse>
 @type json
 </parse>
 path /var/ossec/logs/alerts/alerts.json
 pos_file /var/log/td-agent/ossec_log_json.pos
 tag ossec.process
</source>
<match ossec.process*>
  @type influxdb
  host  localhost
  port  8086
  dbname testdb
  user  testdb
  password  XXXXX
  <buffer>
  @type file
  path /var/ossec/logs/alerts
  flush_mode interval
  chunk_limit_size 524288 # 512 * 1024#
  chunk_limit_records 1024
  flush_interval 5s
  retry_wait 5s
  </buffer>
  tag_keys ["level","comment","sidid","location","full_log"]
</match>

THEASE ARE MY OSSEC LOGS

{"rule":{"level":3,"comment":"Login session opened.","sidid":5501},"location":"/var/log/auth.log","full_log":"Jul 13 00:58:56 kali sudo: pam_unix(sudo:session): session opened for user root by (uid=0)"}
{"rule":{"level":3,"comment":"Login session closed.","sidid":5502},"location":"/var/log/auth.log","full_log":"Jul 13 01:05:16 kali sudo: pam_unix(sudo:session): session closed for user root"}

THEASE ARE MY TD-AGENT LOGS

2019-07-13 00:34:03 -0400 [info]: #0 following tail of /var/ossec/logs/alerts/alerts.json
2019-07-13 00:34:03 -0400 [info]: #0 fluentd worker is now running worker=0
2019-07-13 00:34:03 -0400 [debug]: #0 flush_thread actually running
2019-07-13 00:34:03 -0400 [debug]: #0 enqueue_thread actually running
2019-07-13 00:38:37 -0400 [debug]: #0 Created new chunk chunk_id="58d8899b123ab49d050c8a67f32bfa13" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag="ossec.process", variables=nil>
2019-07-13 00:38:42 -0400 [debug]: #0 taking back chunk for errors. chunk="58d8899b123ab49d050c8a67f32bfa13"
2019-07-13 00:38:42 -0400 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2019-07-13 00:38:47 -0400 chunk="58d8899b123ab49d050c8a67f32bfa13" error_class=InfluxDB::Error error="{\"error\":\"unable to parse 'ossec.process,location=/var/log/syslog,full_log=Jul\\\\ 13\\\\ 00:38:36\\\\ kali\\\\ grafana-server[5404]:\\\\ 2019/07/13\\\\ 00:38:36\\\\ http:\\\\ proxy\\\\ error:\\\\ unsupported\\\\ protocol\\\\ scheme\\\\ \\\"\\\" rule={\\\"level\\\"=\\u003e2, \\\"comment\\\"=\\u003e\\\"Unknown problem somewhere in the system.\\\", \\\"sidid\\\"=\\u003e1002} 1562992717': invalid boolean\"}\n"
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/influxdb-0.7.0/lib/influxdb/client/http.rb:86:in `resolve_error'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/influxdb-0.7.0/lib/influxdb/client/http.rb:35:in `block in post'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/influxdb-0.7.0/lib/influxdb/client/http.rb:53:in `connect_with_retry'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/influxdb-0.7.0/lib/influxdb/client/http.rb:26:in `post'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/influxdb-0.7.0/lib/influxdb/query/core.rb:83:in `write'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/influxdb-0.7.0/lib/influxdb/query/core.rb:62:in `write_points'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-influxdb-2.0.0/lib/fluent/plugin/out_influxdb.rb:200:in `write'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/output.rb:1123:in `try_flush'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/output.rb:1423:in `flush_thread_run'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin/output.rb:452:in `block (2 levels) in start'
  2019-07-13 00:38:42 -0400 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-07-13 00:38:47 -0400 [debug]: #0 taking back chunk for errors. chunk="58d8899b123ab49d050c8a67f32bfa13"
2019-07-13 00:38:47 -0400 [warn]: #0 failed to flush the buffer. retry_time=1 next_retry_seconds=2019-07-13 00:38:52 -0400 chunk="58d8899b123ab49d050c8a67f32bfa13" error_class=InfluxDB::Error error="{\"error\":\"unable to parse 'ossec.process,location=/var/log/syslog,full_log=Jul\\\\ 13\\\\ 00:38:36\\\\ kali\\\\ grafana-server[5404]:\\\\ 2019/07/13\\\\ 00:38:36\\\\ http:\\\\ proxy\\\\ error:\\\\ unsupported\\\\ protocol\\\\ scheme\\\\ \\\"\\\" rule={\\\"level\\\"=\\u003e2, \\\"comment\\\"=\\u003e\\\"Unknown problem somewhere in the system.\\\", \\\"sidid\\\"=\\u003e1002} 1562992717': invalid boolean\"}\n"
  2019-07-13 00:38:47 -0400 [warn]: #0 suppressed same stacktrace
2019-07-13 00:38:49 -0400 [debug]: #0 Created new chunk chunk_id="58d889a686622a4613f259e65887c6ce" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag="ossec.process", variables=nil>
2019-07-13 00:38:52 -0400 [debug]: #0 taking back chunk for errors. chunk="58d8899b123ab49d050c8a67f32bfa13"
2019-07-13 00:38:52 -0400 [warn]: #0 failed to flush the buffer. retry_time=2 next_retry_seconds=2019-07-13 00:39:01 -0400 chunk="58d8899b123ab49d050c8a67f32bfa13" error_class=InfluxDB::Error error="{\"error\":\"unable to parse 'ossec.process,location=/var/log/syslog,full_log=Jul\\\\ 13\\\\ 00:38:36\\\\ kali\\\\ grafana-server[5404]:\\\\ 2019/07/13\\\\ 00:38:36\\\\ http:\\\\ proxy\\\\ error:\\\\ unsupported\\\\ protocol\\\\ scheme\\\\ \\\"\\\" rule={\\\"level\\\"=\\u003e2, \\\"comment\\\"=\\u003e\\\"Unknown problem somewhere in the system.\\\", \\\"sidid\\\"=\\u003e1002} 1562992717': invalid boolean\"}\n"
  2019-07-13 00:38:52 -0400 [warn]: #0 suppressed same stacktrace
2019-07-13 00:39:01 -0400 [debug]: #0 taking back chunk for errors. chunk="58d8899b123ab49d050c8a67f32bfa13"
2019-07-13 00:39:01 -0400 [warn]: #0 failed to flush the buffer. retry_time=3 next_retry_seconds=2019-07-13 00:39:20 -0400 chunk="58d8899b123ab49d050c8a67f32bfa13" error_class=InfluxDB::Error error="{\"error\":\"unable to parse 'ossec.process,location=/var/log/syslog,full_log=Jul\\\\ 13\\\\ 00:38:36\\\\ kali\\\\ grafana-server[5404]:\\\\ 2019/07/13\\\\ 00:38:36\\\\ http:\\\\ proxy\\\\ error:\\\\ unsupported\\\\ protocol\\\\ scheme\\\\ \\\"\\\" rule={\\\"level\\\"=\\u003e2, \\\"comment\\\"=\\u003e\\\"Unknown problem somewhere in the system.\\\", \\\"sidid\\\"=\\u003e1002} 1562992717': invalid boolean\"}\n"
  2019-07-13 00:39:01 -0400 [warn]: #0 suppressed same stacktrace
2019-07-13 00:39:20 -0400 [debug]: #0 taking back chunk for errors. chunk="58d8899b123ab49d050c8a67f32bfa13"
2019-07-13 00:39:20 -0400 [warn]: #0 failed to flush the buffer. retry_time=4 next_retry_seconds=2019-07-13 00:40:02 -0400 chunk="58d8899b123ab49d050c8a67f32bfa13" error_class=InfluxDB::Error error="{\"error\":\"unable to parse 'ossec.process,location=/var/log/syslog,full_log=Jul\\\\ 13\\\\ 00:38:36\\\\ kali\\\\ grafana-server[5404]:\\\\ 2019/07/13\\\\ 00:38:36\\\\ http:\\\\ proxy\\\\ error:\\\\ unsupported\\\\ protocol\\\\ scheme\\\\ \\\"\\\" rule={\\\"level\\\"=\\u003e2, \\\"comment\\\"=\\u003e\\\"Unknown problem somewhere in the system.\\\", \\\"sidid\\\"=\\u003e1002} 1562992717': invalid boolean\"}\n"
  2019-07-13 00:39:20 -0400 [warn]: #0 suppressed same stacktrace
2019-07-13 00:40:02 -0400 [debug]: #0 taking back chunk for errors. chunk="58d8899b123ab49d050c8a67f32bfa13"
2019-07-13 00:40:02 -0400 [warn]: #0 failed to flush the buffer. retry_time=5 next_retry_seconds=2019-07-13 00:41:18 -0400 chunk="58d8899b123ab49d050c8a67f32bfa13" error_class=InfluxDB::Error error="{\"error\":\"unable to parse 'ossec.process,location=/var/log/syslog,full_log=Jul\\\\ 13\\\\ 00:38:36\\\\ kali\\\\ grafana-server[5404]:\\\\ 2019/07/13\\\\ 00:38:36\\\\ http:\\\\ proxy\\\\ error:\\\\ unsupported\\\\ protocol\\\\ scheme\\\\ \\\"\\\" rule={\\\"level\\\"=\\u003e2, \\\"comment\\\"=\\u003e\\\"Unknown problem somewhere in the system.\\\", \\\"sidid\\\"=\\u003e1002} 1562992717': invalid boolean\"}\n"
  2019-07-13 00:40:02 -0400 [warn]: #0 suppressed same stacktrace
2019-07-13 00:41:18 -0400 [debug]: #0 taking back chunk for errors. chunk="58d8899b123ab49d050c8a67f32bfa13"
2019-07-13 00:41:18 -0400 [warn]: #0 failed to flush the buffer. retry_time=6 next_retry_seconds=2019-07-13 00:43:50 -0400 chunk="58d8899b123ab49d050c8a67f32bfa13" error_class=InfluxDB::Error error="{\"error\":\"unable to parse 'ossec.process,location=/var/log/syslog,full_log=Jul\\\\ 13\\\\ 00:38:36\\\\ kali\\\\ grafana-server[5404]:\\\\ 2019/07/13\\\\ 00:38:36\\\\ http:\\\\ proxy\\\\ error:\\\\ unsupported\\\\ protocol\\\\ scheme\\\\ \\\"\\\" rule={\\\"level\\\"=\\u003e2, \\\"comment\\\"=\\u003e\\\"Unknown problem somewhere in the system.\\\", \\\"sidid\\\"=\\u003e1002} 1562992717': invalid boolean\"}\n"
  2019-07-13 00:41:18 -0400 [warn]: #0 suppressed same stacktrace

[ganmacs]

this is not a bug. expected behavior. https://github.com/fluent/fluentd/blob/9dcf9488c3df269db37917ff09b7b91860d46fac/lib/fluent/log.rb#L415-L450

[tirelibirefe]

ok what is the solution of expected behavior?