Fluentd: warning: Section <format> is not used in <match **> of out_file plugin

Created on 23 Oct 2017 · 2Comments · Source: fluent/fluentd

Hi!

I'm using latest td-agent from yum repository on centos 7 x86_64

plugin version:

# td-agent-gem list |grep -i s3
fluent-plugin-s3 (0.8.6)

I have the following config of s3 plugin.

<label @ECS_LOGGING_TEST_APACHE_S3>
  <match **>
    @type s3

    aws_key_id KEYID
    aws_sec_key KEY
    s3_bucket bucket.name
    s3_region ap-southeast-1

    path logs/logging-test/apache/%Y/%m/%d/
    s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}

    <buffer tag,time>
      @type file
      path /var/log/fluentd/s3/logging-test/apache/
      timekey 60
      timekey_wait 1m
      timekey_use_utc true # use utc
    </buffer>
    <format>
      @type json
    </format>
  </match>
</label>

When I start fluentd I get:

2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2017-10-22 23:01:50 +0000 [debug]: fluent/log.rb:296:call: adding store type="relabel"
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: using configuration file: <ROOT>
  <source>
    @type forward
    bind "0.0.0.0"
    port 8889
  </source>
  <match ecs.logging-test-apache>
    @type copy
    <store>
      @type "relabel"
      @label @ECS_LOGGING_TEST_APACHE_S3
    </store>
  </match>
  <label @ECS_LOGGING_TEST_APACHE_S3>
    <match **>
      @type s3
      aws_key_id xxxxxx
      aws_sec_key xxxxxx
      s3_bucket "bucket.name"
      s3_region "ap-southeast-1"
      path "logs/logging-test/apache/%Y/%m/%d/"
      s3_object_key_format "%{path}%{time_slice}_%{index}.%{file_extension}"
      <buffer tag,time>
        @type "file"
        path "/var/log/fluentd/s3/logging-test/apache/"
        timekey 60
        timekey_wait 1m
        timekey_use_utc true
      </buffer>
      <format>
        @type json
      </format>
    </match>
  </label>
</ROOT>
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: starting fluentd-0.14.21 pid=29818
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "-c", "/etc/td-agent/td-agent.conf", "-v", "--under-supervisor"]
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-elasticsearch' version '1.10.1'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-grok-parser' version '2.1.4'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-kafka' version '0.6.1'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-logzio' version '0.0.13'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-mongo' version '0.8.1'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.6'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-s3' version '0.8.6'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-scribe' version '0.10.14'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-td' version '0.10.29'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-td-monitoring' version '0.2.3'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluent-plugin-webhdfs' version '0.7.1'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluentd' version '0.14.21'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: gem 'fluentd' version '0.12.40'
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: adding match in @ECS_LOGGING_TEST_APACHE_S3 pattern="**" type="s3"
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: adding match pattern="ecs.logging-test-apache" type="copy"
2017-10-22 23:01:50 +0000 [debug]: #0 fluent/log.rb:296:call: adding store type="relabel"
2017-10-22 23:01:50 +0000 [info]: fluent/log.rb:316:call: adding source type="forward"
2017-10-22 23:01:50 +0000 [warn]: fluent/log.rb:336:call: section <format> is not used in <match **> of out_file plugin
2017-10-22 23:01:50 +0000 [info]: #0 fluent/log.rb:316:call: starting fluentd worker pid=29822 ppid=29818 worker=0
2017-10-22 23:01:50 +0000 [debug]: #0 fluent/log.rb:296:call: buffer started instance=17269440 stage_size=0 queue_size=0
2017-10-22 23:01:50 +0000 [info]: #0 fluent/log.rb:316:call: listening port port=8889 bind="0.0.0.0"
2017-10-22 23:01:50 +0000 [info]: #0 fluent/log.rb:316:call: fluentd worker is now running worker=0
2017-10-22 23:01:50 +0000 [debug]: #0 fluent/log.rb:296:call: enqueue_thread actually running
2017-10-22 23:01:50 +0000 [debug]: #0 fluent/log.rb:296:call: flush_thread actually running

No matter what format parameters I provide it doesn't seem to work.
I continue getting warning: Section <format> is not used in <match **> of out_file plugin message in logs.

Can you please help me investigate the issue?

Thanks!

Regards,
Sergey

Source