Fluentd: Using match to exclude fluentd logs not working

[dcvtruong]

Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.

Describe the bug
Using to exclude fluentd logs but still getting fluentd logs regularly

To Reproduce

    <match kubernetes.var.log.containers.fluentd-fluentd-elasticsearch**>
      @type null
    </match>

Kubernetes tag,

kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-f5t8p_logging_fluentd-fluentd-elasticsearch-f938c78d6647cdaa9b3f575df5e8501fb11a8d2d28d0ecbf39bb514849aa1333.log

Expected behavior
Not see any fluentd logs

Your Environment

• Fluentd 1.6.3
• Operating system: RHEL 7.6
• Kernel version: 3.10.0-957.35.2.el7.x86_64

If you hit the problem with older fluentd version, try latest version first.

Your Configuration

    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/goapp-log.pos
      tag kubernetes.*
      read_from_head true
      format json
      time_format %Y-%m-%dT%H:%M:%SZ
      time_key @time
      @log_level debug
    </source>

    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <filter kubernetes.**>
      @type parser
      <parse>
        @type json
        json_parser json
        time_format %Y-%m-%dT%H:%M:%S.%NZ
      </parse>
      hash_value_field parsed
      replace_invalid_sequence true
      emit_invalid_record_to_error false
      key_name log
      reserve_data true
    </filter>

    <filter kubernetes.**>
      @type grep
      <exclude>
        key log
        pattern debug
      </exclude>
    </filter>

    <match **>
      @id elasticsearch
      @type elasticsearch
      @log_level debug
      include_tag_key true
      type_name _doc
      host "#{ENV['OUTPUT_HOST']}"
      port "#{ENV['OUTPUT_PORT']}"
      scheme "#{ENV['OUTPUT_SCHEME']}"
      ssl_version "#{ENV['OUTPUT_SSL_VERSION']}"
      logstash_format true
      logstash_prefix "#{ENV['LOGSTASH_PREFIX']}"
      reconnect_on_error true
      reload_on_failure true
      reload_connections false
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 10s
        retry_forever
        retry_max_interval 30
        chunk_limit_size "#{ENV['OUTPUT_BUFFER_CHUNK_LIMIT']}"
        queue_limit_length "#{ENV['OUTPUT_BUFFER_QUEUE_LIMIT']}"
        overflow_action block
      </buffer>
    </match>

    <match kubernetes.var.log.containers.fluentd-fluentd-elasticsearch**>
      @type null
    </match>

Your Error Log
Still getting the fluentd log every minute.

2019-10-25 16:41:31 +0000 [debug]: [elasticsearch] Created new chunk chunk_id="595bed27c1da14c173988d2f4db79e61" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag="kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-f5t8p_logging_fluentd-fluentd-elasticsearch-f938c78d6647cdaa9b3f575df5e8501fb11a8d2d28d0ecbf39bb514849aa1333.log", variables=nil>
2019-10-25 16:42:31 +0000 [debug]: [elasticsearch] Created new chunk chunk_id="595bed60fa342d875b6c053465604526" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag="kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-f5t8p_logging_fluentd-fluentd-elasticsearch-f938c78d6647cdaa9b3f575df5e8501fb11a8d2d28d0ecbf39bb514849aa1333.log", variables=nil>
2019-10-25 16:43:31 +0000 [debug]: [elasticsearch] Created new chunk chunk_id="595bed9a32b9d21076332c7c7232770d" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag="kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-f5t8p_logging_fluentd-fluentd-elasticsearch-f938c78d6647cdaa9b3f575df5e8501fb11a8d2d28d0ecbf39bb514849aa1333.log", variables=nil>

Additional context

From fluentd docs,

* matches zero or more tag parts.
For example, the pattern a.* matches a, a.b and a.b.c

What am I missing?

[ganmacs]

The match feature works well as follows.
You look misunderstand in_tail's tag directive. see https://docs.fluentd.org/input/tail#tag .
BTW please see https://github.com/fluent/fluentd/blob/5e44debc99d0d4f2b59271fa8c7028704a68abfa/CONTRIBUTING.md#got-a-question-or-problem

2019-10-28 11:15:55 +0900 [info]: parsing config file is succeeded path="example/debug/tag.conf"
2019-10-28 11:15:55 +0900 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2019-10-28 11:15:55 +0900 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2019-10-28 11:15:55 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type dummy
    tag "kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-f5t8p_logging_fluentd-fluentd-elasticsearch-f938c78d6647cdaa9b3f575df5e8501fb11a8d2d28d0ecbf39bb514849aa1333.log"
  </source>
  <match kubernetes.var.log.containers.fluentd-fluentd-elasticsearch**>
    @type stdout
  </match>
p</ROOT>
2019-10-28 11:15:55 +0900 [info]: starting fluentd-1.7.3 pid=47497 ruby="2.6.3"
2019-10-28 11:15:55 +0900 [info]: spawn command to main:  cmdline=["/bin/ruby", "-Eascii-8bit:ascii-8bit", "-rbundler/setup", "/Users/yuta.iwama/src/github.com/fluent/fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd", "-c", "example/debug/tag.conf", "--under-supervisor"]
2019-10-28 11:15:56 +0900 [info]: gem 'fluentd' version '1.7.3'
2019-10-28 11:15:56 +0900 [info]: adding match pattern="kubernetes.var.log.containers.fluentd-fluentd-elasticsearch**" type="stdout"
2019-10-28 11:15:56 +0900 [info]: adding source type="dummy"
2019-10-28 11:15:56 +0900 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2019-10-28 11:15:56 +0900 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2019-10-28 11:15:56 +0900 [info]: #0 starting fluentd worker pid=47511 ppid=47497 worker=0
2019-10-28 11:15:56 +0900 [info]: #0 fluentd worker is now running worker=0
2019-10-28 11:15:57.021884000 +0900 kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-f5t8p_logging_fluentd-fluentd-elasticsearch-f938c78d6647cdaa9b3f575df5e8501fb11a8d2d28d0ecbf39bb514849aa1333.log: {"message":"dummy"}
2019-10-28 11:15:58.062220000 +0900 kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-f5t8p_logging_fluentd-fluentd-elasticsearch-f938c78d6647cdaa9b3f575df5e8501fb11a8d2d28d0ecbf39bb514849aa1333.log: {"message":"dummy"}
2019-10-28 11:15:59.091283000 +0900 kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-f5t8p_logging_fluentd-fluentd-elasticsearch-f938c78d6647cdaa9b3f575df5e8501fb11a8d2d28d0ecbf39bb514849aa1333.log: {"message":"dummy"}
^C2019-10-28 11:15:59 +0900 [info]: Received graceful stop
^P2019-10-28 11:16:00.014524000 +0900 kubernetes.var.log.containers.fluentd-fluentd-elasticsearch-f5t8p_logging_fluentd-fluentd-elasticsearch-f938c78d6647cdaa9b3f575df5e8501fb11a8d2d28d0ecbf39bb514849aa1333.log: {"message":"dummy"}
2019-10-28 11:16:00 +0900 [info]: #0 fluentd worker is now stopping worker=0
2019-10-28 11:16:00 +0900 [info]: #0 shutting down fluentd worker worker=0
2019-10-28 11:16:00 +0900 [info]: #0 shutting down input plugin type=:dummy plugin_id="object:3fdaecd47a2c"
2019-10-28 11:16:00 +0900 [info]: #0 shutting down output plugin type=:stdout plugin_id="object:3fdaec19d294"
2019-10-28 11:16:00 +0900 [info]: Worker 0 finished with status 0

[dcvtruong]

@ganmacs Not sure why you close this as your response is not the same to the issue. You mentioned 'misunderstand' please take a look at this fluentd docs for v1.0, https://docs.fluentd.org/output/null

out_null is included in Fluentd's core. No additional installation process is required.
<match pattern>
  @type null
</match>

Why do you thing stdout is relevant to the issue being raised?

  <match kubernetes.var.log.containers.fluentd-fluentd-elasticsearch**>
    @type stdout
  </match>

[ganmacs]

The reason I closed this issue is that you didn't follow our contribution guide. so, for our community, I closed this kind of issue. please check the contribution guide before posting issue.