Fluentd: Multiple Outputs Possible ?

[geek876]

Hi,

Is it possible to emit same event twice ? My use case is below:

All Clients forward their events to a central Fluentd-Server (which is simply running td-agent). This Central Server outputs the events as per the tags.

Sample event1: { field1:x, field2:y, tag:elasticsearch, output:elasticsearch }
Sample event2: { field1:p, field2:y, tag:file, output:file          }

On the Clients (All events forwarded to fluentd-server):

<match **.*>
  type forward
  heartbeat_type tcp
   <server>
    host fluentd-server
    port 24224
   </server>
</match>

On Fluentd-Server (Events output as per tag):

<match elasticsearch>
    type "aws-elasticsearch-service"
    include_tag_key true
    tag_key tag
    logstash_format true
    flush_interval 10s
    <endpoint>
      url https://xxxx
      region xxxx
    </endpoint>
</match>

<match file>
    type file
    path /some/path
</match>

The above works fine if I want to send events to only 'elasticsearch' or 'file'. However, If I want to send certain events to both, I am not sure how I achieve it.

I tried using the rewrite_tag_output filter on Fluentd-Server as below (after tagging such events with a combined tag

Sample event3: { filed1:x, filed2:y, tag:elasticsearchfile, output:elasticsearchfile }

On Fluentd-Server

<match elasticsearchfile>
  @type rewrite_tag_filter
  rewriterule1 output elasticsearch elasticsearch
  rewriterule2 output file  file
</match>

<match elasticsearch>
    type "aws-elasticsearch-service"
    include_tag_key true
    tag_key tag
    logstash_format true
    flush_interval 10s
    <endpoint>
      url https://xxxx
      region xxxx
    </endpoint>
</match>

<match file>
    type file
    path /some/path
</match>

I see that the events are filtered and goes to elasticsearch but not to file. Basically the first rewriterule1 is getting applied so was wondering if there is a way of sending output to multiple locations.

I am aware of the copy plugin but for me to use that, I would need to use it on the clients and hence will require to install the various output-plugins like elasticsearch-aws on the clients which I want to refrain from.

[repeatedly]

How about using following configuration?

<match elasticsearch file>
  @type copy
  <store>
     @type elasticsearch
  </store>
  <store>
     @type file
  </store>
</match>

[geek876]

Thanks. However, If I understand it correctly, this will match tags either of elasticsearch or file and events will end up at both locations even if tag is elasticsearch or file. I want events to go to elasticsearch ONLY if tag is elasticsearch and to file ONLY if tag is file. However, if tag is elasticsearchfile, it should go to both and I want to avoid using the copy plugin if possible.

In other words, I want to control the 'output' routing for events dynamically rather than having <match> blocks per output destination and want to drive it entirely via tags and/or custom field (ex: output).

[geek876]

I ended up doing it as below, which works !

<match **.file1,file2>
  @type copy
  <store>
    @type rewrite_tag_filter
    rewriterule1 output file1 file1
  </store>
  <store>
    @type rewrite_tag_filter
    rewriterule1 output file2 file2
  </store>
</match>

<match file1>
  @type file
  path /tmp/file1
</match>

<match file2>
  @type file
  path /tmp/file2
</match>

[repeatedly]

Good.
BTW, this issue is for bug-report or feature request.
If you have a question, use mailing list instead: https://github.com/fluent/fluentd/blob/master/CONTRIBUTING.md#got-a-question-or-problem

[dobesv]

It seems like you can use copy and relabel to copy an event and forward to labelled pipelines.

[Anindya-Kar]

I have similar requirement. However, Central Server outputs the events as per following tags.

Sample event1: { field1:x, field2:y, tag:["value1","value2", "elasticsearch" ,"file"}

How to send events to only 'elasticsearch' or 'file' or both events? Please advice.