Fluent-bit: Feature request for using tag varialbe to route output messages

[AkashKrDutta]

Would we able to use the tag variable in the input section of fluent-bit to route output messages as needed. Example in case of kafka outputs, we create topic names like %{tag}.stream as the topic name? Mainly asking for a provision to use message fields as variables to use in configuration.

[nokute78]

How about such combination?

1. Append tag to record using filter_lua plugin.
2. Use Topic_Key to indicate tag which is appended 1.

We can append tag to each record using this script.

function add_tag(tag, timestamp, record)
    new_record = record
    new_record["tag"] = tag
    return 1, timestamp, new_record
end

It is a result to append tag. In this case, it is "tag"->"mem.0".

$ bin/fluent-bit -i mem  -F lua -m '*' -p script=add_tag.lua -p call=add_tag -o stdout
Fluent-Bit v0.14.0
Copyright (C) Treasure Data

[2018/06/14 22:42:11] [ info] [engine] started (pid=9490)
[0] mem.0: [1528983732.000727891, {"Mem.free"=>547284, "Swap.free"=>2064380, "tag"=>"mem.0", "Swap.used"=>0, "Mem.used"=>3372884, "Mem.total"=>3920168, "Swap.total"=>2064380}]

I think "Topic_Key mem.0" in this case, out_kafka can recognize tag as a topic name.

[AkashKrDutta]

Only if the 0.14 version of fluent-bit comes anytime soon!

[AkashKrDutta]

@nokute78 Is there a dev image that I can use currently to test this out? (20th july is a long time waiting for the release)

[michiel]

@AkashKrDutta check this comment on a different issue for a recent unofficial dev build - https://github.com/fluent/fluent-bit/issues/611#issuecomment-396929245

[AkashKrDutta]

Thank you for this! :smile:

[AkashKrDutta]

@michiel is there any way I can automate the pipeline in a way so that I don't need to update the topics list each time a new input tag is created? (It kills the whole idea right? I want to route logs dynamically based on a cetain key value that may be already present or I add manually)
I need full automation in the input side. Based on a tag key value in the log record, the topic is decided, without the need to update any other thing in the config. Is this possible?

[AkashKrDutta]

Feature request please.