Fluent-bit: How to parse data as a collection of key/value pairs

[lktslionel]

Hi Guys,
I'm a little new to fluent-bit and I want to use the filter "parser" to parse a key on a log line.

Given :

A log line

[0] os.component/abba/3594881dacb8: [1511196039.000000000, {"container_name"=>"/abba", "source"=>"stdout", "log"=>"time="2017-11-20 17:40:39" || component="EX--ABBA" || logger="/os/app/src/index.js" || type="TECHNICAL" || msg="/health endpoint ca", "container_id"=>"3594881dacb8d9d3317a6c9fc6aff2ec627ee43037fec5e609e22c6cb9cf93b7"}]

And

A parser filter with the key_name I want to parse "log"

[FILTER]
    Name parser
    Key_Name log
    Parser os_parser

And

A Parser file with content

[PARSER]
    Name os_parser
    Format regex
    Regex <I_DONT_KNOW_THE_RIGHT_REGEX>

When

I run fluent-bit with the configuration given above

Then

I should get the following output

[3] os.component: [1511195416.002497117, {"time"=>"2017-11-20 17:22:32", "component" => "EX--ABBA", "logger" => "/os/app/src/index.js", "type"=>"TECHNICAL", "msg"=>"/health endpoint ca" }]

Do you have any idea of which regex () I could use to achieve that result ?

Thanks in advance

[lktslionel]

I could contribute by creating a parser function in Go that aims to parser key/val pairs with a given separator; but I don't know how to create one not in C but in Go instead.
Thanks

[nokute78]

regexp

fluentbit supports named capture.
It can define the key name.

e.g (?<key_name>[^ ]*)

could you check this document?
http://fluentbit.io/documentation/current/parser/regular_expression.html

http://rubular.com/
is helpful to test regexp.

[lktslionel]

I agree but I need to capture key_name/key_value without having to specify the key_name by its position in the log line.
The behavior I need is something similar to logstash kv plugin

Thanks

[lktslionel]

Hello, @edsiper
Does your commit solves my issue ?
Thanks

[dvlato]

Hi, We also need to have a parser that allows us to extract key=value pairs without a need to defining the name and order of each field... Is there anything like that? Tons of our applications would break with this restriction!

[hernandesbsousa]

I came across this issue while checking the same for myself and it seems to me that the logfmt parser is what you're looking for. Please notice I still haven't tested it, but the docs seem to indicate that it's the way to go

[lktslionel]

Thanks @hernandesbsousa ! I will try this one a let you guys, know :-)

[edsiper]

thanks everyone.

ref: https://docs.fluentbit.io/manual/parser/logfmt