Fluent-bit: [plugin][out_gelf] error [flb_msgpack_to_gelf] missing short_message key

Created on 18 Dec 2018 · 1Comment · Source: fluent/fluent-bit

Bug Report

Describe the bug
Gelf output plugin reports [flb_msgpack_to_gelf] missing short_message key.

To Reproduce
$ vi test.conf

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf
    Plugins_File plugins.conf
    HTTP_Server  Off
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

[INPUT]
    Name cpu
    Tag  cpu.local
    Interval_Sec 1

[OUTPUT]
    Name stdout
    Match *

[OUTPUT]
    Name  gelf
    Match *
    Host  gray_log_server_ip
    Port  12201

Run fluent-bit:latest docker image(v1.1.0)

$ docker run -it -p 24224:24224 -v $(pwd)/test.conf:/fluent-bit/etc/fluent-bit.conf -v $(pwd):/test fluent/fluent-bit:latest
...
Fluent Bit v1.1.0
Copyright (C) Treasure Data

[2018/12/18 17:21:37] [debug] [storage] [cio stream] new stream registered: cpu.0
[2018/12/18 17:21:37] [ info] [storage] initializing...
[2018/12/18 17:21:37] [ info] [storage] in-memory
[2018/12/18 17:21:37] [ info] [storage] normal synchronization mode, checksum disabled
[2018/12/18 17:21:37] [ info] [engine] started (pid=1)
[2018/12/18 17:21:37] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2018/12/18 17:21:37] [debug] [router] match rule cpu.0:stdout.0
[2018/12/18 17:21:37] [debug] [router] match rule cpu.0:gelf.0
[2018/12/18 17:21:41] [debug] [task] created task=0x7f2636c3b0c0 id=0 OK
[0] cpu.local: [1545153698.000247909, {"cpu_p"=>4.000000, "user_p"=>3.000000, "system_p"=>1.000000, "cpu0.p_cpu"=>4.000000, "cpu0.p_user"=>3.000000, "cpu0.p_system"=>1.000000}]
...
[2018/12/18 17:21:41] [error] [flb_msgpack_to_gelf] missing short_message key
[2018/12/18 17:21:41] [error] [out_gelf] error encoding to GELF
...

Expected behavior
CPU status messages sent to GrayLog

Your Environment

Version used: Docker image(v1.1.0)
Configuration: See above conf file
Operating System and version: Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-42-generic x86_64)
Filters and plugins: Gelf output
Docker: Docker version 18.06.1-ce, build e68fc7a

Additional context
I couldn't find any relavant document about GELF output plugin. When I blindly tried, I got the above error. Please let me know proper configuration for the plugin. I'd like to use only fluent-bit for graylog without having fluentd.

Thanks for great sw.

Source

pointbre

Most helpful comment

I ended up finding a solution for this issue.

GELF has 3 mandatory fields - version, host and short message, so the following is what I've done:

1.Putting version and host using record_modifier
[FILTER]
Name record_modifier
Match *
Record version 1.1
Record host host_name

2.Specifying short message key for GELF output plugin:
[INPUT]
Name tail
Path test.log
Key log
By default, tail input plugin uses "log" as the key of message populated from the target file.

[OUTPUT]
Name gelf
Match *
Host host_name
Port 12201
Mode udp
Gelf_Short_Message_Key log <<<<<< Mandatory configuration, log is the key of the message

Please update document if this is the right way to use GELF output plugin and also feel free to close this issue.

pointbre on 19 Dec 2018

👍4

>All comments