Graylog2-server: Sync behaviour of GELF plugin and Graylog (re: has empty mandatory "short_message")

[hangstl]

My setup: Logstash -> GELF -> Graylog 2.4.6

Some of my logfiles contain lines with only whitspace characters (TAB + end of line). This leads to lots of "has empty mandatory "short_message" field" Exceptions in Graylog.

imho this is a bug, because there _is_ a message (the TAB character). GelfCodec.java has this line here:

if (StringUtils.isBlank(shortMessageNode.asText()) && StringUtils.isBlank(messageNode.asText())) {
                throw new IllegalArgumentException(prefix + "has empty mandatory \"short_message\" field.");
}

Would it make sense to change StringUtils.isBlank() to StringUtils.isEmpty()

If not, I would appreciate to sync the behaviour of the GELF Plugin and the GelfCodec in Graylog. I mean, why send messages to Graylog if it rejects it anyway?

[jalogisch]

REF https://community.graylog.org/t/errors-empty-mandatory-short-message-field/7060

[eduault]

I have the same problem with my Graylog 2.4.6 server on Ubuntu Linux (created in AWS from The AMI provided by Graylog).

The problematic messages are sent by Graylog itself (or a plugin).

root@graylog:/data/log/graylog/server# netstat | grep 36028
tcp        0      0 graylog:36028           graylog:12202           ESTABLISHED
tcp6       0     68 graylog:12202           graylog:36028           ESTABLISHED

Here are the logs :

2018-10-15_09:43:24.46613 java.lang.IllegalArgumentException: GELF message <a304ef41-d054-11e8-bee0-0600ea37eac6> (received from <graylog_ip:36028>) is missing mandatory "short_message" or "message" field.
2018-10-15_09:43:24.46615   at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:262) ~[graylog.jar:?]
2018-10-15_09:43:24.46616   at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:134) ~[graylog.jar:?]
2018-10-15_09:43:24.46617   at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:150) ~[graylog.jar:?]
2018-10-15_09:43:24.46617   at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
2018-10-15_09:43:24.46617   at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:74) [graylog.jar:?]
2018-10-15_09:43:24.46618   at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:42) [graylog.jar:?]
2018-10-15_09:43:24.46618   at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
2018-10-15_09:43:24.46618   at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
2018-10-15_09:43:24.46618   at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
2018-10-15_09:43:24.46619 WARN  [GelfCodec] GELF message <a30627c1-d054-11e8-bee0-0600ea37eac6> (received from <graylog_ip:36028>) is missing mandatory "host" field.
2018-10-15_09:43:24.46619 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=a30627c1-d054-11e8-bee0-0600ea37eac6, journalOffset=450585286, codec=gelf, payloadSize=239, timestamp=2018-10-15T08:30:56.572Z, remoteAddress=/graylog_ip:36028} on input <5b8cfa3d61d4970cb039a638>.
2018-10-15_09:43:24.46620 ERROR [DecodingProcessor] Error processing message RawMessage{id=a30627c1-d054-11e8-bee0-0600ea37eac6, journalOffset=450585286, codec=gelf, payloadSize=239, timestamp=2018-10-15T08:30:56.572Z, remoteAddress=/graylog_ip:36028}

Those logs sent by "runit-service" facility may be the same :

facility
    runit-service
from_gelf
    true
level
    6
message
    ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=61ee7a20-d050-11e8-bee0-0600ea37eac6, journalOffset=449355463, codec=gelf, payloadSize=308, timestamp=2018-10-15T08:00:29.378Z, remoteAddress=/172.20.128.41:36028} on input <5b8cfa3d61d4970cb039a638>.
source
    graylog-server
timestamp
2018-10-15T09:11:11.089Z

[eduault]

In GELFMessageTest.java, GELF_JSON has no "short_message".
In the test methods of this class, some new GELFMessage(...) are created with json data having no "short_message".

Idem in GELFMessageChunkTest.java : GELF_JSON has no "short_message" nor "version".

And Graylog GELF inputs apparently accepts GELF JSON having no "version", which is said to be mandatory !

[piellick]

Hi everyone,

there is a workaround to this before waiting the bug correction ?

[n-rodriguez]

Hi there! Any news?