Beats: Syslog input to support RFC5424

[ph]

The current implementation of the parser only support RFC3164, some newer system uses RFC5424. It's just a matter of adding new state machines to the Ragel parser and add new tests for it.

We have to investigate if we can detect it on the fly OR we make it a user configurable settings.

Lets make sure we add integration test for this all list with one major producer.

[anandsinghkunwar]

Is anyone working on this?

[ph]

linked to https://github.com/elastic/beats/issues/8307

[seanmil]

We're an Elastic customer and are interested in RFC-5424 support as well. Any idea if this will get worked on soon?

[rwaight]

If it helps, while troubleshooting an ERROR [syslog] syslog/input.go:132 can't parse event as syslog rfc3164 error for the cisco module; I found that using the workaround mentioned in this discussion comment allows Filebeat to parse Cisco ASA logs.

For clarification, the module/cisco/asa/config/input.yml file is modified from this:

{{ if eq .input "syslog" }}

type: syslog
protocol.udp:
  host: "{{.syslog_host}}:{{.syslog_port}}"

To this:

{{ if eq .input "syslog" }}

type: udp
host: "{{.syslog_host}}:{{.syslog_port}}"

[andrewkroh]

It looks like https://github.com/elastic/beats/pull/15467 may have stalled out. @faec, is this still something you are looking into addressing?

[faec]

It's not stalled in the sense of being blocked, but it is a back-burner task for me (it was never an explicit release goal, just something I hoped to fit in between my other work). If someone has spare capacity I'm happy to hand it off / help get started, otherwise I'll pick it up again when time and triage allows.

[getkub]

hi, RFC5424 is really important in large scale datasets/datalake. Any file if you are working, I can also help to make this rolled out sooner.

Just looking into @faec code, it seems complex. Is it better to have this written as a pattern file? Something like what logstash have https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

For example,

pattern_name: RFC5424_non_structured

payload:
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

pattern:
<%{NUMBER:syslog_priority}>%{NUMBER:syslog_version} %{TIMESTAMP_ISO8601:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{SYSLOGPROG:syslog_prog} %{NOTSPACE:syslog_procid} %{NOTSPACE:syslog_msgid} %{NOTSPACE:syslog_structured_msg} %{GREEDYDATA:syslog_non_structured_msg}

[abraxxa]

The Checkpoint log exporter sends the logs in RfC5424 format by default which is why this is required for the Checkpoint module to work.
Currently it fails with can't parse event as syslog rfc3164.

Please prioritize this, we're also a paying Elastic customer, thanks!

[WPH95]

Hi, I would like to find out if this issue still need?
I would be happy to re-open the pr #20246 if this is still a desired feature.

20246 pr add RFC-5424 support. (Previously the pr was automatically closed by bot because it hadn't been reviewed for too long)

• The code style is very similar to the rfc3164 format impl, using Ragel.
• Have unit test, Full coverage the example in rfc5424 spec docs