Fluent-bit: udp json input support

[nvx]

Is your feature request related to a problem? Please describe.
There is currently a json tcp input, but no corresponding option for udp

Describe the solution you'd like
Having the ability to consume json events sent over udp would be beneficial

[edsiper]

would you please elaborate your use case ?

[nvx]

Reliably transporting logs sent from applications via UDP.

In some instances it is desirable for the application to emit logs via UDP rather than handle reconnection/buffering/blocking/etc in the application where dropping logs in the event of logging infrastructure failures are acceptable. The main problem with this approach is that sending UDP over the network offers very little guarantees.

A pattern I have used is ingesting UDP logs over localhost to a local agent, which in turn then takes care of buffering and reliably sending over the network. This pattern can be achieved today with the in_udp fluentd input, Logstash, or nxlog. The downside is the former two agents are quite heavyweight, while nxlog meets the lightweight requirement it's capabilities are rather limited compared to eg fluent-bit.

The use-case is similar to syslog ingestion over tcp vs udp which I note fluent-bit supports both today.

[newrushbolt]

I got pretty much the same case that @nvx, so i'd like to vote up for this feature.

[proligde]

Same scenario here - are there any plans to make that happen?

[dontspamterry]

Likewise, would be nice to have UDP input

[p53]

+1 for UDP input, we would like to replace rsyslog forwarder with fluent-bit

[nvx]

Is there still any information needed? I notice the awaiting for user label is still on this @edsiper

[jwitrick]

Is there any timeline/update for this? UDP input would be a HUGE win for streaming logs with fluent bit.

[bluebike]

Actually in_syslog could be used as _generic_ UDP input.. .just by giving suitable parser.
I thought that if we just take in_syslog and copy it to as in_udp (some search/replace).
Then add right method to parse tag name from message... and we have generic UDP input.

Also this could be used for something like in_influxdb for influxdb line protocol with UDP.
In that case it would better to InfluxDB line protocol parser (regex is not really enough for that).

[JimHagan]

We have this use case. I'm an engineer at New Relic and we would like to be able to offer this option to our users who use fluentbit as a forwarder.

[bluebike]

@JimHagan Few weeks ago I copied in_syslog to in_udp ... replaces syslog to udp...
and did some testing (didn't have to time do more...)
I could try to make PR in few days..

[bluebike]

I made PR #2414 in_udp module...
@nvx @JimHagan You could have test... or give comments if module would cover your needs.

I have tested it only in my macOS (10.13) so far.

[frzb]

One further use case:

Sending kernel and early user space (initramfs) logs via the netconsole kernel module over UDP before any other logging daemon is running during system boot.

I use that with Coinboot for a huge number of diskless nodes without any KVM over IP capabilities for debugging early boot stages.
These kernel message don't align with RFC 3164 or RFC 5424 and so fail to be ingested by the in_syslogmodule.