Beats: Journald input in Filebeat

[kvch]

Add new input to Filebeat to collect entries from journald journals. The feature's already been under development. But now it's blocked.

Input files

If paths is empty, the default journal is opened. It's possible to provide directories and single journal files as inputs.

Filtering

It is possible to filter entries at journald level by providing key-value pairs. Thus, Filebeat does not need to filter at all or needs to filter less incoming events. Filter expressions needs to be match exactly the values of fields.

Example configuration

- type: journald
  paths:
    - /dev/log
    - /var/log/messages/my-journal-file
  filters:
    unit: nginx.service
    level: error

Why is it blocked?

The way journald tracks its offsets is not yet supported by Filebeat registry. Handling and saving positions if Filebeat needs a refactoring, so it becomes possible to save journald state info.

[kvch]

Journalbeat issue: https://github.com/elastic/beats/issues/8323

[ksemaev]

@kvch the current situation is - we have both systemd and syslog type of logs at the same time on most of operating systems. That is not truely desirable to have two type of beats running on each instance just because of different log types. As I see - journalbeat is like fluentd - it just supports journald logs. Do we have any beat that can handle both systemd and syslog types of logs?

[kvch]

Unfortunately, right now there is no Beat which supports both inputs at the same time. However, we are still planning to add journald input to Filebeat. The necessary refactoring are in progress, but we don't know exactly when the new input going to be added.

For users who does not mind running a separate Beat to collect journald entries we would like to provide a new Journalbeat in a future release.

[jain108shubhamtbt]

hi,

when will we get journald input support in filebeat ? Please update.

[kvch]

Unfortunately, there hasn't been any notable updates since my last post. The registry refactoring is still in progress. In the meantime, Journalbeat is being developed, so when the time comes, you are getting a mature input.

[earlpotter0]

Any update?

[kvch]

Unfortunately, there is no update. For future reference, when there is an update with the Journald input, it will be added to this ticket. So if one subscribes, he/she can get notified.

[zgfh]

Any update?

[stevehorsfield]

This might be useful for people here: https://medium.com/@stevehorsfield/send-your-systemd-journal-logs-to-graylog-a2cbcd982cb4?source=friends_link&sk=e6801624a3fa2be715c31af98750cab4

[earlpotter0]

While the above is cool, it would be nice to have an Elastic supported tool.
1) Journalbeat works but is experimental and not supported by Elastic.
2) Beats and the ECS make a lot of sense, but having 3 or 4 (File, Metric, Audit, Journal) beats running on a machine starts to chew up a lot of resources. It does make a lot of sense to have these all as a module in one beat that are enabled as needed.