Splunk-connect-for-kubernetes: How to get everything from journal?

[lindsjan]

System works fine with unit filters, but how to get everything from journal , e.g as plain journalctl would return?

tried with empty filters [], assuming that _TRANSPORT would always return something and thus not block jq transformer changing the record.source.
<source> @id journald-all @type systemd @label @SPLUNK tag journald.all:all path "/run/log/journal" filters [] read_from_head true <storage> @type local persistent true </storage> <entry> field_map {"MESSAGE": "log", "_TRANSPORT": "source"} field_map_strict true </entry> </source>

fluentd log looks perfectly fine, conf is read ok, but nothing from systemd comes out to Splunk ( all other source types do continue working) . If i switch back to systemd s with unit filters, starts working.

Any help is appreciated..

Thanks in advance,
Jan

[matthewmodestino]

Hi lindsjan,

Thanks for raising this question! I will try and see if I can put together a workable config that would allow the entire journal to be dumped, though I must point out that there may be different data parsing needs/wants based on the service logging to journal, like we do with kube and docker.

Are you currently collecting the entire journal another way today, beyond on the host?

[lindsjan]

Hi,
thanks for looking into this!
currently not doing that with other means. Thinking if of getting /var/log/messages with source.files.conf.

[matthewmodestino]

Hey lindsjan,

The logging pod we ship uses https://rubygems.org/gems/fluent-plugin-systemd/versions/0.3.1

Checking the repo https://github.com/reevoo/fluent-plugin-systemd/, it looks like they dive into the behavior and matching config. https://github.com/reevoo/fluent-plugin-systemd/blob/master/docs/matching.md

I believe if you simply leave the match empty you will get it all. But have a look in there and see if it leads you to a config that gets the entire journal. I'll let you know if I get it working.

[lindsjan]

tried with matches[] but got error.. (cannot recall what it was exactly, but pointing to direction that matches would not be supported ). also tried with listing all priorities in filters [{ "PRIORITY": [0,1,2,3,4,5,6] }] but that did not solve it either.

[sayeedc]

Any luck with getting everything journal into Splunk via fluentd? I tried using the matches [] but no luck. The unit filters are working fine.

[sayeedc]

This should work
<source> @id journald-all @type systemd @label @SPLUNK tag journald.journal:all path "/run/log/journal" matches [] read_from_head true <storage> @type local persistent true </storage> <entry> field_map {"MESSAGE": "log", "_SYSTEMD_UNIT": "source"} field_map_strict true </entry> </source>

[chaitanyaphalak]

Closing the issue as @sayeedc has got it working.