Fluent-bit: Fluent-bit stops reading from systemd input when combining with tail input

[dshmatov]

I'm using v0.12.7

So, the problem is appears when I start to combine two input plugins in the same configuration file (tail and systemd):

[SERVICE]
    Flush        3
    Daemon       Off
    Log_Level    debug
    Parsers_File parsers.conf

[INPUT]
    Name         systemd
    Tag          default.*

[INPUT]
    Name         tail
    Path         /tmp/sni*.log
    Tag          default.*

[OUTPUT]
    Name         forward
    Host         127.0.0.1
    Port         24999
    Match        *

tail input plugin works normaly, I get an reaction on adding new lines to /tmp/sni_dev.log. But in the same time I got no reaction on adding new events to journald service. When I remove tail input plugin from configuration - systemd starts to work normaly and I get logs from it.
I'm using fluent-bit as Docker container and mount systemd journal and /tmp/ folder directly to it from host machine using -v option. I've tried different versions of fluent-bit (0.12.*) but result was the same. Problem starts appearing after fluent-bit boot or after sveral minutes (5-10) or an hour.

[edsiper]

@dshmatov thanks for reporting this problem, I am troubleshooting now.

[edsiper]

Issue confirmed.

[edsiper]

@dshmatov

I've pushed a fix into GIT master and backported to 0.12 branch, the fix will be available this week on v0.12.8 release.

[dshmatov]

@edsiper Great! waiting for 0.12.8. Thanks!

[edsiper]

0.12.8 is out: http://fluentbit.io/announcements/v0.12.8/

closing this issue as fixed. thanks again.

[AkshayDubey29]

@edsiper Seems this issue is in 0.14.8 as well

[SERVICE] Flush 1 Daemon Off Log_Level debug Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 [INPUT] Name systemd Tag host.* Systemd_Filter _SYSTEMD_UNIT=kubelet.service Path /var/run/systemd/journal/ [FILTER] Name kubernetes Match host.* Kube_URL https://kubernetes.default.svc:443 Merge_Log On Use_Journal On [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token K8S-Logging.Parser On K8S-Logging.Exclude On [FILTER] Name record_modifier Match kube.* Record sourcetype kubernetes Record index stg_sde [FILTER] Name modify Match kube.* Rename log message [OUTPUT] Name stdout Match host.*

[edsiper]

If you log into Fluent Bit container, can you see the journal logs with:

$ journalctl -u kubelet.service 

?

also, your Kubernetes filter section needs to have the option "Use_Journal On"