Fluentd-kubernetes-daemonset: SystemD.conf throwing excessive errors in latest Debian+Elasticsearch

[fadingdust]

Mostly because systemD is not installed:
#dpkg -L systemd-services
dpkg-query: package 'systemd-services' is not installed

Docker Image:
fluent/fluentd-kubernetes-daemonset:v1.2-debian-elasticsearch

FluentD version: 1.2.1

Linux Version:
#uname -a
Linux fluentd-es-5d86499857-2ww8k 4.9.93-linuxkit-aufs #1 SMP Wed Jun 6 16:55:56 UTC 2018 x86_64 GNU/Linux

Errors:
[warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s

[okkez]

Confirmed.
The base image of fluent/fluentd-kubernetes-daemonset is fluent/fluentd.
The base image of fluent/fluentd is debian:stretch-slim.
debian:stretch-slim does not include systemd daemons.
So we don't need to install fluent-plugin-systemd and its configurations, I think.

@repeatedly How do you think about this? If you are ok, I will create the PR to remove fluent-plugin-systemd and its configurations.

[repeatedly]

Hmm... how about adding systemd?
It collects system logs so colleting it on debian images is good for me.

[okkez]

Installing systemd to the fluentd pod (fluentd daemonset) makes no sense.
Because the fluentd daemonset wants to collect node's journal.
Maybe we can mount node's path to the fluentd daemonset like following:

diff --git a/fluentd-daemonset-elasticsearch.yaml b/fluentd-daemonset-elasticsearch.yaml
index 8c59cc1..70aa956 100644
--- a/fluentd-daemonset-elasticsearch.yaml
+++ b/fluentd-daemonset-elasticsearch.yaml
@@ -56,6 +56,9 @@ spec:
         - name: varlibdockercontainers
           mountPath: /var/lib/docker/containers
           readOnly: true
+        - name: journal
+          mountPath: /var/log/journal
+          readOnly: true
       terminationGracePeriodSeconds: 30
       volumes:
       - name: varlog
@@ -64,3 +67,6 @@ spec:
       - name: varlibdockercontainers
         hostPath:
           path: /var/lib/docker/containers
+      - name: journal
+        hostPath:
+          path: /run/log/journal

I've tested on minikube environment.
I've confirmed that this configuration avoid errors described in this issue.

[repeatedly]

Hmm... I don't catch up recent k8s changes but systemd plugins works before.

https://github.com/fluent/fluentd-kubernetes-daemonset/pull/127
https://github.com/fluent/fluentd-kubernetes-daemonset/pull/130

Does k8s or daemonset change these behaviours recently?

[okkez]

This is not k8s daemonset issue.
This is the node's systemd-journald configuration issue.

/var/log is mounted to the fluentd daemonset here.
But /var/log/journal does not exist in my node (created by minikube).
This is caused by node's systemd-journald configuration.
My node's systemd-journald stores journald logs to /run/log/journal by default, but others' will store journald logs to /var/log/journal.

See more details: https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html

[repeatedly]

@errm Do you have similar problem before?

[repeatedly]

@okkez How about following changes for this issue.

• Use env vars to disable systemd configuration like @include "#{ENV['FLUENTD_SYSTEMD_CONF_NAME'] || 'systemd'}.conf". This doesn't break existing environment.
• Document journal note and your mount setting in README for collecting node's journal

[okkez]

Looks good to me.

[repeatedly]

Just merged https://github.com/fluent/fluentd-kubernetes-daemonset/pull/258
Check to disable systemd inputs: https://github.com/fluent/fluentd-kubernetes-daemonset#disable-systemd-input