Fluentd-kubernetes-daemonset: failed to read data from plugin storage file: fluentd-journald-docker-cursor.json

[schmidtjohannes]

Hi,
the pod keeps crashing and shows the following error:

2018-12-28 14:22:30 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2018-12-28 14:22:30 +0000 [info]: [out_es] Connection opened to Elasticsearch cluster => {:host=>"1.2.3.4", :port=>1234, :scheme=>"http", :path=>""}
2018-12-28 14:22:30 +0000 [info]: [out_es] Detected ES 6.x: ES 7.x will only accept _doc in type_name.
2018-12-28 14:22:30 +0000 [error]: [in_systemd_docker] failed to read data from plugin storage file path="/var/log/fluentd-journald-docker-cursor.json" error_class=Fluent::ConfigError error="Invalid contents (not object) in plugin storage file: '/var/log/fluentd-journald-docker-cursor.json'"
2018-12-28 14:22:30 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unexpected error: failed to read data from plugin storage file: '/var/log/fluentd-journald-docker-cursor.json'"

After deleting the file manually the pod restarts ok. What is this file for?

I observe this when detecting a machine crash and restarting the node. Before the restart it shows over 500 restarts of the fluentd pod and finally k8s has problems allocating ram and kills the machine. Until now I could not identify if fluentd is the bottleneck or the daemonset has problems because other pods crash/restart.
After a machine restart fluentd keeps throwing this error and restarts.

Any ideas?

I am running K8s 1.11 on centos 7 and fluent/fluentd-kubernetes-daemonset:v1.3.2-debian-elasticsearch

[rrichardson]

I get a similar error.. oddly it is only on 1 machine of 17.. and it persists whether I delete the pod or not. It's always on the same machine.. I even wiped out the /var/lib/docker dir and unmounted all layers and restarted.. everything works except for this fluentd pod.

[rrichardson]

@schmidtjohannes - I figured it out: the executable in this package is a plugin for docker to route logs. It creates /var/log/fluentd-journald-docker-cursor.json /var/log/journald-container-runtime.pos/ and others.. On a restart.. those files could get corrupted.. what I found I had to do was rm -rf /var/lib/journald-container-runtime.pos/ and when fluentd restarted, it re-created the dir and all was well.

[schmidtjohannes]

@sivabalans this file is on the host, the path indicated in the error.

Am Mi., 20. Feb. 2019 um 14:31 Uhr schrieb sivabalans <
[email protected]>:

@rrichardson https://github.com/rrichardson I am running into the same
issue as you are. 2 of my fluentd daemonsets are always in a
CrashLoopBackOff status and the 3rd one works just fine. I see the
following error:
2019-02-20 17:29:53 +0000 [info]: [elasticsearch] Connection opened to
Elasticsearch cluster => {:host=>"elasticsearch-client.telemetry.svc",
:port=>9200, :scheme=>"http"}
2019-02-20 17:29:53 +0000 [info]: [elasticsearch] Detected ES 6.x: ES 7.x
will only accept _doc in type_name.
2019-02-20 17:29:53 +0000 [error]: [journald-docker] failed to read data
from plugin storage file
path="/var/log/journald-docker.pos/worker0/storage.json"
error_class=Fluent::ConfigError error="Invalid contents (not object) in
plugin storage file: '/var/log/journald-docker.pos/worker0/storage.json'"
2019-02-20 17:29:53 +0000 [error]: config error
file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError
error="Unexpected error: failed to read data from plugin storage file:
'/var/log/journald-docker.pos/worker0/storage.json'"

How did you login to a container that is in a "CrashLoopBackOff" status to
run the command rm -rf?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fluent/fluentd-kubernetes-daemonset/issues/245#issuecomment-465677831,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APNN_RTf7iGM_HLWSjbVNq17nXzKxX3Zks5vPYZrgaJpZM4Zj6kv
.

[rrichardson]

Yes. I fixed it by stopping docker, then rm -rf ing all *.pos
directories under the host's /var/log. I think there are some json files as
well

On Wed, Feb 20, 2019, 8:01 PM Johannes Schmidt [e-mail]
wrote:

@sivabalans this file is on the host, the path indicated in the error.

Am Mi., 20. Feb. 2019 um 14:31 Uhr schrieb sivabalans <
[email protected]>:

@rrichardson https://github.com/rrichardson I am running into the same
issue as you are. 2 of my fluentd daemonsets are always in a
CrashLoopBackOff status and the 3rd one works just fine. I see the
following error:
2019-02-20 17:29:53 +0000 [info]: [elasticsearch] Connection opened to
Elasticsearch cluster => {:host=>"elasticsearch-client.telemetry.svc",
:port=>9200, :scheme=>"http"}
2019-02-20 17:29:53 +0000 [info]: [elasticsearch] Detected ES 6.x: ES 7.x
will only accept _doc in type_name.
2019-02-20 17:29:53 +0000 [error]: [journald-docker] failed to read data
from plugin storage file
path="/var/log/journald-docker.pos/worker0/storage.json"
error_class=Fluent::ConfigError error="Invalid contents (not object) in
plugin storage file: '/var/log/journald-docker.pos/worker0/storage.json'"
2019-02-20 17:29:53 +0000 [error]: config error
file="/etc/fluent/fluent.conf" error_class=Fluent::ConfigError
error="Unexpected error: failed to read data from plugin storage file:
'/var/log/journald-docker.pos/worker0/storage.json'"

How did you login to a container that is in a "CrashLoopBackOff" status
to
run the command rm -rf?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/fluent/fluentd-kubernetes-daemonset/issues/245#issuecomment-465677831
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/APNN_RTf7iGM_HLWSjbVNq17nXzKxX3Zks5vPYZrgaJpZM4Zj6kv

.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fluent/fluentd-kubernetes-daemonset/issues/245#issuecomment-465818724,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHlC5E1oY7WnPedJWCOqMi6QuTOjCbcks5vPe_kgaJpZM4Zj6kv
.

[admssa]

I can easily reproduce it by rebooting nodes.
two nodes rebooted = two pods in CrashLoopBackOff.

fluentd-es-v1.3.3-55jp8                               1/1       Running            0          14m
fluentd-es-v1.3.3-jjmfd                               1/1       Running            0          14m
fluentd-es-v1.3.3-kjmqx                               1/1       Running            0          7m
fluentd-es-v1.3.3-ktbf7                               0/1       CrashLoopBackOff   5          14m
fluentd-es-v1.3.3-v4h5g                               0/1       Error              2          14m
fluentd-es-v1.3.3-zmwtj                               1/1       Running            0          14m

[schmidtjohannes]

I switched to fluent/fluentd-kubernetes-daemonset:v1.3.3-debian-elasticsearch-1.3 image which internally uses 1.3 of fluentd.

Then I added the following env variables FLUENT_UID=0 and FLUENTD_SYSTEMD_CONF=disable like described in the documentation and I did not face the error anymore.

[schmidtjohannes]

The update of image and adding flags worked for me

[admssa]

Yeah, it will work if you don't use systemd source :)

[shiveshabhishek]

@admssa is there any solution for this if I use systemd source?

[philippebn]

same probleme here [error]: [journald-docker] failed to read data from plugin storage file path="/var/log/journald-docker.pos/worker0/storage.json" error_class=Fluent::ConfigError error="Invalid contents (not object) in plugin storage file: '/var/log/journald-docker.pos/worker0/storage.json'"

[repeatedly]

I assume the content is 0 byte, right?
if so, the problem is related to https://github.com/fluent/fluentd/issues/2373#issuecomment-480856719
I will send a patch in this week.

[admssa]

@shiveshabhishek We added init container temporary, which checks if files are empty and remove them.
Something like

if [ ! -f $KUBELET_CURSOR_FILE ] || [ -s $KUBELET_CURSOR_FILE ]; then echo "is ok"; else rm -f $KUBELET_CURSOR_FILE && echo $KUBELET_CURSOR_FILE removed; fi;

[errm]

Seems this was an upstream bug with fluentd that is now patched!