Fluentd-kubernetes-daemonset: Update to 1.x

[jonmoter]

According to https://www.fluentd.org/download, v1.0 is "currently stable", and I see the 1.0 images at https://hub.docker.com/r/fluent/fluentd/

Can we update the daemonset definitions to refer to 1.0?

Also, should probably update the README to reflect the current stable version.

[sonnysideup]

I agree with @jonmoter here. I would expect this particular repo to say up-to-date with the mainstream changes in FluentD. Is that a realistic expectation? Or should we be building our own DaemonSet images using https://hub.docker.com/r/fluent/fluentd/?

[kenden]

v1.1 of fluent/fluentd is now released. Can you update the title of this issue?

[StevenACoffman]

Check out kubernetes/kubernetes#58525

If this has been merged by the time you read this, then you can just refer to the kubernetes/cluster/addons/fluentd-elasticsearch fluentd daemonset manifest: kubernetes/kubernetes:cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml@master

[jonmoter]

If the Kubernetes cluster addons directory is the official supported way to run Fluentd, that's fine. In that case, I request that we update the README of this repo to say this repo is deprecated, and link to the correct place.

If this repo is still intended to be instructions for running Fluentd in Kubernetes, then I expect it to be kept reasonably up to date.

[StevenACoffman]

@jonmoter I understand your excitement and frustration. I'm sure the maintainers would appreciate a pull request from you. The sort of maintenance you describe in this open source repo takes time and energy, and such contributions are always appreciated by the other volunteers in the open source community.

[StevenACoffman]

By the way, these other repositories could also use updating:

• samsung-cnct/kraken-logging-fluentd-daemonset
• fabric8io docker image + fabric8 Config
• heptio/fluentd-yaml

I'm sure there's more.

[jonmoter]

Okay, fair point. Since there wasn't anything on the README about contribution guidelines, I didn't know whether the maintainers want PRs, or if they have their own process for updates.

I'll try to whip up a PR over the weekend.

[edsiper]

we will start switching now, I will do a separate branch to test 1.0, indeed we need your help.

[monotek]

I've added a fluentd 1.1.0 version to kubernetes recently:

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/

Maybe you can adapt from there...

[repeatedly]

@edsiper open PR with https://github.com/fluent/fluentd-kubernetes-daemonset/tree/fluentd-v1 is better for checking patch from other developers.

[vide]

@monotek the fluentd-elasticsearch image in k8s addons doesn't build at all currently (missing ffi library)

[StevenACoffman]

@vide Check kubernetes/kubernetes#59665 got merged and try again now

[vide]

@StevenACoffman oh, great, missed it by a few hours :/ thanks for pointing it out. Anyway I fixed it by myself in my fork but I couldn't get fluentd 1.1.0 working reliably in my (minikube) environment, so I ended up using old 0.12.40. I will try again anyway. Thanks.

[ghost]

+1

[Lawouach]

Yeah, I would welcome such an update to! :)

[repeatedly]

@edsiper What is the blocker for v1 images?

[rajatjindal]

@edsiper thanks for providing the daemonset, we are currently using it. But we are running into some issues which I believe are fixed with latest version of fluentd.

I see there is a branch for v1.x already. is there anything I can do to get that work completed?

[repeatedly]

We merged https://github.com/fluent/fluentd-kubernetes-daemonset/pull/116 and v1.1 images are avilable on docker hub.
But currently, v1.1 image has a permission problem for /var/log directory, read logs and write pos/buffers. This is because newer fluentd images run fluentd as fluent user, not root.

2018-04-27 09:49:15 +0000 [error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/kube-apiserver-audit.log.pos"

What approch is better for resolving this problem?

[hobbsh]

@repeatedly is there a reason (as noted in #102) why Logz.io images are not being built/pushed?

[StevenACoffman]

@repeatedly The /var/log/ directory on my ubuntu 14.04 has permissions like these:

$ ls -la 
drwxrwxr-x 11 root syslog 4096 Apr 29 01:17 log
$ ls -n
drwxrwxr-x 11 0 104 4096 Apr 29 01:17 log

This shows that the owner is root (uid 0) and the group is syslog (gid 104). If your container is executing as uid 0 or gid 104, then you will have write access to this directory. I am not sure if this is the standard gid for all unix distributions.

[repeatedly]

@StevenACoffman Docker's debian image uses root:root for /var/log

$ ls -al /var/
total 44
drwxr-xr-x 1 root root  4096 Apr 27 07:11 log

and recent fluentd-docker-image runs fluentd as fluent user

$ id
uid=1000(fluent) gid=1000(fluent) groups=1000(fluent)

In OpenShift, add scc to fluent: https://github.com/fluent/fluentd-kubernetes-daemonset#running-on-openshift
I'm not sure how to use same way in k8s with yaml setting or
k8s has other way for it.

[monotek]

Just create a new docker image where you add the fluentd user to the right group.

I did this for logstash & journald logging a while ago:

https://github.com/monotek/docker-logstash-journald/blob/master/Dockerfile

Should work the same way for fluentd...

[micdah]

A workaround others might find useful, after much trial and error with trying to grant the fluent user enough permissions to write to /var/log and read log files in /var/log/containers (_it was complaining that it was unable to read any log file_), I simply gave up and tried to find a way to circumvent running it as the fluent user.

In the end, I decided to overwrite the Entrypoint defined in the root fluentd:1.1.3 image (_which is were it uses gosu to run the CMD as defined by sub-images as the fluent user_).

Dockerfile:

FROM fluent/fluentd-kubernetes-daemonset:v1.1-debian-elasticsearch

USER root
WORKDIR /home/fluent

# Copy custom configuratoin
COPY ./entrypoint.sh /bin/entrypoint.sh
RUN chmod +x /bin/entrypoint.sh

CMD ["/fluentd/entrypoint.sh"]

entrypoint.sh:

#!/usr/bin/dumb-init /bin/sh

exec "$@"

This way, fluentd will be running as root and all of the permission problems vanished.

I honestly see no benefit to running as an unprivileged user while running inside a _container_, I find that more useful for when running directly on the host OS.

[repeatedly]

I use micdah's approach, https://github.com/fluent/fluentd-kubernetes-daemonset/pull/119, and latest image works without permission error.
I want feedback for v1.1 images. If no problem, I will add v1.2 images.

[avila3a]

@repeatedly

I have openshift problem:

0 unexpected error error_class = Errno :: EACCES error = "Permission denied @ rb_sysopen - /var/log/kube-apiserver-audit.log.pos"

Use the latest version v1.9 and it doesn't work for me either, I already sent the parameter FLUENT_UID = 0. It doesn't give me the same error either.

My yaml is:
https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days