Fluentd-kubernetes-daemonset: Unique tags for docker images

[gtrafimenkov]

I had a kubernetes cluster of two nodes.

fluent/fluentd-kubernetes-daemonset was collecting logs to elasticsearch.

      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:elasticsearch

Everything was fine.

Then I added another node to the cluster and on this new node fluentd didn't want to start. Error was:

2018-01-24 09:02:32 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="start_namespace_watch: Exception encountered setting up namespace watch from Kubernetes API v1 endpoint https://10.96.0.1:443/api: namespaces is forbidden: User \"system:serviceaccount:kube-system:fluentd\" cannot list namespaces at the cluster scope ({\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"namespaces is forbidden: User \\"system:serviceaccount:kube-system:fluentd\\" cannot list namespaces at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"namespaces\"},\"code\":403}\n)"

Output of kubectl get pods:

fluentd-4gmfg                 0/1       CrashLoopBackOff   31         1h        10.244.0.116      h5
fluentd-npst8                 1/1       Running            0          1h        10.244.1.43       h13
fluentd-pnkxh                 1/1       Running            0          1h        10.244.2.72       h11

After some digging and troubleshouting I found that the new node was running different version of fluent/fluentd-kubernetes-daemonset:elasticsearch image.

Image on the old nodes:

$ docker images | grep fluentd-kubernetes-daemonset
fluent/fluentd-kubernetes-daemonset                           elasticsearch                              c04d7276b4a1        3 months ago        60.3MB

Image on the new node:

fluent/fluentd-kubernetes-daemonset                        elasticsearch       f85edd2602f8        42 hours ago        67.9MB

After copying docker image from one of the old nodes to the new one, fluentd started successfully.

I want to suggest giving docker images you publish unique tags, so when you publish a new version:

• the new version is uniquely identified
• the old one is not overridden and still available on docker hub

[ritasker]

This has also caused me issues today.
A node on GKE was restarted last night and fluentd wont start on that node.
With a different error though
sed: can't move '/fluentd/etc/fluent.confPHNfhN' to '/fluentd/etc/fluent.conf': Resource busy.

[edsiper]

the enhancement request for tagging is associated to #53

would you please log into the Fluentd container that's working now and send us the output of the following command:

$ gem list fluent-plugin-kubernetes_metadata_filter

since I see a permission issue when accessing the API server and the newer images likely picked an updated version of the filter, I would like to check the main differences.

[gtrafimenkov]

@edsiper , here is the output:

# gem list fluent-plugin-kubernetes_metadata_filter

*** LOCAL GEMS ***

fluent-plugin-kubernetes_metadata_filter (0.29.0)

[mihailgmihaylov]

Hello,

We are using your fluentd-kubernetes-daemonset:v0.12-alpine-elasticsearch image which we pull from https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset/builds/ .
Long time ago, we had an issue in which a change in the underlying image that the fluentd-kubernetes-daemonset alpine image depends on was changed and the change was backported into the image with tag v0.12-alpine-elasticsearch.
Yesterday, we had maybe the same issue in which all new instances of the containers had a configurational issue. The image name and tag was not change and this creates a inconsistency in our infrastructure.

Although, we appreciate the service and that you create and store the docker images for us, I was wondering if there you can somehow add an incremental tag on the image tag so that you are still able to backport new functionalities into the old versions of fluentd but have the ability for us to opt-out of them.

For example: fluentd-kubernetes-daemonset:v0.12-alpine-elasticsearch-1.23.4 or something similar.

[repeatedly]

Forgot to close. This issue has been resolved.