Fluentd-kubernetes-daemonset: Images unable to start with Read-Only ConfigMaps

[nrmitchi]

In response to https://github.com/kubernetes/kubernetes/issues/60814, the Kubernetes team pushed out https://github.com/kubernetes/kubernetes/pull/58720. This was also patched into the latest versions of 1.7.x, 1.8.x, and 1.9.x.

It makes mounted ConfigMaps ready only by default.

As a result, the sed commands in the default entrypoint for all images in this repo fail (combined with set -e causes the full entrypoint to fail, and thus enter a CrashLoopBackoff.

Entrypoint: https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/templates/entrypoint.sh

As a work-around, I've bypassed the entrypoint.sh file by specifying the command specifically, ie:

      containers:
      - name: fluentd
         image: fluent/fluentd-kubernetes-daemonset:v0.12-alpine-s3
         command:
           - fluentd
           - -c
           - /fluentd/etc/fluent.conf
         ...

This works for now, however it's definitely not the best, or an appropriate permanent, solution.

Any thoughts on how we can get this updated to avoid writes to mounted ConfigMaps?

[rdpa]

Seeing this same issue, my (hopefully temporary) hack workaround was to add

            - name: FLUENT_ELASTICSEARCH_USER
              value: none
            - name: FLUENT_ELASTICSEARCH_PASSWORD
              value: none

to the container env section of the daemonset manifest.

[pierrebeaucamp]

Further, shouldn't the two conditionals in entrypoint.sh us -n instead of -z?

[Itfly]

@nrmitchi should add "-p /fluentd/plugins" under command to add the plugins

[nrmitchi]

Good call; I'll probably switch to @rdpa 's solution which seems slightly more clean until it's fixed

[kfox1111]

You can use an init container to copy the configmap's data to an emptydir. I pull this trick all the time when I want to tweak the configmap in the pod during deployment.

[jpke]

@kfox1111, there is a pr in for the helm chart to do that https://github.com/kubernetes/charts/pull/5210

[repeatedly]

Which approach is better for fixing this issue?
Modify entrypoint.sh or update daemonset yaml prototype with jpke's patch way?

[aonz]

We got a similar problem when we tried to install multiple daemonsets (we are testing our a few log management systems like loggly and logz.io).

The error (when installing the 2nd daemonset) was sed: cannot rename /fluentd/etc/<id>: Device or resource busy which should be from the sed command on entrypoint.sh.

It seems like that only Elasticsearch image that needs these 2 parameters, would it be possible to remove them for the other images?

[whereisaaron]

Here is an example workaround the AWS EKS Workshop website suggests, using an init container similar to the helm chart approach.

https://eksworkshop.com/logging/deploy/

https://eksworkshop.com/logging/deploy.files/fluentd.yml

      # Because the image's entrypoint requires to write on /fluentd/etc but we mount configmap there which is read-only,
      # this initContainers workaround or other is needed.
      # See https://github.com/fluent/fluentd-kubernetes-daemonset/issues/90
      initContainers:
      - name: copy-fluentd-config
        image: busybox
        command: ['sh', '-c', 'cp /config-volume/..data/* /fluentd/etc']
        volumeMounts:
        - name: config-volume
          mountPath: /config-volume
        - name: fluentdconf
          mountPath: /fluentd/etc

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions[bot]]

This issue was automatically closed because of stale in 30 days