Fluentd-kubernetes-daemonset: Make the fluentd conf configurable from configMap

[vidhy]

We can have the config files retrieved from configMap ,Instead of baking the fluentd conf files inside the docker image. This would eneble user to give custom configurations.

[vpistis]

In my fluentd version 1.2.2 installation I replace config files using a config map mounted into the fluent config path like so:

volumeMounts:
  - name: config-volume
    mountPath: /fluentd/etc

and for the volume:

volumes:
  - name: config-volume
    configMap:
      name: fluentd-config

*I have added the fluentd version

[tuchodi]

@vpistis

I replace config files

will fail in k8s, because of:

1. configMap is RO
   https://github.com/kubernetes/kubernetes/issues/60814
   https://github.com/kubernetes/kubernetes/issues/62099#issuecomment-378770047

Fix impact:
Secret, configMap, downwardAPI and projected volumes will be mounted as read-only volumes. Applications that attempt to write to these volumes will receive read-only filesystem errors.

1. attempt to sed here https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/docker-image/v0.12/alpine-elasticsearch/entrypoint.sh#L6-L12

if [ -z ${FLUENT_ELASTICSEARCH_USER} ] ; then
   sed -i  '/FLUENT_ELASTICSEARCH_USER/d' /fluentd/etc/${FLUENTD_CONF}
fi

if [ -z ${FLUENT_ELASTICSEARCH_PASSWORD} ] ; then
   sed -i  '/FLUENT_ELASTICSEARCH_PASSWORD/d' /fluentd/etc/${FLUENTD_CONF}
fi

1. will crash with error:

sed: can't create temp file '/fluentd/etc/fluent.confXXXXXX': Read-only file system

the issue is still open for k8s

[vpistis]

@tuchodi I'm sorry, but I'm not specified that I used the configMap for fluentd 1.2.2 version, and I don't see the RO error.

[tuchodi]

@vpistis
configMap clearly specified here https://github.com/fluent/fluentd-kubernetes-daemonset/issues/174#issuecomment-405254408:

and for the volume:

volumes:
 - name: config-volume
   configMap:
     name: fluentd-config

it will fail unless you are providing FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD in env

[vpistis]

@tuchodi

it will fail unless you are providing FLUENT_ELASTICSEARCH_USER and FLUENT_ELASTICSEARCH_PASSWORD in env

correct!

...and last but not least, I use fluentd with MongoDB and not Elasticsearch 😎 .
Using the configMap for me solve the problem, I think is not the final solution, but it works ☺️

[repeatedly]

Hmm... Does anyone know what is the best way to apply configMap with elasticsearch image?

[itayariel]

I used a config map with init container that copies the config files from a config map to an empty dir volume, this way it doesn't fail with the write permissions error
spec: template: metadata: labels: k8s-app: fluentd-logging version: v1 kubernetes.io/cluster-service: "true" spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule initContainers: - name: config-fluentd image: busybox imagePullPolicy: IfNotPresent command: ["/bin/sh","-c"] args: - cp /fluentd/etc2/fluent.conf /fluentd/etc/fluent.conf; cp /fluentd/etc2/kubernetes.conf /fluentd/etc/kubernetes.conf; volumeMounts: - name: config-path mountPath: /fluentd/etc - name: config-source mountPath: /fluentd/etc2 containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:elasticsearch env: - name: FLUENT_ELASTICSEARCH_HOST value: elasticsearch.default.svc - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" - name: FLUENT_UID value: "0" - name: FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX value: "fluentd" resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: config-path mountPath: /fluentd/etc terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: config-source configMap: name: fluentd-config - name: config-path emptyDir: {}

[hkailantzis]

tried the above recommendation but it fails with:

adduser: uid '0' in use
chown: unknown user fluent
chown: unknown user fluent
sed: /fluentd/etc/fluent.conf: No such file or directory

Any idea of what I'm doing wrong ? using same fluentd-kubernetes-daemonset:elasticsearch image and trying to configure fluentd via a configMap.

passing the following args in the init container:

cp /fluentd/etc2/system.conf /fluentd/etc/system.conf;
cp /fluentd/etc2/containers.input.conf /fluentd/etc/containers.input.conf;
cp /fluentd/etc2/system.input.conf /fluentd/etc/system.input.conf;
cp /fluentd/etc2/forward.input.conf /fluentd/etc/forward.input.conf;
cp /fluentd/etc2/monitoring.conf /fluentd/etc/monitoring.conf;
cp /fluentd/etc2/output.conf /fluentd/etc/output.conf;

[itayariel]

Check this out https://github.com/fluent/fluentd-kubernetes-daemonset/issues/173

[hkailantzis]

I have FLUENT_UID set to "0". Removed it and now I get:

sed: /fluentd/etc/fluent.conf: No such file or directory

[hkailantzis]

it seems that it was expecting fluent.conf to be present. renamed it in my configmap and now works fine.

[kurktchiev]

can you elaborate on what you did @hkailantzis

[hkailantzis]

hey @kurktchiev. just used the example above posted by "itayariel", but in my case, had to name the config as "fluent.conf" inside my configMap. : e.g.

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluentd-es-config
  namespace: logging
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  fluent.conf: |-
 .....

[coders-kitchen]

wouldn't it be an option to keep the configuration for eg. elasticsearch in the container - where the sed command will work - but move the other configs in a folder like

/etc/fluentd/config.d

and reference it than via

@include /etc/fluentd/config.d/*

which is a mount from the config maps?

[Madhansudhan]

kind: ConfigMap
apiVersion: v1
metadata:
name: fluentd-es-config
namespace: logging
labels:
addonmanager.kubernetes.io/mode: Reconcile
data:
fluent.conf: |-

@itayariel & @hkailantzis
Am also getting the same error as "hkailantzis" reported ,
sed: /fluentd/etc/fluent.conf: No such file or directory

Even I have created configmap(fluent.conf) on kube-system namespace as well. Please find the events for the pod(fluent)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m44s default-scheduler Successfully assigned kube-system/fluentd-5gcgr to
Normal Pulled 2m43s kubelet, Container image "busybox" already present on machine
Normal Created 2m43s kubelet, Created container
Normal Started 2m43s kubelet, Started container
Normal Created 2m4s (x4 over 2m43s) kubelet, Created container
Normal Started 2m3s (x4 over 2m43s) kubelet, Started container
Warning BackOff 85s (x9 over 2m41s) kubelet, Back-off restarting failed container
Normal Pulled 74s (x5 over 2m43s) kubelet, Container image "fluent/fluentd-kubernetes-daemonset:elasticsearch" already present on machine
Can some one help me to overcome this please...

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[andrecp]

I'm running into the same issue as @Madhansudhan . I don't quite understand why.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days