Fluentd-kubernetes-daemonset: Get "403 Forbidden" message when running the pod

[shagabutdinov]

The pod does not start in my cluster, here is how I do that:

• I run the pod with following command:

kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch.yaml

• Wait for pod to run:

$ kubectl get pods | grep fluentd                      
fluentd-d7z2f                    0/1       CrashLoopBackOff   3          1m

• Pod reports failure:

$ kubectl logs fluentd-d7z2f
2017-04-26 06:44:32 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2017-04-26 06:44:32 +0000 [info]: starting fluentd-0.12.32
2017-04-26 06:44:32 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.9.2'
2017-04-26 06:44:32 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.26.2'
2017-04-26 06:44:32 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.8.3'
2017-04-26 06:44:32 +0000 [info]: gem 'fluent-plugin-secure-forward' version '0.4.3'
2017-04-26 06:44:32 +0000 [info]: gem 'fluentd' version '0.12.32'
2017-04-26 06:44:32 +0000 [info]: adding match pattern="fluent.**" type="null"
2017-04-26 06:44:32 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2017-04-26 06:44:33 +0000 [info]: adding match pattern="**" type="elasticsearch"
2017-04-26 06:44:33 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="Exception encountered fetching metadata from Kubernetes API endpoint: 403 Forbidden"
2017-04-26 06:44:33 +0000 [info]: process finished code=256
2017-04-26 06:44:33 +0000 [warn]: process died within 1 second. exit.

• The error is:

config error file="/fluentd/etc/fluent.conf" error="Exception encountered fetching metadata from Kubernetes API endpoint: 403 Forbidden

• After some digging I found out that server returns:

system:serviceaccount:kube-system:default cannot list pods at the cluster scope

How can I start fluentd with this issue? Thank you in advance!

[shagabutdinov]

It was appeared that cluster runs RBAC authrorization which requires to create additional configuration in order to allow fluentd to access to cluster information:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: system
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: system

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd
  namespace: system
data:
  fluent.conf: |
    @include kubernetes.conf

    <match **>
       type elasticsearch
       log_level info
       include_tag_key true
       host elastic.system.svc.cluster.local
       port 9200
       user elastic
       password <...>
       logstash_format true
       buffer_chunk_limit 2M
       buffer_queue_limit 32
       flush_interval 5s
       max_retry_wait 30
       disable_retry_limit
       num_threads 8
    </match>

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: system
  labels:
    k8s-app: fluentd-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:elasticsearch
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
            - name: config
              mountPath: /fluentd/etc/fluent.conf
              subPath: fluent.conf
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: config
          configMap:
            name: fluentd

Can you point me please, where should I put this information (in order to create the PR)?

[edsiper]

Since the PR was created and merged on https://github.comfabric8io/fluent-plugin-kubernetes_metadata_filter, considering to close this issue:

• https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/pull/71/files

@shagabutdinov is everything addressed or are you still facing some problem ?

[shagabutdinov]

Yes, everything is fine, thank you! The issue may be closed.

[jpds]

@edsiper - sorry, this doesn't fix the RBAC issues, it just reports a better error message - I've deployed the latest daemonset (https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch.yaml) from Git, and I get this when it deploys:

2017-06-22 20:41:04 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error="Exception encountered fetching metadata from Kubernetes API endpoint: 403 Forbidden (User \"system:serviceaccount:kube-system:default\" cannot list pods at the cluster scope.)"
2017-06-22 20:41:04 +0000 [info]: process finished code=256

Can you please include RBAC support into the manifest? Basically any Kubernetes 1.6.x+ deployment is broken with this as that's when RBAC is on by default.

[Maellooou]

Hello,

@edsiper do you find any workaround ? I faced to the same issue.

[falau]

Hello, @Maellooou
The manifest @shagabutdinov pasted should be working.
Specifically, Apply the ServiceAccount, ClusterRoleBinding, and ClusterRole templates,
and using fluentd to access k8s in your daemonset template: spec.template.spec.serviceAccountName: fluentd

[rotmond]

I hava a 3 nodes cluster and use “_https://github.com/fluent/fluentd-kubernetes-daemonset/blob/master/fluentd-daemonset-elasticsearch.yaml_” to deploy fluentd，it‘s working on 2 of 3 nodes，but on the last one i get ：

config error file="/fluentd/etc/fluent.conf" error="Exception encountered fetching metadata from Kubernetes API endpoint: pods is forbidden: User \"system:serviceaccount:kube-system:default\" cannot list pods at the cluster scope ({\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"pods is forbidden: User \\"system:serviceaccount:kube-system:default\\" cannot list pods at the cluster

i am confused ......

[vladget]

@rotmond I have same issue, have you been resolved it?

[pirankashani]

It seems you are using older docker image.
I would suggest you to see the docker images which are already downloaded on your cluster with this command:
docker images
and then remove the images which are tagged relevant to the elasticsearch images with this command:
docker rmi <image ID>

[marcwickenden]

I was having the same issue on GKE. It's basically the same as https://github.com/coreos/prometheus-operator/issues/357. To fix you need to give your GCloud user account the cluster-admin role:

kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=your.google.cloud.[email protected]

Note that _your-user-cluster-admin-binding_ above is arbitrary.

The email address is case sensitive. Use the shell script below to find it as GCloud sees it (code from link). Make sure to change your project ID (needs jq installed).

gcloud projects get-iam-policy "_your-project-id-here_" --format json \
| jq -r '.bindings[] | select(.role == "roles/owner") | .members[]' \
| awk -F':' '{print $2}'

[shagabutdinov]

Found out that with api version apiVersion: v1 following should be used:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: logs

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluentd
  namespace: logs
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch

[AkshayDubey29]

@rotmond I am also facing the same issue now, can you tell me how did you resolve the same.

config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Exception encountered fetching metadata from Kubernetes API endpoint: pods is forbidden: User \"system:serviceaccount:kube-system:default\" cannot list pods at the cluster scope ({\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"pods is forbidden: User \\"system:serviceaccount:kube-system:default\\" cannot list pods at the cluster scope\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"pods\"},\"code\":403}\n)"

[edsiper]

Fixed by #89