flannel in kubeadm can't be started

Created on 13 Apr 2017  路  8Comments  路  Source: coreos/flannel

kubectl describe -n default po kube-flannel-ds-tjhx5

Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  run:
    Type:   HostPath (bare host directory volume)
    Path:   /run
  cni:
    Type:   HostPath (bare host directory volume)
    Path:   /etc/cni/net.d
  flannel-cfg:
    Type:   ConfigMap (a volume populated by a ConfigMap)
    Name:   kube-flannel-cfg
    Optional:   false
  flannel-token-w1tz3:
    Type:   Secret (a volume populated by a Secret)
    SecretName: flannel-token-w1tz3
    Optional:   false
QoS Class:  BestEffort
Node-Selectors: beta.kubernetes.io/arch=amd64
Tolerations:    node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
  FirstSeen LastSeen    Count   From        SubObjectPath           Type        Reason      Message
  --------- --------    -----   ----        -------------           --------    ------      -------
  1m        1m      1   kubelet, master spec.containers{kube-flannel}   Normal      Created     Created container with id e1233499873cf87c5307cbc54fe73e2101ea2c519f59651b64121111f4245928
  1m        1m      1   kubelet, master spec.containers{kube-flannel}   Normal      Started     Started container with id e1233499873cf87c5307cbc54fe73e2101ea2c519f59651b64121111f4245928
  1m        1m      1   kubelet, master spec.containers{install-cni}    Normal      Created     Created container with id f7632878416353606590536277fc6e2a3f6ef72cfd464a28b0dc9489ac179352
  1m        1m      1   kubelet, master spec.containers{install-cni}    Normal      Started     Started container with id f7632878416353606590536277fc6e2a3f6ef72cfd464a28b0dc9489ac179352
  1m        1m      1   kubelet, master spec.containers{kube-flannel}   Normal      Created     Created container with id 322aca8eaba8427519138e3d10162b734bf35475598ac52fbc7f47df4fdd7c02
  1m        1m      1   kubelet, master spec.containers{kube-flannel}   Normal      Started     Started container with id 322aca8eaba8427519138e3d10162b734bf35475598ac52fbc7f47df4fdd7c02
  1m        1m      2   kubelet, master                 Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "kube-flannel" with CrashLoopBackOff: "Back-off 10s restarting failed container=kube-flannel pod=kube-flannel-ds-tjhx5_default(51712f7f-2030-11e7-aa7b-000c292d1d85)"

  1m    1m  1   kubelet, master spec.containers{kube-flannel}   Normal  Started     Started container with id 8f4cfac604e3f35819c541943d26b33c96d36cf95fce8e39adcc32f3f252ae2b
  1m    1m  1   kubelet, master spec.containers{kube-flannel}   Normal  Created     Created container with id 8f4cfac604e3f35819c541943d26b33c96d36cf95fce8e39adcc32f3f252ae2b
  1m    50s 2   kubelet, master                 Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "kube-flannel" with CrashLoopBackOff: "Back-off 20s restarting failed container=kube-flannel pod=kube-flannel-ds-tjhx5_default(51712f7f-2030-11e7-aa7b-000c292d1d85)"

  37s   37s 1   kubelet, master spec.containers{kube-flannel}   Normal  Started     Started container with id 8b9190d85518c80f1f6f17a2f071c466e59ad2d70eeffa27f5b651efb037a51e
  1m    37s 5   kubelet, master spec.containers{kube-flannel}   Normal  Pulled      Container image "quay.io/coreos/flannel:v0.7.0-amd64" already present on machine
  37s   37s 1   kubelet, master spec.containers{kube-flannel}   Normal  Created     Created container with id 8b9190d85518c80f1f6f17a2f071c466e59ad2d70eeffa27f5b651efb037a51e
  1m    12s 7   kubelet, master spec.containers{kube-flannel}   Warning BackOff     Back-off restarting failed container
  35s   12s 3   kubelet, master                 Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "kube-flannel" with CrashLoopBackOff: "Back-off 40s restarting failed container=kube-flannel pod=kube-flannel-ds-tjhx5_default(51712f7f-2030-11e7-aa7b-000c292d1d85)"

and this is my manifests

apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "udp"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      serviceAccountName: flannel
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      - name: install-cni
        image: quay.io/coreos/flannel:v0.7.0-amd64
        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

Below is all images used by kubeadm

gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64     1.14.1              a147db3998d8        7 days ago          44.84 MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64          1.14.1              a4ce3166e6c7        7 days ago          52.36 MB
gcr.io/google_containers/k8s-dns-sidecar-amd64           1.14.1              794e58ca73ca        7 days ago          44.52 MB
gcr.io/google_containers/etcd-amd64                      3.0.17              a1bd75caef62        7 days ago          168.9 MB
gcr.io/google_containers/kube-proxy-amd64                v1.6.0              402a9d5eef82        7 days ago          109.2 MB
gcr.io/google_containers/pause-amd64                     3.0                 1a7cadfe945d        7 days ago          746.9 kB
gcr.io/google_containers/kube-scheduler-amd64            v1.6.0              05bd5dd1634d        7 days ago          76.75 MB
gcr.io/google_containers/kube-controller-manager-amd64   v1.6.0              2e90cc5aac1f        7 days ago          132.7 MB
gcr.io/google_containers/kube-apiserver-amd64            v1.6.0              8063f60cd9d9        7 days ago          150.5 MB
gcr.io/google_containers/kubernetes-dashboard-amd64      v1.6.0              2b33fc9a20e1        13 days ago         108.6 MB
quay.io/coreos/flannel                                   v0.7.0-amd64        0ddb384f4496        4 weeks ago         73.75 MB
aredocumentation componenkubernetes

Most helpful comment

@singhsankalp The solution with Kubernetes v1.6.1 is to create the cluster role and role bindings first, as described in the header here.

Or just execute:

curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml |  kubectl create -f -
curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml |  kubectl create -f -

All 8 comments

cc @luxas @jbeda @lukemarsden

@TracyBin Can you provide the logs from the flannel container?

@tomdee hi, this is logs of the flannel container

[root@slave1 containers]# tail -f /var/log/pods/fbc13661-20bd-11e7-b15d-000c293e1335/kube-flannel_3.log
{"log":"E0414 02:56:51.415549       1 main.go:127] Failed to create SubnetManager: error retrieving pod spec for 'default/kube-flannel-ds-kz6jl': the server does not allow access to the requested resource (get pods kube-flannel-ds-kz6jl)\n","stream":"stderr","time":"2017-04-14T02:56:51.418396354Z"}

This look like a rbac permission error.New feature in k8s 1.6.0.
issues

kube-flannel.yml already requires k8s v1.6+, you should really add the RBAC contents to the main manifest file. Forcing the user to apply two manifests when everyone else has one isn't optimal.

Is there a workaround of this issue right now?

@singhsankalp The solution with Kubernetes v1.6.1 is to create the cluster role and role bindings first, as described in the header here.

Or just execute:

curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml |  kubectl create -f -
curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml |  kubectl create -f -

Looks like this was fixed with https://github.com/coreos/flannel/pull/690, and this issue should be addressed as a release note, not a troubleshooting item.

Was this page helpful?
0 / 5 - 0 ratings