Calico: stat /var/lib/calico/nodename: no such file or directory problem,please help.

[zong400]

Hi, here is a problem in my kubernetes cluster, in the node wx3, I want to create a static pod named jenkins, but kubelet make error log over and over.

E0322 15:59:06.016063 1239 kuberuntime_gc.go:152] Failed to stop sandbox "420698bd9963f65496a5fd0c127f2b23497d678ddcf58362aa35615d8739d372" before removing: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "jenkins-wx3_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ W0322 15:59:14.922384 1239 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available I0322 15:59:17.649057 1239 kuberuntime_manager.go:389] No ready sandbox for pod "jenkins-wx3_default(1d947eff714cafbfcc78ef0291db3291)" can be found. Need to start a new one W0322 15:59:17.651466 1239 cni.go:265] CNI failed to retrieve network namespace path: Cannot find network namespace for the terminated container "aaf3954dc74a610b5da9cfbbcf67d413b64ee49f00d5df0835fb7f340449181b" E0322 15:59:17.756783 1239 cni.go:319] Error deleting network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ E0322 15:59:17.757482 1239 remote_runtime.go:115] StopPodSandbox "aaf3954dc74a610b5da9cfbbcf67d413b64ee49f00d5df0835fb7f340449181b" from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod "jenkins-wx3_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ E0322 15:59:17.757520 1239 kuberuntime_manager.go:781] Failed to stop sandbox {"docker" "aaf3954dc74a610b5da9cfbbcf67d413b64ee49f00d5df0835fb7f340449181b"} E0322 15:59:17.757568 1239 kuberuntime_manager.go:581] killPodWithSyncResult failed: failed to "KillPodSandbox" for "1d947eff714cafbfcc78ef0291db3291" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"jenkins-wx3_default\" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/" E0322 15:59:17.757597 1239 pod_workers.go:182] Error syncing pod 1d947eff714cafbfcc78ef0291db3291 ("jenkins-wx3_default(1d947eff714cafbfcc78ef0291db3291)"), skipping: failed to "KillPodSandbox" for "1d947eff714cafbfcc78ef0291db3291" with KillPodSandboxError: "rpc error: code = Unknown desc = NetworkPlugin cni failed to teardown pod \"jenkins-wx3_default\" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"

when I put the jenkins.yml to wx1, everything ok.
how can I fix it ?

Your Environment

~ # calicoctl version
Client Version: v2.0.1
Build date: 2018-02-23T23:37:37+0000
Git commit: 5fa93655
Cluster Version: v3.0.1-218-gb3b47737
Cluster Type: k8s,bgp

~ # calicoctl get node -o wide
NAME ASN IPV4 IPV6
wx (unknown) 192.168.21.55/24
wx1 (unknown) 192.168.21.56/24
wx3 (unknown) 192.168.21.11/24

~ # calicoctl get workloadEndpoint -o wide
NAME WORKLOAD NODE NETWORKS INTERFACE PROFILES NATS
wx-k8s-dnsmasq--dep--844fb9f48d--wr4qp-eth0 dnsmasq-dep-844fb9f48d-wr4qp wx 172.50.56.6/32 cali3aeaee8bcfc kns.default
wx-k8s-nfsd--555cf7c46b--9q9q9-eth0 nfsd-555cf7c46b-9q9q9 wx 172.50.56.61/32 calie9a5b3f1744 kns.default
wx-k8s-nginx--deployment--77c45bd648--xb2r5-eth0 nginx-deployment-77c45bd648-xb2r5 wx 172.50.56.60/32 cali44402d20873 kns.default
wx-k8s-spark--master-eth0 spark-master wx 172.50.56.63/32 cali54d44e2d0ac kns.default
wx-k8s-spark--slave1-eth0 spark-slave1 wx 172.50.56.2/32 cali9a2eec147dd kns.default
wx-k8s-spark--slave2-eth0 spark-slave2 wx 172.50.56.1/32 cali80f72bad764 kns.default
wx-k8s-spark--slave3-eth0 spark-slave3 wx 172.50.56.5/32 caliac3052224a9 kns.default
wx-k8s-tomcat7--dep--74bf5b7d88--smq2n-eth0 tomcat7-dep-74bf5b7d88-smq2n wx 172.50.56.62/32 cali6c038e3b06b kns.default
wx-k8s-zk3--wx-eth0 zk3-wx wx 172.50.56.7/32 cali8f4bab72ef5 kns.default
wx1-k8s-busybox-eth0 busybox wx1 172.50.255.150/32 cali12d4a061371 kns.default
wx1-k8s-dnsmasq--dep--77bb7f589f--vzbb5-eth0 dnsmasq-dep-77bb7f589f-vzbb5 wx1 172.50.255.169/32 cali1c838e89bdd kns.default
wx1-k8s-hadoop--client-eth0 hadoop-client wx1 172.50.255.152/32 calid54dec8afc4 kns.default
wx1-k8s-hadoop--httpfs--8f757b8cc--qh8zm-eth0 hadoop-httpfs-8f757b8cc-qh8zm wx1 172.50.255.167/32 cali6994c0f1574 kns.default
wx1-k8s-hadoop--httpfs--8f757b8cc--rdt6c-eth0 hadoop-httpfs-8f757b8cc-rdt6c wx1 172.50.255.146/32 cali95554e22362 kns.default
wx1-k8s-nginx--deployment--77c45bd648--n598x-eth0 nginx-deployment-77c45bd648-n598x wx1 172.50.255.153/32 cali16e6132bd14 kns.default
wx1-k8s-nginx--deployment--77c45bd648--zv786-eth0 nginx-deployment-77c45bd648-zv786 wx1 172.50.255.159/32 calid24d442f2ea kns.default
wx1-k8s-tomcat7--dep--74bf5b7d88--4hpfr-eth0 tomcat7-dep-74bf5b7d88-4hpfr wx1 172.50.255.163/32 calib89ca8a389d kns.default
wx1-k8s-tomcat7--dep--74bf5b7d88--8sbjb-eth0 tomcat7-dep-74bf5b7d88-8sbjb wx1 172.50.255.149/32 cali98af15efd2b kns.default
wx1-k8s-tomcat7--dep--74bf5b7d88--9htnx-eth0 tomcat7-dep-74bf5b7d88-9htnx wx1 172.50.255.151/32 cali893197594b5 kns.default
wx1-k8s-tomcat7--dep--74bf5b7d88--qcn9f-eth0 tomcat7-dep-74bf5b7d88-qcn9f wx1 172.50.255.162/32 cali93dfdd66d35 kns.default
wx1-k8s-zk2--wx1-eth0 zk2-wx1 wx1 172.50.255.157/32 cali36493d30616 kns.default

ubuntu@ubuntu1:~$ sudo kubectl describe po jenkins-wx3
Name: jenkins-wx3
Namespace: default
Node: wx3/192.168.21.11
Start Time: Thu, 22 Mar 2018 15:42:03 +0800
Labels: app=jenkins
Annotations: kubernetes.io/config.hash=1d947eff714cafbfcc78ef0291db3291
kubernetes.io/config.mirror=1d947eff714cafbfcc78ef0291db3291
kubernetes.io/config.seen=2018-03-22T15:42:03.107778114+08:00
kubernetes.io/config.source=file
Status: Pending
IP:
Containers:
jenkins:
Container ID:
Image: jenkins:alpine
Image ID:
Ports: 8080/TCP, 50000/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
Mounts:
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
QoS Class: BestEffort
Node-Selectors:
Tolerations: :NoExecute
Events:

[tmjd]

Are you using rc versions of any components? Specifically I'm wondering if you are using an release candidate of the CNI plugin but have not updated calico-node.
What version of the calico/node are your running and what version of calico/cni are you using?

[tmjd]

How did you install your calico components?
With the new v3.1.0-rc1 components it is necessary to mount /var/lib/calico to the calico-node container so that it can write the file /var/lib/calico/nodename that is being referenced in the logs you reported. If you installed from one of the master docs manifests from the docs site please let us know so we can update it.

[zong400]

Hi, @tmjd ,this is my calico yaml, a DaemonSet of k8s, image is quay.io/calico/node:master

ubuntu@ubuntu1:~$ sudo kubectl get ds calico-node -o yaml -n kube-system
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  creationTimestamp: 2017-12-12T06:44:39Z
  generation: 3
  labels:
    k8s-app: calico-node
  name: calico-node
  namespace: kube-system
  resourceVersion: "11801514"
  selfLink: /apis/extensions/v1beta1/namespaces/kube-system/daemonsets/calico-node
  uid: ecf5370f-df07-11e7-9cdd-00e066414888
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: calico-node
  template:
    metadata:
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ""
        scheduler.alpha.kubernetes.io/tolerations: |
          [{"key": "dedicated", "value": "master", "effect": "NoSchedule" },
           {"key":"CriticalAddonsOnly", "operator":"Exists"}]
      creationTimestamp: null
      labels:
        k8s-app: calico-node
    spec:
      containers:
      - env:
        - name: ETCD_ENDPOINTS
          valueFrom:
            configMapKeyRef:
              key: etcd_endpoints
              name: calico-config
        - name: CALICO_NETWORKING_BACKEND
          valueFrom:
            configMapKeyRef:
              key: calico_backend
              name: calico-config
        - name: CLUSTER_TYPE
          value: k8s,bgp
        - name: CALICO_DISABLE_FILE_LOGGING
          value: "true"
        - name: CALICO_K8S_NODE_REF
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
          value: ACCEPT
        - name: CALICO_IPV4POOL_CIDR
          value: 172.50.0.0/16
        - name: CALICO_IPV4POOL_IPIP
          value: cross-subnet
        - name: FELIX_IPV6SUPPORT
          value: "false"
        - name: FELIX_LOGSEVERITYSCREEN
          value: info
        - name: FELIX_IPINIPMTU
          value: "1440"
        - name: ETCD_CA_CERT_FILE
          valueFrom:
            configMapKeyRef:
              key: etcd_ca
              name: calico-config
        - name: ETCD_KEY_FILE
          valueFrom:
            configMapKeyRef:
              key: etcd_key
              name: calico-config
        - name: ETCD_CERT_FILE
          valueFrom:
            configMapKeyRef:
              key: etcd_cert
              name: calico-config
        - name: IP
        - name: FELIX_HEALTHENABLED
          value: "true"
        image: quay.io/calico/node:master
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          httpGet:
            path: /liveness
            port: 9099
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: calico-node
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readiness
            port: 9099
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 250m
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /lib/modules
          name: lib-modules
          readOnly: true
        - mountPath: /var/run/calico
          name: var-run-calico
        - mountPath: /calico-secrets
          name: etcd-certs
      - command:
        - /install-cni.sh
        env:
        - name: CNI_CONF_NAME
          value: 10-calico.conflist
        - name: ETCD_ENDPOINTS
          valueFrom:
            configMapKeyRef:
              key: etcd_endpoints
              name: calico-config
        - name: CNI_NETWORK_CONFIG
          valueFrom:
            configMapKeyRef:
              key: cni_network_config
              name: calico-config
        image: quay.io/calico/cni:master
        imagePullPolicy: IfNotPresent
        name: install-cni
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /host/opt/cni/bin
          name: cni-bin-dir
        - mountPath: /host/etc/cni/net.d
          name: cni-net-dir
        - mountPath: /calico-secrets
          name: etcd-certs
      dnsPolicy: ClusterFirst
      hostNetwork: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: calico-node
      serviceAccountName: calico-node
      terminationGracePeriodSeconds: 0
      volumes:
      - hostPath:
          path: /lib/modules
          type: ""
        name: lib-modules
      - hostPath:
          path: /var/run/calico
          type: ""
        name: var-run-calico
      - hostPath:
          path: /opt/cni/bin
          type: ""
        name: cni-bin-dir
      - hostPath:
          path: /etc/cni/net.d
          type: ""
        name: cni-net-dir
      - name: etcd-certs
        secret:
          defaultMode: 420
          secretName: calico-etcd-secrets
  templateGeneration: 3
  updateStrategy:
    type: OnDelete
status:
  currentNumberScheduled: 3
  desiredNumberScheduled: 3
  numberAvailable: 3
  numberMisscheduled: 0
  numberReady: 3
  observedGeneration: 3
  updatedNumberScheduled: 3

ubuntu@ubuntu1:~$ sudo kubectl get po calico-node-vb6gj -o yaml -n kube-system
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"DaemonSet","namespace":"kube-system","name":"calico-node","uid":"ecf5370f-df07-11e7-9cdd-00e066414888","apiVersion":"extensions","resourceVersion":"11793622"}}
    scheduler.alpha.kubernetes.io/critical-pod: ""
    scheduler.alpha.kubernetes.io/tolerations: |
      [{"key": "dedicated", "value": "master", "effect": "NoSchedule" },
       {"key":"CriticalAddonsOnly", "operator":"Exists"}]
  creationTimestamp: 2018-03-22T02:29:06Z
  generateName: calico-node-
  labels:
    controller-revision-hash: "3331114009"
    k8s-app: calico-node
    pod-template-generation: "3"
  name: calico-node-vb6gj
  namespace: kube-system
  ownerReferences:
  - apiVersion: extensions/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: DaemonSet
    name: calico-node
    uid: ecf5370f-df07-11e7-9cdd-00e066414888
  resourceVersion: "11801513"
  selfLink: /api/v1/namespaces/kube-system/pods/calico-node-vb6gj
  uid: cb0cce48-2d78-11e8-ad73-00e066414888
spec:
  containers:
  - env:
    - name: ETCD_ENDPOINTS
      valueFrom:
        configMapKeyRef:
          key: etcd_endpoints
          name: calico-config
    - name: CALICO_NETWORKING_BACKEND
      valueFrom:
        configMapKeyRef:
          key: calico_backend
          name: calico-config
    - name: CLUSTER_TYPE
      value: k8s,bgp
    - name: CALICO_DISABLE_FILE_LOGGING
      value: "true"
    - name: CALICO_K8S_NODE_REF
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
      value: ACCEPT
    - name: CALICO_IPV4POOL_CIDR
      value: 172.50.0.0/16
    - name: CALICO_IPV4POOL_IPIP
      value: cross-subnet
    - name: FELIX_IPV6SUPPORT
      value: "false"
    - name: FELIX_LOGSEVERITYSCREEN
      value: info
    - name: FELIX_IPINIPMTU
      value: "1440"
    - name: ETCD_CA_CERT_FILE
      valueFrom:
        configMapKeyRef:
          key: etcd_ca
          name: calico-config
    - name: ETCD_KEY_FILE
      valueFrom:
        configMapKeyRef:
          key: etcd_key
          name: calico-config
    - name: ETCD_CERT_FILE
      valueFrom:
        configMapKeyRef:
          key: etcd_cert
          name: calico-config
    - name: IP
    - name: FELIX_HEALTHENABLED
      value: "true"
    image: quay.io/calico/node:master
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 6
      httpGet:
        path: /liveness
        port: 9099
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: calico-node
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /readiness
        port: 9099
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      requests:
        cpu: 250m
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /lib/modules
      name: lib-modules
      readOnly: true
    - mountPath: /var/run/calico
      name: var-run-calico
    - mountPath: /calico-secrets
      name: etcd-certs
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: calico-node-token-wncgc
      readOnly: true
  - command:
    - /install-cni.sh
    env:
    - name: CNI_CONF_NAME
      value: 10-calico.conflist
    - name: ETCD_ENDPOINTS
      valueFrom:
        configMapKeyRef:
          key: etcd_endpoints
          name: calico-config
    - name: CNI_NETWORK_CONFIG
      valueFrom:
        configMapKeyRef:
          key: cni_network_config
          name: calico-config
    image: quay.io/calico/cni:master
    imagePullPolicy: IfNotPresent
    name: install-cni
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /host/opt/cni/bin
      name: cni-bin-dir
    - mountPath: /host/etc/cni/net.d
      name: cni-net-dir
    - mountPath: /calico-secrets
      name: etcd-certs
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: calico-node-token-wncgc
      readOnly: true
  dnsPolicy: ClusterFirst
  hostNetwork: true
  nodeName: wx3
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: calico-node
  serviceAccountName: calico-node
  terminationGracePeriodSeconds: 0
  tolerations:
  - effect: NoExecute
    key: node.alpha.kubernetes.io/notReady
    operator: Exists
  - effect: NoExecute
    key: node.alpha.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  volumes:
  - hostPath:
      path: /lib/modules
      type: ""
    name: lib-modules
  - hostPath:
      path: /var/run/calico
      type: ""
    name: var-run-calico
  - hostPath:
      path: /opt/cni/bin
      type: ""
    name: cni-bin-dir
  - hostPath:
      path: /etc/cni/net.d
      type: ""
    name: cni-net-dir
  - name: etcd-certs
    secret:
      defaultMode: 420
      secretName: calico-etcd-secrets
  - name: calico-node-token-wncgc
    secret:
      defaultMode: 420
      secretName: calico-node-token-wncgc
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2018-03-22T02:29:11Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2018-03-22T03:54:27Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2018-03-22T02:30:03Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://241393942aa6815f708798d9c76fd476ad542f767eccc6294bcde612ca1c2301
    image: quay.io/calico/node:master
    imageID: docker-pullable://quay.io/calico/node@sha256:742875b3e0d16ddcff5eee80011dc43db4016d76cf2d4c9c4a9bacd72e87d012
    lastState: {}
    name: calico-node
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2018-03-22T03:53:45Z
  - containerID: docker://e3ea81d980f1eb1c00a4e868f177e263c995334938e7315ce74ba34a8afe7456
    image: quay.io/calico/cni:master
    imageID: docker-pullable://quay.io/calico/cni@sha256:9f30f6e2f81bd5757ee48ed64f1e7212fbc5c9098c112833932c4e15e9372d6f
    lastState: {}
    name: install-cni
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2018-03-22T03:54:23Z
  hostIP: 192.168.21.11
  phase: Running
  podIP: 192.168.21.11
  qosClass: Burstable
  startTime: 2018-03-22T02:29:11Z

[tmjd]

If you take a look at the latest manifest https://docs.projectcalico.org/master/getting-started/kubernetes/installation/hosted/calico.yaml you'll see that there is a volume mount of /var/lib/calico. You need to either add that mount to your manifest or switch to using the latest manifest.

[zong400]

Thanks @tmjd ,what 's the difference between master and latest?
in the DaemonSet, should i change master to a specific version like v3.0.1 ?
i don't want to change the version when i add a new node even pass a long time

[tmjd]

Sorry by latest I meant master. The one I linked to https://docs.projectcalico.org/master/getting-started/kubernetes/installation/hosted/calico.yaml. (You can look at the master docs by selecting 'nightly' in the Version drop down on the docs site.)

I would suggest you use one of the released manifests, it will have specific versions of the Calico components that have been tested together, if you are looking for a good and consistent experience with Calico. The master versions are updated regularly and, as you have ran in to, the manifests could be mis-matched with the master container image versions.

[caseydavenport]

what 's the difference between master and latest?

Just to clarify this - master is the latest build of the code from the master branch, and isn't guaranteed to be stable.

latest points to the latest stable release.

I'd still recommend pinning to a specific release to avoid pulling in unexpected changes.

[zong400]

@tmjd @caseydavenport very helpful and thanks

[ggaurav10]

i am getting a similar error when deploying a k8s cluster. Below are the kubelet log snippet:

E0518 09:33:50.990832 19864 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-autoscaler-6966fd6fb6-29lqj_kube-system(4c2a3a53-5a64-11e8-b577-daa39fff8710)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-autoscaler-6966fd6fb6-29lqj_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ E0518 09:33:50.990858 19864 kuberuntime_manager.go:647] createPodSandbox for pod "kube-dns-autoscaler-6966fd6fb6-29lqj_kube-system(4c2a3a53-5a64-11e8-b577-daa39fff8710)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-autoscaler-6966fd6fb6-29lqj_kube-system" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/ E0518 09:33:50.990928 19864 pod_workers.go:186] Error syncing pod 4c2a3a53-5a64-11e8-b577-daa39fff8710 ("kube-dns-autoscaler-6966fd6fb6-29lqj_kube-system(4c2a3a53-5a64-11e8-b577-daa39fff8710)"), skipping: failed to "CreatePodSandbox" for "kube-dns-autoscaler-6966fd6fb6-29lqj_kube-system(4c2a3a53-5a64-11e8-b577-daa39fff8710)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-dns-autoscaler-6966fd6fb6-29lqj_kube-system(4c2a3a53-5a64-11e8-b577-daa39fff8710)\" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod \"kube-dns-autoscaler-6966fd6fb6-29lqj_kube-system\" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/"

The daemonset calico-node has the /var/lib/calico volume and volumemount, which i can verify by exec-ing into the calico/node pod and reading the file /var/lib/calico/nodename

The images used for calico/node and calico/cni:
image: quay.io/calico/node:v3.1.0
image: quay.io/calico/cni:v3.1.0

Please let me know if I am missing anything.

Thanks.

[caseydavenport]

@ggaurav10 do you see the /var/lib/calico/nodename file on the host filesystem?

Also, are you running a containerized kubelet by chance? If so, you'll also need to mount that directory into the kubelet container so that the CNI plugin can see it.

[ggaurav10]

thanks for the response.
yes. i can see the file on the host, and yes, the kubelet is running in a container.
Mounting the directory in the kubelet container solved the issue. :)

Thanks again.

[r7vme]

Whoever will be struggling with the same error and it's not always quick to upgrade kubelet config (add /var/lib/calico mount) on all clusters. There is a compatibility mode if calico nodename == hostname.

Add to configmap

"nodename_file_optional": true,

So final cni_network_config looks like that:

            cni_network_config: |-
              {    
                "name": "k8s-pod-network",
                "cniVersion": "0.3.0",
                "plugins": [
                  {    
                    "type": "calico",
                    "log_level": "info",
                    "etcd_endpoints": "__ETCD_ENDPOINTS__",
                    "etcd_key_file": "__ETCD_KEY_FILE__",
                    "etcd_cert_file": "__ETCD_CERT_FILE__",
                    "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
                    "mtu": __CNI_MTU__,
                    "nodename_file_optional": true,
                    "ipam": {
                        "type": "calico-ipam"
                    },   
                    "policy": {
                        "type": "k8s"
                    },   
                    "kubernetes": {
                        "kubeconfig": "__KUBECONFIG_FILEPATH__"
                    }    
                  },   
                  {    
                    "type": "portmap",
                    "snat": true,
                    "capabilities": {"portMappings": true}
                  }    
                ]    
              } 

In this case, for nodes w/o /var/lib/calico in kubelet CNI plugin will use hostname, for nodes with mount it will use /var/lib/calico/nodename file.

[tmjd]

@r7vme

it's not always quick to upgrade kubelet config (add /var/lib/calico mount)

Could you explain why that is needed in your case? Is your kubelet being run with rkt or something like that (where it doesn't have full host filesystem access? Lots of K8s deployments run the kubelet as a service I believe, and there would be no need for any kubelet config changes.

[r7vme]

Could you explain why that is needed in your case?

We run kubelet in docker container, so i need to provide access to /var/lib/calico host path. It isn't easy not from config change perspective, but from perspective of releasing two dependant changes. I need to make sure all our customers updated to release with mount, before i can apply new calico. All doable, but nodename_file_optional makes it possible to release new calico in single step. We already discussed changes and it's completely safe procedure, because nodename will be fetched by calling hostname only when master already upgraded (applied new calico manifest), but worker still not. When worker will be rolled out with kubelet change (mount /var/lib/calico) CNI immediately will start using /var/lib/calico/nodename file. In total it's about 1 hour from our experience. Bam! :)

[bhargavtruecom]

@r7vme

Facing same issue.

My calico.yml file is https://docs.projectcalico.org/v3.5/getting-started/kubernetes/installation/hosted/calico.yaml

Error

Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b577ddbdd5fbd6cbe79e5b1bf20648e981590ecd0df545a0158ce909d9179096" network for pod "frontend-784f75ddb7-nbz7t": NetworkPlugin cni failed to set up pod "frontend-784f75ddb7-nbz7t_default" network: stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kubectl get pods --all-namespaces

````
NAMESPACE NAME READY STATUS RESTARTS AGE
default frontend-784f75ddb7-nbz7t 0/1 ContainerCreating 0 91m
default redis-master-97979696c-hcgdm 0/1 ContainerCreating 0 91m
default redis-slave-6fd879d46c-klp4r 0/1 ContainerCreating 0 91m
default ripple-app-dashboard-58d49bb867-wj44k 0/1 ContainerCreating 0 110m
kube-system calico-etcd-b7wqf 1/1 Running 0 143m
kube-system calico-kube-controllers-74887d7bdf-wxhkd 1/1 Running 0 144m
kube-system calico-node-58fqj 1/1 Running 0 144m
kube-system calico-node-mchcc 0/1 CrashLoopBackOff 25 100m
kube-system coredns-86c58d9df4-7ncdk 1/1 Running 0 158m
kube-system coredns-86c58d9df4-g4jcp 1/1 Running 0 158m
kube-system etcd-kmaster 1/1 Running 0 157m
kube-system kube-apiserver-kmaster 1/1 Running 0 157m
kube-system kube-controller-manager-kmaster 1/1 Running 0 157m
kube-system kube-proxy-njx5c 1/1 Running 0 137m
kube-system kube-proxy-pkxx5 1/1 Running 0 158m
kube-system kube-scheduler-kmaster 1/1 Running 0 157m
kube-system kubernetes-dashboard-57df4db6b-zcvcc 1/1 Running 0 141m

````

Kubercates version
v1.13