kubectl describe -n default po kube-flannel-ds-tjhx5
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
run:
Type: HostPath (bare host directory volume)
Path: /run
cni:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
flannel-cfg:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-flannel-cfg
Optional: false
flannel-token-w1tz3:
Type: Secret (a volume populated by a Secret)
SecretName: flannel-token-w1tz3
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/arch=amd64
Tolerations: node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
1m 1m 1 kubelet, master spec.containers{kube-flannel} Normal Created Created container with id e1233499873cf87c5307cbc54fe73e2101ea2c519f59651b64121111f4245928
1m 1m 1 kubelet, master spec.containers{kube-flannel} Normal Started Started container with id e1233499873cf87c5307cbc54fe73e2101ea2c519f59651b64121111f4245928
1m 1m 1 kubelet, master spec.containers{install-cni} Normal Created Created container with id f7632878416353606590536277fc6e2a3f6ef72cfd464a28b0dc9489ac179352
1m 1m 1 kubelet, master spec.containers{install-cni} Normal Started Started container with id f7632878416353606590536277fc6e2a3f6ef72cfd464a28b0dc9489ac179352
1m 1m 1 kubelet, master spec.containers{kube-flannel} Normal Created Created container with id 322aca8eaba8427519138e3d10162b734bf35475598ac52fbc7f47df4fdd7c02
1m 1m 1 kubelet, master spec.containers{kube-flannel} Normal Started Started container with id 322aca8eaba8427519138e3d10162b734bf35475598ac52fbc7f47df4fdd7c02
1m 1m 2 kubelet, master Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kube-flannel" with CrashLoopBackOff: "Back-off 10s restarting failed container=kube-flannel pod=kube-flannel-ds-tjhx5_default(51712f7f-2030-11e7-aa7b-000c292d1d85)"
1m 1m 1 kubelet, master spec.containers{kube-flannel} Normal Started Started container with id 8f4cfac604e3f35819c541943d26b33c96d36cf95fce8e39adcc32f3f252ae2b
1m 1m 1 kubelet, master spec.containers{kube-flannel} Normal Created Created container with id 8f4cfac604e3f35819c541943d26b33c96d36cf95fce8e39adcc32f3f252ae2b
1m 50s 2 kubelet, master Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kube-flannel" with CrashLoopBackOff: "Back-off 20s restarting failed container=kube-flannel pod=kube-flannel-ds-tjhx5_default(51712f7f-2030-11e7-aa7b-000c292d1d85)"
37s 37s 1 kubelet, master spec.containers{kube-flannel} Normal Started Started container with id 8b9190d85518c80f1f6f17a2f071c466e59ad2d70eeffa27f5b651efb037a51e
1m 37s 5 kubelet, master spec.containers{kube-flannel} Normal Pulled Container image "quay.io/coreos/flannel:v0.7.0-amd64" already present on machine
37s 37s 1 kubelet, master spec.containers{kube-flannel} Normal Created Created container with id 8b9190d85518c80f1f6f17a2f071c466e59ad2d70eeffa27f5b651efb037a51e
1m 12s 7 kubelet, master spec.containers{kube-flannel} Warning BackOff Back-off restarting failed container
35s 12s 3 kubelet, master Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "kube-flannel" with CrashLoopBackOff: "Back-off 40s restarting failed container=kube-flannel pod=kube-flannel-ds-tjhx5_default(51712f7f-2030-11e7-aa7b-000c292d1d85)"
and this is my manifests
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"type": "flannel",
"delegate": {
"isDefaultGateway": true
}
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "udp"
}
}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds
labels:
tier: node
app: flannel
spec:
template:
metadata:
labels:
tier: node
app: flannel
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
serviceAccountName: flannel
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.7.0-amd64
command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
securityContext:
privileged: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run
- name: flannel-cfg
mountPath: /etc/kube-flannel/
- name: install-cni
image: quay.io/coreos/flannel:v0.7.0-amd64
command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
Below is all images used by kubeadm
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64 1.14.1 a147db3998d8 7 days ago 44.84 MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64 1.14.1 a4ce3166e6c7 7 days ago 52.36 MB
gcr.io/google_containers/k8s-dns-sidecar-amd64 1.14.1 794e58ca73ca 7 days ago 44.52 MB
gcr.io/google_containers/etcd-amd64 3.0.17 a1bd75caef62 7 days ago 168.9 MB
gcr.io/google_containers/kube-proxy-amd64 v1.6.0 402a9d5eef82 7 days ago 109.2 MB
gcr.io/google_containers/pause-amd64 3.0 1a7cadfe945d 7 days ago 746.9 kB
gcr.io/google_containers/kube-scheduler-amd64 v1.6.0 05bd5dd1634d 7 days ago 76.75 MB
gcr.io/google_containers/kube-controller-manager-amd64 v1.6.0 2e90cc5aac1f 7 days ago 132.7 MB
gcr.io/google_containers/kube-apiserver-amd64 v1.6.0 8063f60cd9d9 7 days ago 150.5 MB
gcr.io/google_containers/kubernetes-dashboard-amd64 v1.6.0 2b33fc9a20e1 13 days ago 108.6 MB
quay.io/coreos/flannel v0.7.0-amd64 0ddb384f4496 4 weeks ago 73.75 MB
cc @luxas @jbeda @lukemarsden
@TracyBin Can you provide the logs from the flannel container?
@tomdee hi, this is logs of the flannel container
[root@slave1 containers]# tail -f /var/log/pods/fbc13661-20bd-11e7-b15d-000c293e1335/kube-flannel_3.log
{"log":"E0414 02:56:51.415549 1 main.go:127] Failed to create SubnetManager: error retrieving pod spec for 'default/kube-flannel-ds-kz6jl': the server does not allow access to the requested resource (get pods kube-flannel-ds-kz6jl)\n","stream":"stderr","time":"2017-04-14T02:56:51.418396354Z"}
This look like a rbac permission error.New feature in k8s 1.6.0.
issues
kube-flannel.yml already requires k8s v1.6+, you should really add the RBAC contents to the main manifest file. Forcing the user to apply two manifests when everyone else has one isn't optimal.
Is there a workaround of this issue right now?
@singhsankalp The solution with Kubernetes v1.6.1 is to create the cluster role and role bindings first, as described in the header here.
Or just execute:
curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml | kubectl create -f -
curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml | kubectl create -f -
Looks like this was fixed with https://github.com/coreos/flannel/pull/690, and this issue should be addressed as a release note, not a troubleshooting item.
Most helpful comment
@singhsankalp The solution with Kubernetes v1.6.1 is to create the cluster role and role bindings first, as described in the header here.
Or just execute: