Calico: CrashLoopBackOff for calico-node pods - Kubernetes local install

Created on 27 Dec 2019 · 7Comments · Source: projectcalico/calico

I am trying to setup a multi node cluster on my ubuntu 18.04.
After running the kubeadm init command when I tried to install a pod network, the pods wouldn't come up as Running. Even the coredns pods are stuck in ContainerCreating mode.

Also I have matched the ipaddress which I passed in while running the init command and the one on calico.yaml file.
kubeadm init --pod-network-cidr=192.168.0.0/16

The logs from calico-node pods are below :

kubectl logs calico-node-np22b -n kube-system

2019-12-27 21:42:58.366 [INFO][8] customresource.go 101: Error getting resource Key=GlobalFelixConfig(name=CalicoVersion) Name="calicoversion" Resource="GlobalFelixConfigs" error=the server could not find the requested resource (get GlobalFelixConfigs.crd.projectcalico.org calicoversion)
2019-12-27 21:42:58.372 [INFO][8] startup.go 385: Initialize BGP data
2019-12-27 21:42:58.373 [WARNING][8] startup.go 617: Unable to auto-detect an IPv4 address: no valid IPv4 addresses found on the host interfaces
2019-12-27 21:42:58.373 [WARNING][8] startup.go 407: Couldn't autodetect an IPv4 address. If auto-detecting, choose a different autodetection method. Otherwise provide an explicit address.
2019-12-27 21:42:58.373 [INFO][8] startup.go 213: Clearing out-of-date IPv4 address from this node IP=""
2019-12-27 21:42:58.376 [WARNING][8] startup.go 1122: Terminating
Calico node failed to start

➜ ~ kubectl get pods --all-namespaces --watch
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-648f4868b8-wxmw8 0/1 ContainerCreating 0 8m52s
kube-system calico-node-np22b 0/1 CrashLoopBackOff 6 8m52s
kube-system coredns-6955765f44-dzp2m 0/1 ContainerCreating 0 13m
kube-system coredns-6955765f44-rp88k 0/1 ContainerCreating 0 13m
kube-system etcd-broadsword 1/1 Running 0 13m
kube-system kube-apiserver-broadsword 1/1 Running 0 13m
kube-system kube-controller-manager-broadsword 1/1 Running 0 13m
kube-system kube-proxy-8pvml 1/1 Running 0 13m
kube-system kube-scheduler-broadsword 1/1 Running 0 13m

Can someone please help in this problem. I have tried searching for this issue but it doesn't look like going away.

Environment:
Calico version-3.11
Kubernetes version v1.17.0
Ubuntu 18.04

kinsupport

Source

abhishakebansal

Most helpful comment

@caseydavenport I think our default exclusion logic is too greedy. If you take a look at the default interfaces we exclude https://github.com/projectcalico/node/blob/master/pkg/startup/startup_linux.go#L12, you'll see "lo" as one of them. Also look at where that is used https://github.com/projectcalico/node/blob/master/pkg/startup/autodetection/interfaces.go#L54 and you may see the problem, we don't anchor the start and end of the string for matching so the 'lo' will match your wlo1 interface which will cause node to skip using it.
I've also tested this out here https://play.golang.org/p/LH3B9qI8_SJ.
@abhishakebansal I think you can work around this by changing the node ip autodetection in your manifest, see the docs here https://docs.projectcalico.org/v3.11/reference/node/configuration#ip-autodetection-methods. I think you'll need IP6_AUTODETECTION_METHOD=interface=wlo.*, this is assuming all your hosts are the same, if not then you can specify multiple interface regexs by separating them with a comma (,).

tmjd on 7 Jan 2020

🚀1 ❤1 🎉1 👍1

All 7 comments

2019-12-27 21:42:58.373 [WARNING][8] startup.go 617: Unable to auto-detect an IPv4 address: no valid IPv4 addresses found on the host interfaces

It looks like Calico is failing to find a v4 address on your node. Are you running in an IPv6 only cluster?

caseydavenport on 2 Jan 2020

No! I am not. I specified the ip address 192.168.0.0/16 on calico.yaml as well as on the command below.
kubeadm init --pod-network-cidr=192.168.0.0/16
Not sure what is the problem here.

abhishakebansal on 3 Jan 2020

Could you provide an ifconfig or ip a from one of your hosts?

tmjd on 6 Jan 2020

@tmjd

➜  ~ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 00:d8:61:9f:e7:93 brd ff:ff:ff:ff:ff:ff
3: wlo1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether dc:71:96:46:83:e5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.159/24 brd 192.168.1.255 scope global dynamic noprefixroute wlo1
       valid_lft 86290sec preferred_lft 86290sec
    inet6 fe80::1005:5fa7:e3cf:2ff0/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:76:ed:5e:24 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

abhishakebansal on 7 Jan 2020

tmjd on 7 Jan 2020

🚀1 ❤1 🎉1 👍1

@tmjd @caseydavenport Its running fine now with the entry you suggested on calico.yaml

Thanks a ton!

abhishakebansal on 15 Jan 2020

@abhishakebansal I don't think closing this was a good idea as the greedy matching of network interfaces with "lo" in it still exists:
https://github.com/projectcalico/node/blob/master/pkg/startup/startup_linux.go#L31