Calico: Calico 3.2.1 with K8s 1.11.1: Readiness probe failed

[mrsherlock88]

I follow instruction from kubernetes to install kubeadm, and calico project. It seems to work well, but the calico-node READY becomes 1/2 after add worker node.

Current Behavior

I deploy k8s-dashboard and see the error. On the calico-node in master node:
Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 10.7.0.238, with 10.7.0.238 is worker node 's IP address.
On calico-node pod in worker node:
Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 172.26.0.1, and I don't know 172.26.0.1 is? After run kubeadm reset then init again, this IP will be changed.
I see the logs but on master's pod, it seems normal, no error shows. On worker's pod, there is one thing I notice is:
bird: BGP: Unexpected connect from unknown address 10.7.0.134 (port 41056) with 10.7.0.134 is master node's IP.

Steps to Reproduce (for bugs)

1. Install kubeadm on both master and 1 node from this link https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
2. Run kubeadm init on master node and then install calico-etcd, rbac, calico from this link https://docs.projectcalico.org/v3.2/getting-started/kubernetes/. Wait a few minutes then I see calico-node pod is ready 2/2, it's ok.
3. kubeadm join, then wait, new calico-node pod is created and both calico-node pods READY 1/2.
4. I try to delete worker node by kubectl delete node work1-node, 1 calico-node is deleted and the other pod in master node becomes READY 2/2.
5. Try to join again, the same behaviour appears.

Context

I don't have another pod, so don't know which problem I will have if it does not work well.

Your Environment

• Calico 3.2.1
• Kubernetes 1.11.1 with kubeadm
• Ubuntu 16.04
• Docker 18.06 and 18.03.

[caseydavenport]

Readiness probe failed: calico/node is not ready: BIRD is not ready: BGP not established with 172.26.0.1, and I don't know 172.26.0.1 is?

Interesting. None of the nodes have that IP address? Could you show the output of the following commands?

On the master:

ETCD_ENDPOINTS=http://localhost:6666 calicoctl get nodes -o yaml

ETCD_ENDPOINTS=http://localhost:6666 calicoctl get bgppeers

And then on the node:

calicoctl node status

That should help identify if there's a weird configuration somewhere.

[mrsherlock88]

I reinstall again, now 172.26.0.1 become 172.22.0.1, I also update my first post. I don't know where these IP comes from, maybe it's IP of the pod, not the node.
I export the result on master:

apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
  kind: Node
  metadata:
    creationTimestamp: 2018-09-18T03:20:44Z
    name: linhnh13-test
    resourceVersion: "1419"
    uid: d452a661-baf1-11e8-8397-005056871a40
  spec:
    bgp:
      ipv4Address: 10.7.0.238/24
      ipv4IPIPTunnelAddr: 192.168.135.192
    orchRefs:
    - nodeName: linhnh13-test
      orchestrator: k8s
- apiVersion: projectcalico.org/v3
  kind: Node
  metadata:
    creationTimestamp: 2018-09-10T09:38:53Z
    name: shipment-test
    resourceVersion: "1402"
    uid: 54b54d34-b4dd-11e8-b3ff-005056870042
  spec:
    bgp:
      ipv4Address: 172.22.0.1/16
      ipv4IPIPTunnelAddr: 192.168.82.0
    orchRefs:
    - nodeName: shipment-test
      orchestrator: k8s
kind: NodeList
metadata:
  resourceVersion: "1422"

On node ETCD_ENDPOINTS=http://10.96.232.136:6666 calicoctl get bgppeers show empty:

NAME   PEERIP   NODE   ASN

Then calicoctl node status:

Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+---------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |  INFO   |
+--------------+-------------------+-------+----------+---------+
| 172.22.0.1   | node-to-node mesh | start | 03:20:47 | Connect |
+--------------+-------------------+-------+----------+---------+

IPv6 BGP status
No IPv6 peers found.

I also check my ifconfig, it seems normal on node:

docker0   inet: 172.17.0.1   Mask: 255.255.0.0
ens32       inet: 10.7.0.238   Mask: 255.255.255.0
tunl0        inet: 192.168.135.192 Mask: 255.255.255.255

But too many br-xxxxxx network on the master, include 172.22.0.0, is it normal? because I run kubeadm reset - init several times.

[mrsherlock88]

It's OK for now, I download the calico.yaml file, then add IP_AUTODETECTION_METHOD="interface=ens*" environment, now I see it works.
I created too many bridge networks in master node because deployed many things here before, not by re-create cluster by kubeadm.
Thanks,