Calico: kubeadm+calico seem to ignore custom CIDR

[joewilliams]

Following the kubeadm docs and calico related docs pods that get created end up in the 192.168.0.0/16 CIDR even when --pod-network-cidr= and the calico.yaml have been adjusted.

root@kube1:~# diff calico.yaml.1 calico.yaml
221c221
<               value: "192.168.0.0/16"
---
>               value: "172.16.0.0/16"

  509  kubeadm init --pod-network-cidr=172.16.0.0/16
root@kube1:~#
root@kube1:~# grep -A1 CIDR calico.yaml
            - name: CALICO_IPV4POOL_CIDR
              value: "172.16.0.0/16"
root@kube1:~#
root@kube1:~#
root@kube1:~# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                      READY     STATUS    RESTARTS   AGE       IP              NODE
kube-system   calico-etcd-wvkz4                         1/1       Running   0          1m        192.168.1.16    kube1
kube-system   calico-kube-controllers-d669cc78f-z9txr   1/1       Running   0          1m        192.168.1.16    kube1
kube-system   calico-node-vgq8p                         2/2       Running   0          1m        192.168.1.16    kube1
kube-system   etcd-kube1                                1/1       Running   0          10m       192.168.1.16    kube1
kube-system   kube-apiserver-kube1                      1/1       Running   0          9m        192.168.1.16    kube1
kube-system   kube-controller-manager-kube1             1/1       Running   0          9m        192.168.1.16    kube1
kube-system   kube-dns-6f4fd4bdf-k7x9f                  3/3       Running   0          10m       192.168.18.62   kube1
kube-system   kube-proxy-r5swx                          1/1       Running   0          10m       192.168.1.16    kube1
kube-system   kube-scheduler-kube1                      1/1       Running   0          9m        192.168.1.16    kube1

root@kube1:~# calicoctl get ippools
NAME                  CIDR
default-ipv4-ippool   192.168.0.0/16

This does not seem to happen when the calico 2.6 manifest is used.

Expected Behavior

Specifying the CIDR in the calico yaml and --pod-network-cidr= should result in that CIDR getting used for the pod network rather than the default.

Possible Solution

As a work around I delete the incorrect CIDR and add the correct one using calicoctl.

Context

Your Environment

• Calico version: v3
• Orchestrator version (e.g. kubernetes, mesos, rkt): kube 1.9.2
• Operating System and version: debian jessie
• Link to your project (optional):

[tmjd]

There are a couple gotchas that I want to make sure you're aware of:

• The variable CALICO_IPV4POOL_CIDR is only applicable when no IPPool exists, this means if you're applying calico.yaml, editing it, and then applying it again the IPPool will be created the first time and then it won't be updated on the 2nd.
• Kind of along the same lines but a bit different is that the Calico kubeadm manifest mounts a folder for the etcd data on the host system. This means if you are using kubeadm reset that data will persist on your host and a subsequent kubeadm init and then applying the calico manifest will have the same etcd data that was previously written.

I'm not sure why you would see a different behavior between v2.6 and v3.0 as the behavior I'm describing is the same for both.

If you're aware of the above then we'll have to dig a bit deeper to understand what's happening.

[joewilliams]

Kind of along the same lines but a bit different is that the Calico kubeadm manifest mounts a folder for the etcd data on the host system. This means if you are using kubeadm reset that data will persist on your host and a subsequent kubeadm init and then applying the calico manifest will have the same etcd data that was previously written.

This seems to have been the issue.

[tmjd]

It sounds like this issue is resolved so I'm going to close it. If I've misunderstood please let me know and I'll reopen the issue.

[ho0ber]

Kind of along the same lines but a bit different is that the Calico kubeadm manifest mounts a folder for the etcd data on the host system. This means if you are using kubeadm reset that data will persist on your host and a subsequent kubeadm init and then applying the calico manifest will have the same etcd data that was previously written.

This seems to have been the issue.

I also just lost hours on exactly this but couldn't figure out where my state was bleeding into my subsequent attempts.

[caseydavenport]

I also just lost hours on exactly this but couldn't figure out where my state was bleeding into my subsequent attempts.

Sorry to hear that :(

FWIW I think we should get rid of that etcd.yaml altogether - it's not production ready and generally causes problems, and we can use the Kubernetes API instead of etcd anyway now so we should just default to that.

I've got a WIP which does that here: https://github.com/projectcalico/calico/pull/2435

[ho0ber]

I also just lost hours on exactly this but couldn't figure out where my state was bleeding into my subsequent attempts.

Sorry to hear that :(

FWIW I think we should get rid of that etcd.yaml altogether - it's not production ready and generally causes problems, and we can use the Kubernetes API instead of etcd anyway now so we should just default to that.

I've got a WIP which does that here: #2435

That's the way it goes! I was mostly just sharing that I had a similar experience, but I was happy to finally understand what was up. I'm delighted by the changes in #2435; thanks for all of the hard work on Calico!