Kubeadm: Calico and podsubnet CIDR docs for kubeadm install need updating

[andrevtg]

/kind bug

It seems to me that pod subnets in kubernetes 1.12 must be /24 (and not /16). Subnets are defined by "--pod-network-cidr" argument in kubeadm OR by a config file such as below:

apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration
api:
  advertiseAddress: 0.0.0.0
  bindPort: 6443
kubernetesVersion: v1.12.1
---
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
networking:
  podSubnet: 192.168.0.0/24

This behaviour also breaks calico setup even if you follow the docs for kubeadm setup.

I had to change CALICO_IPV4POOL_CIDR in "calico.yaml" to "192.168.0.0/24" and things worked fine.

Environment:

• Kubernetes version
  Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Same behaviour on digital ocean CentOS droplets and in local on-prem CentOS VMs.
  CentOS 7.5

• OS (e.g. from /etc/os-release):
  CentOS 7.5 (on-prem)

• Install tools:
  kubeadm

@kubernetes/sig-network-bugs

[k8s-ci-robot]

@andrevtg: Reiterating the mentions to trigger a notification:
@kubernetes/sig-network-bugs

In response to this:

/kind bug

It seems to me that pod subnets in kubernetes 1.12 must be /24 (and not /16). Subnets are defined by "--pod-network-cidr" argument in kubeadm OR by a config file such as below:

apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration
api:
 advertiseAddress: 0.0.0.0
 bindPort: 6443
kubernetesVersion: v1.12.1
---
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
networking:
 podSubnet: 192.168.0.0/24

This behaviour also breaks calico setup even if you follow the docs for kubeadm setup.

I had to change CALICO_IPV4POOL_CIDR in "calico.yaml" to "192.168.0.0/24" and things worked fine.

Environment:

• Kubernetes version
  Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Same behaviour on digital ocean CentOS droplets and in local on-prem CentOS VMs.
  CentOS 7.5

• OS (e.g. from /etc/os-release):
  CentOS 7.5 (on-prem)

• Install tools:
  kubeadm

@kubernetes/sig-network-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[caseydavenport]

I had to change CALICO_IPV4POOL_CIDR in "calico.yaml" to "192.168.0.0/24" and things worked fine.

Something seems wrong here - this value shouldn't be set to a /24 - it's meant to include the entire cluster's pod IP range, and a /24 is typically not large enough.

It seems to me that pod subnets in kubernetes 1.12 must be /24 (and not /16). Subnets are defined by "--pod-network-cidr" argument in kubeadm OR by a config file such as below:

What makes you say this? I'm not aware of this changing.

[yagonobre]

@andrevtg can you share the calico log when you try to use 192.168.0.0/24, a /24 is not enough for the pod cidr.

[rosti]

I can confirm this. My test setup features stock CentOS 7.5.1804 with k8s 1.12.2.
If I install Calico with 192.168.0.0/24 everything works like a charm. If I use the normal 192.168.0.0/16 it appears that pods don't have connectivity outside this range.
I'll investigate on this further.

An interesting thing is also that CoreDNS 1.2.2 crashes when using /16 with the familiar loop detected message:

2018/11/16 15:05:29 [FATAL] plugin/loop: Seen "HINFO IN 5415706924263518008.7756288275848504382." more than twice, loop detected

After updating it to 1.2.6, this now is more meaningful and it's not crashing:

 [ERROR] plugin/errors: 2 7175345282990691314.173103285019062108. HINFO: unreachable backend: read udp 192.168.0.18:43025->192.168.217.2:53: i/o timeout
 [ERROR] plugin/errors: 2 7175345282990691314.173103285019062108. HINFO: unreachable backend: read udp 192.168.0.18:57106->192.168.217.2:53: i/o timeout
 [ERROR] plugin/errors: 2 localhost.localdomain. AAAA: unreachable backend: read udp 192.168.0.18:44461->192.168.217.2:53: i/o timeout
 [ERROR] plugin/errors: 2 localhost.localdomain. A: unreachable backend: read udp 192.168.0.18:49719->192.168.217.2:53: i/o timeout

This may explain recent reports about detected loops by CoreDNS.

[caseydavenport]

@rosti I've seen the CoreDNS errors as a result of running network manager on the host - if that is running could you disable that and see if it fixes the issue?

[rosti]

Network Manager and firewalld are disabled. SELinux in permissive mode.
Rebuilding the cluster with Calico 3.2 didn't change anything either.

/assign

[neolit123]

/16 seems to be working for me using their 3.1 and 3.3 yams with a 1.13 control plane on Ubuntu 17.10:

marking this as kind/ecosystem.

It seems to me that pod subnets in kubernetes 1.12 must be /24 (and not /16). Subnets are defined by "--pod-network-cidr" argument in kubeadm OR by a config file such as below:

@andrevtg please report this to the calico developers as this is not a kubeadm problem and also /24 does not make sense as that is a very small range of addresses. well, it can work for some use cases but we cannot have that in the kubeadm docs.

this is also part of the official calico documentation:
https://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/kubeadm/

cc @timothysc @fabriziopandini @rdodev @chuckha
i recommend that we start using weave in our HA tutorials, not because of this particular issue, but rather due to the fact that i've seen the most stability out of it. also a user confirmed the other day that it works in their HA setup.

also kind is using weave by default now.

[rdodev]

@neolit123 I just did 2 different HA control plane with 1.12.2 using 192.168.0.0/16 for calico and didn't run into any issues. So perhaps a fluke?

All conformance passing: https://scanner.heptio.com/3a9be7cb7e2457c4654e1fb3e8f6dbaf/diagnostics/

[neolit123]

@rdodev

given @rosti reproduced as well this could be an issue on cent OS.
what distro are your running on the Heptio setup?

[neolit123]

i'm seeing Ubuntu 18.04.1 LTS in the logs.

[rosti]

I'll try replicating this on Ubuntu too (with the same setup, that is failing in CentOS). I presume it's more "Calico with CentOS host" issue.
It's really easy to reproduce it on latest CentOS. Happens every time.

[fasaxc]

How is core DNS configured, what upstream DNS servers is it using? Has it somehow pulled in a resolv.conf that loops back to 127.x.x.x or its own service IP? Saw that with kube-dns on Ubuntu at one point when they switched to using a local DNS proxy on 127.0.0.10, which of course resolved back to kube-dns when attempted inside the kube-dns pod.

[rosti]

Coming after a week back to this issue. I checked the routing table of my test machine. It turned out, that one of the network interfaces had an IP in 192.168.217.0/24. Worse still, the gateway in that network was the default gateway. This explained why PODs with Calico CIDR IPs could not connect to the outside world and why the 192.168.0.0/24 CIDR worked for some users.
Changing the Calico CIDR to non-overlapping one got things up and running again.
We need to document this though.

[rosti]

@neolit123 what is the best place to document that? Is the troubleshooting guide, the network plugin section in the "create cluster with kubeadm" page or the network plugin requirements page?

[rdodev]

@rosti fwiw, I think these belong to in the latter. I'm getting the sense we're overloading the kubeadm guide with all sorts caveats and tips -- while useful they tend to distract from the "happy path" steps we provide in the guide.

[neolit123]

i think we should add this in the CNI->Calico section as a short paragraph.
the plugin is still widely used and this need to be seen by those who install it.

[rosti]

I'd probably place it in the CNI paragraph directly, since all of the plugins can suffer from this (despite the fact, that Calico is most likely since it's using the 192.168.0.0/16 CIDR).

[rdodev]

@rosti couple of things:

1. IIRC, calico is the only CNI plug-in that requires explicit pod CIDR declaration. All other plug-ins discover the CIDR via the API

1. We should also warn the user that choosing a pod network cidr other than 192.168.0.0/24 means they can't use the manifests from Calico directly, so they need to download them, modify the pertinent entries and then apply them.

[caseydavenport]

I'd probably place it in the CNI paragraph directly, since all of the plugins can suffer from this (despite the fact, that Calico is most likely since it's using the 192.168.0.0/16 CIDR).

Yep, that's exactly right. Agreed this should be in the main CNI paragraph. Potentially worth noting that this applies to the Service IP range as well - nodes, pods, and services should have distinct IP ranges assigned to them.

IIRC, calico is the only CNI plug-in that requires explicit pod CIDR declaration. All other plug-ins discover the CIDR via the API

Unfortunately, there isn't a good way to get this information from the Kubernetes API today. Calico defaults to using 192.168.0.0/16 but that can easily be changed. Other plugins simply have different defaults. We should perhaps consider modifying the default value Calico uses if it would simplify the workflow.

(Even better would be a way to remove the desire for kube-proxy to know about the --pod-cidr option, or create a central configuration point via the k8s API somehow. But, those are longer conversations!)

this is also part of the official calico documentation:
https://docs.projectcalico.org/v2.0/getting-started/kubernetes/installation/hosted/kubeadm/

FWIW, that's a _really_ old link, for Calico v2.0. The equivalent for Calico v3.3 is here: https://docs.projectcalico.org/v3.3/getting-started/kubernetes/

It contains a snippet which touches on the issue described here:

If 192.168.0.0/16 is already in use within your network you must select a different pod network CIDR, replacing 192.168.0.0/16 in the above command as well as in any manifests applied below.

[neolit123]

IIRC, calico is the only CNI plug-in that requires explicit pod CIDR declaration. All other plug-ins discover the CIDR via the API

Flannel and Canal also require explicit pod CIDR.

[neolit123]

FWIW, that's a really old link, for Calico v2.0. The equivalent for Calico v3.3 is here:

that's true. but the problem can be observed with 3.3 as well.

It contains a snippet which touches on the issue described here:

If 192.168.0.0/16 is already in use within your network you must select a different pod network CIDR,
replacing 192.168.0.0/16 in the above command as well as in any manifests applied below.

yes, that's exactly the problem.

[caseydavenport]

@neolit123 yep, my point with the docs link was just to make sure we're on the same page, and to point out that the newer docs do mention that 192.168.0.0/16 can't be already in-use within your network.

Just spotted this PR, which looks like it covers the issue here. Nice! https://github.com/kubernetes/website/pull/11225

[rdodev]

Unfortunately, there isn't a good way to get this information from the Kubernetes API today.

The Node object has all this info, I believe.

[caseydavenport]

@rdodev The Node object has a spec.PodCIDR field, which is _not_ the same thing as the --pod-cidr argument passed to kubeadm. Each node gets a small piece of the total CIDR assigned to the cluster.

In fact, Calico _does_ monitor the spec.PodCIDR field of the Node object. However, that field will only get populated if --pod-cidr is passed to kubeadm :D

What I meant is there is no way to view the entire CIDR used for allocating node spec.PodCIDRs.

[rdodev]

@caseydavenport tyvm for the clarification ... I think I understand the subtle difference now ... emphasis on _think_. :)

[rosti]

Yes, this is an issue, that is best to be solved at a more broad level.
For now, we have somewhat plugged the gap in documentation with kubernetes/website#11225.
We can try to implement a pre-flight check during init and join, that verifies that the CIDR does not overlap with any of the host networks (as present in the routing table).

[caseydavenport]

Is there something left we're tracking with this issue, or can it be closed?

It seems to me we have addressed the lack of documentation, and any future work should be tracked in a new issue?

[neolit123]

no, we are good for now.
thanks.

[hsitaa]

I think issue still persists, it's fixed by fluke and occurs as fluke.

Running two clusters. Both at 1.13.10 k8s and calico latest. 3.3 as well. Still one cluster running fine. Other not.. In both cluster podciDr was not define in Init. Instead was later added in kube config and cluster cidr kube proxy daemon. Juggling with it for 8 hrs now. Almost at verge to hate calico.

[yagonobre]

Hey, @hsitaa 1.13 is no longer supported, please try again with a new version and create a new issue if the problem persists.