Kubespray: Mismatch between Kubespray and the upstream Calico CRDs

[mikejoh]

Environment:

• Cloud provider or hardware configuration: VMs on OpenStack
• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): Ubuntu 20.04.1
• Version of Ansible (ansible --version):
• Version of Python (python --version):

Kubespray version (commit) (git rev-parse --short HEAD): From tag v2.14.0
Network plugin used: Calico 3.16.0

Anything else do we need to know:

Since the 3.16.0 release of Calico the BGPPeer CRD includes the following spec field: keepOriginalNextHop. This was added here: https://github.com/projectcalico/libcalico-go/pull/1266.

The CRDs we apply (currently on master) does not include this field and probably missing others too:
https://github.com/kubernetes-sigs/kubespray/blob/c25d624524f8be9a6f0191797745b5883f3a7f2c/roles/network_plugin/calico/templates/kdd-crds.yml.j2#L742-L806

Here's the upstream BGPPeer CRD as it looks like in 3.16.0:
https://github.com/projectcalico/libcalico-go/blob/release-v3.16/config/crd/crd.projectcalico.org_bgppeers.yaml

We're defaulting to version 3.16.2 Calico here: https://github.com/kubernetes-sigs/kubespray/blob/c25d624524f8be9a6f0191797745b5883f3a7f2c/roles/download/defaults/main.yml#L68

We basically need to sync the CRDs from upstream so that they match the version of Calico we install here (in Kubespray).

I can start with a PR correcting this.

[hafe]

The split of upstream calico manifests into several manifest files in kubespray is a mistake. There is no way you can diff upstream with what is in kubespray, you just discover errors very late like you did

[mikejoh]

@hafe I guess i could diff them by just splitting up the kdd-crds.yml.j2 file by hand, not automating anything just cropping out the actual manifest and do that for each one of them. That's the naive and "quick" way i can think of, long term it maybe would make sense to (one of the two):

1. Follow upstream and have seperate manifest files for each CRD in a e.g. subdir to files/ within the Calico role. AFAIK there's no need to have the CRDs as one Jinja2 template file since we cleaned up the file from anything related to templating. Not saying that we never want to do templating again across these files though!
2. Fetch (download) and apply the correct versions of the CRDs at runtime from upstream using the calico_version variable.

Suggestion 2 is not a preferable solution and not very air-gap friendly, i can understand the need of vendoring these CRDs but then we'll need to double check upstream and sync each time we bump Calico.

I would gladly help out with finding a solution to this!

[hafe]

For this issue I think you should do the smallest possible change. Addressing the bigger problem needs to be done in some other context. Calico has a lot of interesting new features like wireguard and eBPF mode that kubespray will miss out on if we don't do something.

(with upstream I meant the officially documented manifests such as: https://docs.projectcalico.org/archive/v3.16/manifests/calico.yaml)

[hafe]

Another alternative though a bit controversial/complex would be to use the helm client plus the original helm templates in the calico repo. Helm is much better to render templates then ansible/jinja