Kubespray: Cilium Permission error "cannot list resource endpointslices in API group discovery.k8s.io"

[chgl]

There is a permission error as cilium is unable to query EndpointSlice resources when installed via kubespray:

level=error msg=k8sError error="github.com/cilium/cilium/pkg/k8s/watchers/endpoint_slice.go:116: Failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User \"system:serviceaccount:kube-system:cilium\" cannot list resource \"endpointslices\" in API group \"discovery.k8s.io\" at the cluster scope" subsys=k8s

The fix for this is to add EndpointSlices read permissions to the ClusterRole (these permissions are also set in the quick install manifests (see https://github.com/cilium/cilium/blob/master/install/kubernetes/quick-install.yaml).

I've prepared a patch in https://github.com/kubernetes-sigs/kubespray/compare/master...chgl:add-endpointslice-query-permissions

There are additional permission issues which I am addressing (and maybe breaking something) in the patch. If it's OK, I can create a PR.

I don't think the details below are necessary, but I'll gladly add them if requested!

Environment:

• Cloud provider or hardware configuration:

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

• Version of Ansible (ansible --version):

• Version of Python (python --version):

Kubespray version (commit) (git rev-parse --short HEAD):

v2.12.5

Network plugin used:

Cilium v1.7.2

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

Command used to invoke ansible:

[Miouge1]

@chgl thank you for reporting this. Can you make a pull request with the appropriate changes?

I'v also noticed this error in CI which is one of the blocker for v2.13 release.

I'll also flag the Kubespray Cilium maintainers: @EppO @npajkovsky @kuja53 maybe they can help with a PR if needed.

[floryut]

@chgl while you're at it, maybe bump cilium to 1.7.2 (https://github.com/cilium/cilium/releases/tag/v1.7.2)

roles/download/defaults/main.yml:83

[npajkovsky]

What kernel version do you have?

[Miouge1]

@npajkovsky I have tested this with Ubuntu 18.04, kernel 4.15.0-43-generic

Does it matter?

[kuja53]

Kernel version doesnt matter. I find that many cilium manifests in kube-spray is really old with missing pieces from actual cilium upstream manifests. I will try to find another bugs which I have in mind and prepare PR with it. But this RBAC issue is really clear and it helps a lot with move towards. Thank you for appointment.

[kuja53]

I will make PR from this patch or @chgl you could make PR with reviewers on our group. Thank you.

[chgl]

@kuja53 I'll create a PR. Thanks for your feedback!

[npajkovsky]

@chgl I have pulled your changes and did diff between my work and your work and we end up with exactly the same patch, with one exception. I did that for v2.12.5 tag.

Anyway as @kuja53 said, cilium-config should be updated as well and we could wait for his
pullreq.

Since this is release blocker, I'm pretty fine with patchset as is.

[chgl]

@npajkovsky Sorry, just to clarify: should I keep my PR open or close it and have it replaced by @kuja53 and his additional fixes and improvements (which will include updating cilium-config?)?

[floryut]

@chgl I'll say to keep yours, as this is totally blocking Cilium and should be ship with 2.13.
If @kuja53 can make a PR asap it will be nice, if not this will be ship later

[npajkovsky]

Agreed.