Contour: Extensions missing and makes my domain inaccessible

[vongohren]

What steps did you take and what happened:
I have had a working cluster and it started to fail.
I decided to start fresh and update things so I did this

After this I followed this guide to install again: https://projectcontour.io/guides/cert-manager/, with newest versions.

And everything was working fine immediatly, and through out the day. But I wake up to this issue:

Where I now have done the fresh clean again so its working, and done these minor changes to my kubernetes files:

But not sure if I will wake up to issue tomorrow as well?

What did you expect to happen:
Not get the error, but uncertain what I have that triggers this. As it was working, and I did deploy new versions of the app, but this morning it was failing.

Anything else you would like to add:
Where do I look further?
Is it the cert manager?
Contour together with my yamls, where might it be?

Environment:

• Contour version: 1.10.0
• Kubernetes version: (use kubectl version): Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.14-gke.1200", GitCommit:"7c407f5cc8632f9af5a2657f220963aa7f1c46e7", GitTreeState:"clean", BuildDate:"2020-12-01T09:20:59Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}
• Kubernetes installer & version: Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:41:55Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
• Cloud provider or hardware configuration: Google cloud, kubernetes as a service
• OS (e.g. from /etc/os-release): N/A

[skriss]

@vongohren are you using the Contour quickstart YAML files, or your own custom YAML?

Our example YAML includes the following in the contour cluster role: https://github.com/projectcontour/contour/blob/main/examples/contour/02-role-contour.yaml#L77-L84

Can you confirm whether or not that exists in the cluster role you're using?

[vongohren]

I have a local copy because im adding lodaBalancerIP, but it is directly from the quickstart file.
To confirm, they are there

Might there be some cleanup issues or something?

[skriss]

Is there any chance the ClusterRole/ClusterRoleBinding could've been inadvertently modified or deleted on the cluster? The YAML looks fine, but the error messages sure look like the RBAC was revoked on the cluster somehow.

[vongohren]

Well modfied or deleted is a bit strange, but they can be mismatching because I have deleted and removed stuff.
Deleting the contour namespace, maybe does not remove these things?

Or any thoughts there?

[skriss]

The ClusterRole is not namespaced, it's cluster-scoped, so yeah deleting the projectcontour namespace would not delete it.

[vongohren]

So should there be any notice of cleaning up the clusterroles if need be?

[youngnick]

Yes, we should probably change the uninstall instructions to do a kubectl delete -f instead of just deleting the namespace, to ensure that the RBAC is gone.

[skriss]

Yes, we should probably change the uninstall instructions to do a kubectl delete -f instead of just deleting the namespace, to ensure that the RBAC is gone.

Makes sense, assuming the install was done from a file. The other cluster-scoped items that would need to be removed are the 3 CRDs (HTTPProxy, TLSCertificateDelegation, and ExtensionService).

[youngnick]

I don't think we've touched the uninstall instructions for some time; they are due an update.

[youngnick]

I've logged #3209 to cover updating the uninstallation instructions. @vongohren, do you want to leave this open until that's complete?

[vongohren]

@youngnick nope, sounds good too me :D Il close it