K8s-config-connector: Upgrading from v1.12.1 to v1.14.0 makes all GKE cluster and nodepools update fail

[tonybenchsci]

Describe the bug
All GKE clusters:

message: 'Update call failed: error validating configuration: "ip_allocation_policy":
      conflicts with cluster_ipv4_cidr'
reason: UpdateFailed
status: "False"

All GKE nodepools:

message: 'Update call failed: error fetching live state: error importing resource:
      Import id "<REDACTED_project-id>/us-central1-a//<REDACTED_clustername>" doesn''t match any of the accepted
      formats: [projects/(?P<project>[^/]+)/zones/(?P<location>[^/]+)/clusters/(?P<cluster>[^/]+)/nodePools/(?P<name>[^/]+)
      projects/(?P<project>[^/]+)/locations/(?P<location>[^/]+)/clusters/(?P<cluster>[^/]+)/nodePools/(?P<name>[^/]+)
      (?P<project>[^/]+)/(?P<location>[^/]+)/(?P<cluster>[^/]+)/(?P<name>[^/]+) (?P<location>[^/]+)/(?P<cluster>[^/]+)/(?P<name>[^/]+)]'
reason: UpdateFailed
status: "False"

ConfigConnector Version
When going from v1.12.1 to v1.14.0

To Reproduce
Follow the manual upgrade instructions for the Workload Identity installation.

YAML snippets:
N/A

[jcanseco]

Hi @tonybenchsci, thanks for reporting this. It's a known issue and we have a fix coming in the next release.

[tonybenchsci]

Thanks @jcanseco in that case I will revert the upgrade and wait for your fix. Mind enlightening us a bit on the root cause and expected patch release date?

[jcanseco]

Sure, we introduced validation logic recently for many of our resources to better help our users from writing faulty configurations, and this is a case where the validation logic itself turned out be faulty. See this for (somewhat) more details.

[jcanseco]

Though to clarify, the incoming fix will fix the validation issue with ContainerCluster.

The ContainerNodePool error you shared is due to the referenced ContainerCluster failing, so once the ContainerCluster stops failing to update, the ContainerNodePool should also be able to reach an UpToDate state.

That said, the ContainerNodePool should not be outputting that error due to the referenced ContainerCluster failing. This too is a known issue, but a separate one that will be fixed later.

[tonybenchsci]

Gotcha thanks @jcanseco. Just a FYI, reverting back to v1.12.1 resolved this issue, but now (though not breaking)

E0715 23:29:59.076378     1 reflector.go:178] 
sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:224: 
Failed to list secretmanager.cnrm.cloud.google.com/v1beta1, Kind=SecretManagerSecret: 
secretmanagersecrets.secretmanager.cnrm.cloud.google.com is forbidden: 
User "system:serviceaccount:cnrm-system:cnrm-controller-manager" cannot list 
resource "secretmanagersecrets" in API group "secretmanager.cnrm.cloud.google.com" at the cluster scope

I assume there is a step missing in https://cloud.google.com/config-connector/docs/how-to/install-upgrade-uninstall#upgrading (manual upgrade), or that to downgrade I need to run an additional kubectl delete command to fully uninstall SecretManager CRD?

[jcanseco]

@tonybenchsci ah, unfortunately it seems you followed the manual upgrade steps to perform an in-place downgrade? We don't officially support in-place downgrades currently.

v1.14.0 introduced the SecretManagerSecret resource, which is probably why you are seeing that error.

Our recommended solution to handle this issue is to do a full uninstall and reinstall. Is this an option for you?

[tonybenchsci]

Our recommended solution to handle this issue is to do a full uninstall and reinstall. Is this an option for you?

That is too risky I'm afraid, and since the SecretManagerSecret issue isn't really breaking, I'm hoping to upgrade to whatever v1.14.x fix is. When does that get released?

[jcanseco]

Gotcha, I'm very glad to hear that the issue is not breaking. I'm sorry for the trouble.

The fix will be part of the next release which will come out by the end of the week.

[tonybenchsci]

Not to keep harping this @jcanseco , but I noticed after the in-place downgrade, sqlinstance resources are no longer part of the reconciliation loop. The kubectl describe shows configs updating and up-to-date but the GCP settings are not tracked/mutated. Is it something to do with the cnrm-lease label?

Understand that downgrades are not fully supported, but wondering if this is something that needs to be fixed in general and if it's specific to sqlinstance CRDs. Hoping there is a simple workaround for me to sync up KCC and GCP

EDIT: I just re-installed (v1.12.1 to v1.12.1) and this is now resolved.

[jcanseco]

Hi @tonybenchsci, awesome, I'm glad the problem has been resolved!

We also just released v1.15.0 which should fix the ContainerCluster validation issue. Please try it out and let us know if it fixes the problem.

[tonybenchsci]

Thanks @jcanseco v1.15.0 seems to have fixed the ContainerCluster issue (and obviously the minor SecretManager issue). I am seeing odd but transient errors below (which then get quickly corrected to UpToDate but wasn't an issue with v1.12.1):

• {"level":"error","ts":1594928212.3182576,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"sqldatabase-controller","request":"b6i-prod/sor","error":"Update call failed: error fetching live state: error importing resource: project: required field is not set"
• {"level":"error","ts":1594928174.0007658,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"containernodepool-controller","request":"b6i-stg/kjyo1","error":"Update call failed: error fetching live state: error importing resource: Import id \"b6i-stg/us-east4-a//kjyo1\" doesn't match any of the accepted formats: [projects/(?P<project>[^/]+)/zones/ (same as original post)

I'm ready to close this issue though, if you could explain both points quickly. To me, sqldatabase and containernodepool are both resources that reference other KCC resources (i.e. sqlinstance and containercluster), and I suspect the validation logic added in v1.13.1 might be too quick to detect and report errors when the reference resource isn't finished re-conciliating?

[jcanseco]

Great, thanks for confirming the fix worked!

And thanks for reporting these log messages and their transient behavior. These do look like something we should fix. I would say these don't really look like issues with the validation logic, but rather, we seem to not be properly detecting that the parent resource is not yet ready (e.g. the parent SQLInstance of a SQLDatabase, or the parent ContainerCluster of a ContainerNodePool) when fetching the live state of the child resource from GCP.

We'll take a look and let you know when we have updates.

[jcanseco]

Oh and if you're ok with it, I'll be closing the issue now since the original problem has been fixed, but please feel free to reopen if you have any further issues!

[jcanseco]

Hi @tonybenchsci, the issue that was causing resources like ContainerNodePool and SQLDatabase to display UpdateFailed due to the referenced ContainerCluster or SQLInstance not being ready is now fixed in v1.16.0.

[tonybenchsci]

Hi @tonybenchsci, the issue that was causing resources like ContainerNodePool and SQLDatabase to display UpdateFailed due to the referenced ContainerCluster or SQLInstance not being ready is now fixed in v1.16.0.

Thanks. Confirming that it did fix the errors.

[dinvlad]

We're experiencing the same issue as in https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/242#issuecomment-659068744 here with the new ConfigConnector GKE add-on. When I check its version in cnrm-webhook-manager yaml, it says 1.13.0. Does it mean we just have to wait until it's upgraded to 1.15.0 by the GKE team? And is there anyone we can ask to do that? Thanks

[jcanseco]

Hi @dinvlad, yes, it takes about 3 weeks for the KCC GKE add-on to pick up a new KCC release. We're looking into separating the KCC GKE add-on upgrade schedule from the GKE upgrade schedule so that users could, for example, get KCC upgrades sooner. However, this is still very much a work-in-progress.

If you want to get the latest KCC release, you could install the KCC operator as a standalone (which is really what the KCC GKE add-on uses under the hood to manage KCC installations). You can do so by following the instructions here.

[tonybenchsci]

From our experience, the manual install using Workload Identity has been very smooth and reliable.
All you need is a shell script with all the upgrade commands like ./install 1.15.0

[dinvlad]

Thanks for the heads-up on 3-weeks schedule, I think that works for us.

The manual install is what we've been using for many months now, fairly smoothly. I've decided to test the new add-on, which is why I was wondering if it could be upgraded. The last "stable" version that worked for us smoothly was 1.14.0.

[dinvlad]

Also, happy to report that I eliminated this particular error for now (since we aren't using SecretManager resources yet, though planning for them) by running

kubectl delete crd secretmanagersecrets.secretmanager.cnrm.cloud.google.com
kubectl delete ConfigConnector configconnector.core.cnrm.cloud.google.com --wait=true

and then re-installing the ConfigConnector resource.

[nebril]

I just hit this issue on 1.15.1 , but updating to 1.24.0 fixed it.