K8s-config-connector: IAMCustomRole not updating after acquiring

[TravRan]

Describe the bug
IAMCustomRoles aren't updating after the role has been acquired. In KCC the role shows that it has the added permission, but the role isn't updating in GCP.

ConfigConnector Version
1.30.0

To Reproduce
Acquire a custom role that was already created. Then try updating the permissions for the custom role.

YAML snippets:
I've added three snippets. The IAMCustomRole spec. The status of the role in KCC showing that it has the "dataflow.jobs.list" permission added to the role. And lastly the status of the role in GCP, showing that the list permission hasn't been added to the role.

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMCustomRole
metadata:
  name: {REDACTED}
spec:
  title: {REDACTED} - Custom DataFlow Developer
  description: Custom dataflow developer role for application service accounts
  permissions:
    - compute.machineTypes.get
    - dataflow.jobs.create
    - dataflow.jobs.get
    - dataflow.jobs.updateContents
    - dataflow.jobs.list
  stage: GA

Name:         {REDACTED}
Namespace:    {REDACTED}
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  cnrm.cloud.google.com/management-conflict-prevention-policy: none
              cnrm.cloud.google.com/project-id: {REDACTED}
              meta.helm.sh/release-name: {REDACTED}-product-serving
              meta.helm.sh/release-namespace: {REDACTED}
API Version:  iam.cnrm.cloud.google.com/v1beta1
Kind:         IAMCustomRole
Metadata:
  Creation Timestamp:  2020-12-04T16:55:53Z
  Finalizers:
    cnrm.cloud.google.com/finalizer
    cnrm.cloud.google.com/deletion-defender
  Generation:  4
  Managed Fields:
    API Version:  iam.cnrm.cloud.google.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cnrm.cloud.google.com/management-conflict-prevention-policy:
          f:cnrm.cloud.google.com/project-id:
          f:meta.helm.sh/release-name:
          f:meta.helm.sh/release-namespace:
        f:labels:
          f:app.kubernetes.io/managed-by:
      f:spec:
        f:description:
        f:stage:
        f:title:
    Manager:      before-first-apply
    Operation:    Update
    API Version:  iam.cnrm.cloud.google.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"cnrm.cloud.google.com/deletion-defender":
          v:"cnrm.cloud.google.com/finalizer":
      f:status:
        .:
        f:conditions:
        f:name:
    Manager:      cnrm-controller-manager
    Operation:    Update
    Time:         2020-12-04T19:53:15Z
    API Version:  iam.cnrm.cloud.google.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:permissions:
    Manager:         Go-http-client
    Operation:       Update
    Time:            2020-12-04T20:06:26Z
  Resource Version:  52527187
  Self Link:         /apis/iam.cnrm.cloud.google.com/v1beta1/namespaces/{REDACTED}/iamcustomroles/{REDACTED}
  UID:               f866207a-d23e-486e-9984-4dd0bf531d97
Spec:
  Description:  Custom dataflow developer role for application service accounts
  Permissions:
    compute.machineTypes.get
    dataflow.jobs.create
    dataflow.jobs.get
    dataflow.jobs.updateContents
    dataflow.jobs.list
  Stage:  GA
  Title:  {REDACTED}- Custom DataFlow Developer
Status:
  Conditions:
    Last Transition Time:  2020-12-04T19:53:15Z
    Message:               The resource is up to date
    Reason:                UpToDate
    Status:                True
    Type:                  Ready
  Name:                    projects/{REDACTED}/roles/{REDACTED}
Events:
  Type    Reason    Age   From                      Message
  ----    ------    ----  ----                      -------
  Normal  UpToDate  14m   iamcustomrole-controller  The resource is up to date

gcloud iam roles describe --project={REDACTED} {REDACTED} 
description: Custom dataflow developer role for application service accounts
etag: {REDACTED} 
includedPermissions:
- compute.machineTypes.get
- dataflow.jobs.create
- dataflow.jobs.get
- dataflow.jobs.updateContents
name: projects/{REDACTED} /roles/{REDACTED} 
stage: GA
title: Applications SA - Custom DataFlow Developer

[travisrandolph-bestbuy]

@caieo Could you or someone from the team please help me with this?

[travisrandolph-bestbuy]

I have another update. Updating a role works if we delete permissions from the role. It's only when we add permissions to the role that it doesn't update.

[caieo]

Hi @travisrandolph-bestbuy / @TravRan , sorry for the delay here! I was halfway through reproducing this issue when I got randomized by something else. This looks a bit fishy -- can you check if there are any errors updating the resource in the logs (kubectl logs -n cnrm-system cnrm-controller-manager-0)? In the meantime, I'll work on reproducing this issue and update this thread.

[travisrandolph-bestbuy]

Alright I think I found one instance of the error.

{"severity":"error","logger":"controller-runtime.controller","msg":"Reconciler error","controller":"iamcustomrole-controller","request":"{REDACTED}","error":"INTERNAL ERROR: panic: observed a panic: expected state's set to have value explicitly provided in spec\n\t/go/src/cnrm.googlesource.com/cnrm/pkg/execution/builtin.go:26 +0x57\n\tpanic(0x34c21e0, 0xc000e6c2a0)\n\t\t/usr/local/go/src/runtime/panic.go:975 +0x3e3\n\tcnrm.googlesource.com/cnrm/pkg/krmtotf.convertTFSetToKCCSet(0x3253140, 0xc002cb2a80, 0x3253140, 0xc003f0b760, 0xc000e4b2c0, 0xc0020479f8, 0x3aed4c2, 0xb, 0x51849a, 0xc0020a6188)\n\t\t/go/src/cnrm.googlesource.com/cnrm/pkg/krmtotf/tftokrm.go:542 +0xa82\n\tcnrm.googlesource.com/cnrm/pkg/krmtotf.convertTFMapToKCCMap(0xc0049d92f0, 0xc005154330, 0xc000e30fc0, 0xc0020479f8, 0x0, 0x0, 0xc002ca5a70, 0x0)\n\t\t/go/src/cnrm.googlesource.com/cnrm/pkg/krmtotf/tftokrm.go:378 +0xa8c\n\tcnrm.googlesource.com/cnrm/pkg/krmtotf.ConvertTFObjToKCCObj(0xc0049d92f0, 0xc005154330, 0xc000e30fc0, 0xc0020479f8, 0x0, 0x0, 0xc002ca5a70, 0xc005154330)\n\t\t/go/src/cnrm.googlesource.com/cnrm/pkg/krmtotf/tftokrm.go:270 +0x7a\n\tcnrm.googlesource.com/cnrm/pkg/krmtotf.GetSpecAndStatusFromState(0xc0020478c0, 0xc001fcad90, 0x0, 0x0)\n\t\t/go/src/cnrm.googlesource.com/cnrm/pkg/krmtotf/tftokrm.go:24 +0xd0\n\tcnrm.googlesource.com/cnrm/pkg/krmtotf.resolveUnmanagedFields(0xc0049d8d80, 0xc0020478c0, 0xc001fcad90, 0x0, 0x0, 0xc0049d8c90)\n\t\t/go/src/cnrm.googlesource.com/cnrm/pkg/krmtotf/krmtotf.go:159 +0xb6\n\tcnrm.googlesource.com/cnrm/pkg/krmtotf.krmResourceToTFResourceConfig(0xc0020478c0, 0xc001fcad90, 0x42c9ce0, 0xc0023d9cc0, 0xc00112d960, 0x1, 0x0, 0xc001fcad90, 0x0, 0x0)\n\t\t/go/src/cnrm.googlesource.com/cnrm/pkg/krmtotf/krmtotf.go:57 +0x17d\n\tcnrm.googlesource.com/cnrm/pkg/krmtotf.KRMResourceToTFResourceConfigWithGCPManagedFields(...)\n\t\t/go/src/cnrm.googlesource.com/cnrm/pkg/krmtotf/krmtotf.go:46\n\tcnrm.googlesource.com/cnrm/pkg/controller/tf.(*Reconciler).sync(0xc0023d9cc0, 0x42ad800, 0xc00005c208, 0xc0020478c0, 0x42f4400, 0x0, 0x0)\n\t\t/go/src/cnrm.googlesource.com/cnrm/pkg/controller/tf/controller.go:229 +0x546\n\tcnrm.googlesource.com/cnrm/pkg/controller/tf.(*Reconciler).Reconcile(0xc0023d9cc0, 0xc002851c00, 0x17, 0xc002851be0, 0x19, 0x0, 0x0, 0x0, 0x0)\n\t\t/go/src/cnrm.googlesource.com/cnrm/pkg/controller/tf/controller.go:142 +0xbf3\n\tcnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc001d6b080, 0x35d66c0, 0xc00403d620, 0x35549da160000)\n\t\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x161\n\tcnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc001d6b080, 0x35549da160000)\n\t\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xae\n\tcnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc001d6b080)\n\t\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 +0x2b\n\tcnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00350ad30)\n\t\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f\n\tcnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00350ad30, 0x421e120, 0xc0028dce40, 0xc000c92b01, 0xc0005c9f80)\n\t\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xa3\n\tcnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00350ad30, 0x3b9aca00, 0x0, 0x1, 0xc0005c9f80)\n\t\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe2\n\tcnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00350ad30, 0x3b9aca00, 0xc0005c9f80)\n\t\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d\n\tcreated by cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x305\n\t"}

[caieo]

Okay, thanks for the update! That error reminded me of #313 , which should be fixed with our most recent release (1.33.0). Can you try upgrading your ConfigConnector version to 1.33.0 and see if your issue is fixed?

[travisrandolph-bestbuy]

@caieo We are using the advanced installation w/ the Kubernetes Operator. I don't see v1.33.0 available yet in the bucket gs://configconnector-operator/.

[caieo]

@travisrandolph-bestbuy , sorry about this, it should be available now -- can you try again?

[travisrandolph-bestbuy]

@caieo I've verified that v1.33.0 fixes my issue.

[caieo]

Great, thank you for reporting this and I'm glad our newest release fixes your issue! I will go ahead and close this issue.