K8s-config-connector: Addon permissions/reconcile error

[jjhuff]

Describe the bug
I have several namespaces that are non-functional with errors like:

{"level":"error","ts":1592854739.1985295,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"iampolicymember-controller","request":"stage/pipeline-stage-storage-pubsub-editor","error":"error getting policy member: error fetching live state for resource: error reading underlying resource: Error reading Resource \"project \\\"projects/biomarkerbase-195223\\\"\" with IAM Member: Role \"roles/pubsub.editor\" Member \"serviceAccount:[email protected]\": Error retrieving IAM policy for project \"projects/biomarkerbase-195223\": googleapi: Error 403: The caller does not have permission, forbidden","stacktrace":"cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr/zapr.go:128\ncnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ncnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

The other namespaces reconcile just fine. I've tried reinstalling for those namespaces and I get the same issue. The per-namespace controller SAs have 'Owner' permissions on the project. The KSAs seem fine as well as the Workload Identity bindings.

I'm at a loss on further debugging.

ConfigConnector Version
1.7.2 AddOn mode

[spew]

Hi @jjhuff can you the following

1. Use this command to open a shell with the cloud-sdk image, replace ksa-name and ksa-namespace with your appropriate values:

kubectl run -it \
  --image google/cloud-sdk:slim \
  --serviceaccount ksa-name \
  --namespace k8s-namespace \
  workload-identity-test

1. Validate your identity, confirm this matches the SA you expect: gcloud auth list.

2. Try describing the IAM policy for the project using gcloud (if this doesn't work then KCC won't be able to do it either): gcloud proejcts get-iam-policy biomarkerbase-195223

[lynncyrin]

I just had a similar problem, and running that command made the source very obvious! It's a very nice command, maybe add it to the troubleshooting docs? At any rate, thanks both ^^

[spew]

Thanks @lynncyrin for your suggestion -- you are right that it would be a good thing to add to the troubleshooting steps. Will do.

[jjhuff]

Hey @spew Yup, the service account picked up by gcloud auth list was correct and get-iam-policy worked fine.

[xiaobaitusi]

hi @jjhuff, does gcloud projects get-iam-policy biomarkerbase-195223 return [email protected] having the roles/owner?

Can you also run gcloud iam service-accounts get-iam-policy [email protected]?
There should be a member serviceAccount:[HOST_PROJECT_ID].svc.id.goog[cnrm-system/cnrm-controller-manager-${NAMESPACE} roles/iam.workloadIdentityUser.

Note that HOST_PROJECT_ID is the project which contains the GKE cluster with KCC addon enabled, biomarkerbase-195223 is the managed project (thought it could be the host project).

[jjhuff]

@xiaobaitusi
No, pipeline-stage doesn't have roles/owner -- and I don't think it should. crnm-stage does, however. pipeline-stage is the IAMServiceAccount I'm trying to create (it does happen to already exist), etc.

# gcloud iam service-accounts get-iam-policy [email protected]
bindings:
- members:
  - serviceAccount:biomarkerbase-195223.svc.id.goog[cnrm-system/cnrm-controller-manager-stage]
  role: roles/iam.workloadIdentityUser

[xiaobaitusi]

aha. Sorry for the misunderstanding. To have more information to debug, can you run the following commands and post the results? I assume that your namespace is stage

kubectl get ns stage -oyaml
kubectl get serviceaccount cnrm-controller-manager-stage -n cnrm-system -oyaml
kubectl get configconnectorcontext configconnectorcontext.core.cnrm.cloud.google.com -n stage -oyaml
kubectl get IAMPolicyMember [PolicyMemberName] -n stage

Plus, can you try to create some other resources other than IAM policy member in stage namespace and see if they can be reconciled or not?

[jjhuff]

No worries!

$ kubectl get ns stage -oyaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    cnrm.cloud.google.com/project-id: biomarkerbase-195223
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{"cnrm.cloud.google.com/project-id":"biomarkerbase-195223"},"name":"stage"}}
  creationTimestamp: "2019-09-20T15:49:51Z"
  name: stage
  resourceVersion: "68262891"
  selfLink: /api/v1/namespaces/stage
  uid: 484b45a2-dbbe-11e9-a226-42010a800086
spec:
  finalizers:
  - kubernetes
status:
  phase: Active



$ kubectl get serviceaccount cnrm-controller-manager-stage -n cnrm-system -oyaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    cnrm.cloud.google.com/version: 1.7.2
    iam.gke.io/gcp-service-account: [email protected]
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{"cnrm.cloud.google.com/version":"1.7.2","iam.gke.io/gcp-service-account":"[email protected]"},"labels":{"cnrm.cloud.google.com/scoped-namespace":"stage","cnrm.cloud.google.com/system":"true","core.cnrm.cloud.google.com/configconnector":"configconnector.core.cnrm.cloud.google.com"},"name":"cnrm-controller-manager-stage","namespace":"cnrm-system","ownerReferences":[{"apiVersion":"core.cnrm.cloud.google.com/v1beta1","blockOwnerDeletion":true,"controller":true,"kind":"ConfigConnector","name":"configconnector.core.cnrm.cloud.google.com","uid":"eaceb57e-819c-49b0-8b8b-42a069866ad3"}]}}
  creationTimestamp: "2020-06-25T20:53:25Z"
  labels:
    cnrm.cloud.google.com/scoped-namespace: stage
    cnrm.cloud.google.com/system: "true"
    core.cnrm.cloud.google.com/configconnector: configconnector.core.cnrm.cloud.google.com
  name: cnrm-controller-manager-stage
  namespace: cnrm-system
  ownerReferences:
  - apiVersion: core.cnrm.cloud.google.com/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: ConfigConnector
    name: configconnector.core.cnrm.cloud.google.com
    uid: eaceb57e-819c-49b0-8b8b-42a069866ad3
  resourceVersion: "202732615"
  selfLink: /api/v1/namespaces/cnrm-system/serviceaccounts/cnrm-controller-manager-stage
  uid: 1c8b0844-0942-4639-a7e2-671edd0d5ca1
secrets:
- name: cnrm-controller-manager-stage-token-dvjws


$ kubectl get configconnectorcontext configconnectorcontext.core.cnrm.cloud.google.com -n stage -oyaml
apiVersion: core.cnrm.cloud.google.com/v1beta1
kind: ConfigConnectorContext
metadata:
  creationTimestamp: "2020-06-25T20:52:48Z"
  finalizers:
  - configconnector.cnrm.cloud.google.com/finalizer
  generation: 2
  name: configconnectorcontext.core.cnrm.cloud.google.com
  namespace: stage
  resourceVersion: "202732254"
  selfLink: /apis/core.cnrm.cloud.google.com/v1beta1/namespaces/stage/configconnectorcontexts/configconnectorcontext.core.cnrm.cloud.google.com
  uid: f73457fc-758e-44f0-9467-1d591f5a326f
spec:
  googleServiceAccount: [email protected]
status:
  healthy: true

The IAMPolicyMembers are all reconciled fine:

$ kubectl get IAMPolicyMember  -n stage  pipeline-stage-storage-admin -o yaml
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"iam.cnrm.cloud.google.com/v1beta1","kind":"IAMPolicyMember","metadata":{"annotations":{},"name":"pipeline-stage-storage-admin","namespace":"stage"},"spec":{"member":"serviceAccount:[email protected]","resourceRef":{"apiVersion":"resourcemanager.cnrm.cloud.google.com/v1beta1","external":"projects/biomarkerbase-195223","kind":"Project"},"role":"roles/storage.admin"}}
  creationTimestamp: "2020-06-25T21:11:51Z"
  finalizers:
  - cnrm.cloud.google.com/finalizer
  - cnrm.cloud.google.com/deletion-defender
  generation: 180
  name: pipeline-stage-storage-admin
  namespace: stage
  resourceVersion: "202760216"
  selfLink: /apis/iam.cnrm.cloud.google.com/v1beta1/namespaces/stage/iampolicymembers/pipeline-stage-storage-admin
  uid: 911f7f2d-321f-425b-a8cd-d66cbc7591b8
spec:
  member: serviceAccount:[email protected]
  resourceRef:
    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
    external: projects/biomarkerbase-195223
    kind: Project
  role: roles/storage.admin
status:
  conditions:
  - lastTransitionTime: "2020-06-25T21:32:12Z"
    message: The resource is up to date
    reason: UpToDate
    status: "True"
    type: Ready

However, IAMServiceAccount's aren't:

$ kubectl get IAMServiceAccount  -n stage  pipeline-stage -oyaml
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
  annotations:
    cnrm.cloud.google.com/management-conflict-prevention-policy: none
    cnrm.cloud.google.com/project-id: biomarkerbase-195223
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"iam.cnrm.cloud.google.com/v1beta1","kind":"IAMServiceAccount","metadata":{"annotations":{},"name":"pipeline-stage","namespace":"stage"},"spec":{"displayName":"Pipeline - stage"}}
  creationTimestamp: "2020-06-25T21:11:48Z"
  generation: 1
  name: pipeline-stage
  namespace: stage
  resourceVersion: "202748017"
  selfLink: /apis/iam.cnrm.cloud.google.com/v1beta1/namespaces/stage/iamserviceaccounts/pipeline-stage
  uid: d5f30806-d03b-4a32-acf1-3e78620542ea
spec:
  displayName: Pipeline - stage

The same object reconciles fine in other namespaces:

$ kubectl get IAMServiceAccount  -n dev  pipeline-dev -oyaml
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
  annotations:
    cnrm.cloud.google.com/management-conflict-prevention-policy: none
    cnrm.cloud.google.com/project-id: biomarkerbase-195223
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"iam.cnrm.cloud.google.com/v1beta1","kind":"IAMServiceAccount","metadata":{"annotations":{},"name":"pipeline-dev","namespace":"dev"},"spec":{"displayName":"Pipeline - dev"}}
  creationTimestamp: "2020-06-22T18:27:35Z"
  finalizers:
  - cnrm.cloud.google.com/finalizer
  - cnrm.cloud.google.com/deletion-defender
  generation: 9
  name: pipeline-dev
  namespace: dev
  resourceVersion: "197575604"
  selfLink: /apis/iam.cnrm.cloud.google.com/v1beta1/namespaces/dev/iamserviceaccounts/pipeline-dev
  uid: 6349d8fa-1886-4203-ac00-d0effba5521c
spec:
  displayName: Pipeline - dev
status:
  conditions:
  - lastTransitionTime: "2020-06-22T19:08:44Z"
    message: The resource is up to date
    reason: UpToDate
    status: "True"
    type: Ready
  email: [email protected]
  name: projects/biomarkerbase-195223/serviceAccounts/[email protected]
  uniqueId: "103371522474544416052"

[jjhuff]

Also, I tried creating a new IAMServiceAccount in stage and it doesn't reconcile.
The cnrm annotations were added, but that's it.

Spec was:

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMServiceAccount
metadata:
  name: jjhuff-test
  namespace: stage
spec:
  displayName: jjhuff-test

[xiaobaitusi]

It looks that the service account CRs were not picked up by the controller for some reason.
How many service account CRs are in the stage namespace? If you wait long enough, will it be reconciled?
Do you mind trying creating a new testing IAMServiceAccount CR and then checking the log of the controller pod after a while?
kubectl logs pod/cnrm-controller-manager-stage-0 -n cnrm-system manager

There should be some logging from iamserviceaccount-controller like
{"level":"info","ts":1594073639.0799773,"logger":"iamserviceaccount-controller","msg":"starting reconcile","resource":{"namespace":"test-ns","name":"iamserviceaccount-sample"}}?

[jjhuff]

@xiaobaitusi Sorry for taking a bit to get back. Yeah, I did add a test CR and it didn't reconcile. There weren't very many (3 or 4), and I tried over several days, so I don't think it was time.

However, it IS working now! I did do some cluster upgrades. It looks like I'm running 1.10.0 now. So, fixed, I guess.