K8s-config-connector: Error to create service account that has been recently deleted

[jonnylangefeld]

Describe the bug
I recently ran into an error, that is probably an edge case, but I thought I would let you guys know.
I created an IAMPolicyMember and got the following error in the cnrm manager logs:

{
    "level": "error",
    "ts": 1600362898.9673097,
    "logger": "controller-runtime.controller",
    "msg": "Reconciler error",
    "controller": "iampolicymember-controller",
    "request": "clusters/mango-roles-monitoring.metricwriter",
    "error": "error setting policy member: error applying changes: Request \"Create IAM Members roles/monitoring.metricWriter serviceAccount:<sa-name>@<project-id>.iam.gserviceaccount.com for \\\"project \\\\\\\"projects/<project-id>\\\\\\\"\\\"\" returned error: Batch request and retried single request \"Create IAM Members roles/monitoring.metricWriter serviceAccount:<sa-name>@<project-id>.iam.gserviceaccount.com for \\\"project \\\\\\\"projects/<project-id>\\\\\\\"\\\"\" both failed. Final error: Error applying IAM policy for project \"projects/<project-id>\": Error setting IAM policy for project \"projects/<project-id>\": googleapi: Error 400: Inconsistent representation of deleted member: 'deleted:serviceAccount:<sa-name>@<project-id>.iam.gserviceaccount.com?uid=104998493554765980505' and 'serviceAccount:<sa-name>@<project-id>.iam.gserviceaccount.com' cannot be on the policy simultaniously., badRequest",
    "stacktrace": "cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/github.com/go-logr/zapr/zapr.go:128\ncnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\ncnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\ncnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ncnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/cnrm.googlesource.com/cnrm/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"
}

I did a little bit of research and found that gcp allows to undelete service accounts. It does that by saving a service account of the schema deleted:serviceAccount:{emailid}?uid={uniqueid} (source).

Now when I recreated that service account I ran into the error and had to manually delete the remainders of the previously created service account.

Since we use the config connector to heavily automate our stack, it would be great if this scenario could be detected and in this case the undelete method should be called, rather than the create method. Or maybe there's a force delete to not even keep the deleted:serviceAccount around?

ConfigConnector Version
Run the following command to get the current ConfigConnector version
1.20.1

To Reproduce
Steps to reproduce the behavior:

• create IAMPolicyMember
• delete IAMPolicyMember
• create same IAMPolicyMember again

YAML snippets:

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
  name: test
  namespace: default
spec:
  member: test-default@<project-id>.iam.gserviceaccount.com
  resourceRef:
    apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
    external: projects/<project-id>
    kind: Project
  role: roles/containeranalysis.occurrences.viewer

[spew]

Hi @jonnylangefeld thanks for the detailed reproduction steps of this scenario. We will look into supporting this and get back to you.

[TravRan]

This is also impacting our team. I would also assume that if an SA with the same name is detected in a deleted state that it would undelete it instead of creating a new SA.

[travisrandolph-bestbuy]

@spew Has there been any update on this issue? This is still impacting our team.

[jcanseco]

Hi @travisrandolph-bestbuy, thanks for checking in. No updates yet from our front, though we're aware of the issue and we have some internal discussions coming up to discuss the issue.

For now, our current recommendation is to have a periodic job strip deleted: members if possible.