K8s-config-connector: Folder / Project - make folder and org id a resourceRef (external or direct)

[tedelwartowski-bestbuy]

This enhancement request is similar to what has been requested in #181 however I would like to propose changing how the reference to a folder or org id is handled in both a Folder and Project resource. Currently, reference to a folder or org id is handled via an annotation which makes it hard to maintain referential relationships between folders and projects. I would like to propose switching the reference of the folder or org id to be a resourceRef (external or direct) such that we can then maintain a referential hierarchy directly with our config connector manifests. Without this, maintaining relationships between folders and projects is manually intensive and impedes on our ability to properly implement automation for the platform.

[spew]

Thanks @tedelwartowski-bestbuy for this suggestion. What you described is the direction we want to take the product with eventually de-emphasizing the annotation.

[tedelwartowski-bestbuy]

@spew - thank you for confirming this is something for consideration on the product roadmap. I believe #181 would still be relevant as an interim solution until what has been proposed here can be implemented.

[spew]

We are looking into accelerating the timeline on #181 @jcanseco for visibility

[travisrandolph-bestbuy]

@spew @jcanseco Has there been any progress on this feature?

[jcanseco]

Hi @travisrandolph-bestbuy and @tedelwartowski-bestbuy, yes this is something we're working on actively. There's a design proposal undergoing internal review right now meant to address the particular problem described in this issue. We'll let you know when we have more updates on progress.

[jcanseco]

Hi all, we've added support for folderRef and organizationRef references in Project and Folder in 1.43.0 which should give you the ability to reference Folder resources by name.

Please note, however, that there is a known issue where you will not be able to delete a Folder and its child resources at the same time. Otherwise, the deletion will get stuck. You must either delete the parent Folder first and wait for it to be gone from the K8s API Server or delete the child resources first. See the Warning here for more details. We intend to look into how to fix this issue in the future to remove the need to order your deletions.

Also note that the folder-id and organization-id annotations for Project and Folder are now deprecated as a result. To migrate Project and Folder resources across folders/orgs, please update the folderRef or organizationRef fields instead. Note that migrations from folder to organization and vice versa are still not supported, however.

To maintain consistency across our resources, we intend to roll out support for projectRef, folderRef, and organizationRef in place of the project-id, folder-id, and organization-id annotations for our other resources over the next few weeks.

[yashsaini77]

So when we want to create a project in a folder so in the manifest file of the project, how do we reference the folder id if we are trying to work in an automated way?

[yashsaini77]

If we want to create a folder and then create a project under that folder then how do we reference the folder in the resource Project's manifest file here also remember we need the automation? @jcanseco

apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Project
metadata:
annotations:
# Set this field after finding the folder number.
cnrm.cloud.google.com/folder-id: ""
cnrm.cloud.google.com/auto-create-network: "false"
name: kcc-dev
spec:
name: kcc-dev
billingAccountRef:
# Replace "${BILLING_ACCOUNT_ID?}" with the numeric ID for your billing account
external: ""

[jcanseco]

Here's an example:

apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Project
metadata:
  name: my-project
spec:
  name: My Project
  folderRef:
    name: my-folder
  billingAccountRef:
    external: "AAAAA-BBBBB-CCCCC-DDDDD"
---
apiVersion: resourcemanager.cnrm.cloud.google.com/v1beta1
kind: Folder
metadata:
  name: my-folder
spec:
  displayName: My Folder
  organizationRef:
    external: "123456789"

[yashsaini77]

So both resources will be in the same manifest and the dependency will be done by itself like firstly folder will be created and then the project will be created under that folder. right? @jcanseco

[jcanseco]

the dependency will be done by itself like firstly folder will be created and then the project will be created under that folder. right?

Yep that's correct @yashsaini77.

The Project and Folder don't _need_ to be in the same manifest file though. They can be in different files which you can kubectl apply separately.

[tonybenchsci]

"Also note that the folder-id and organization-id annotations for Project and Folder are now deprecated as a result. "

@jcanseco @spew We actually worked out a directory structure using the annotation method. Is there no plan to keep both annotation and resourceRef working? Are there migration instructions? (because it sounds like upgrading to v1.43.0 may result in errors now)

Basically we have a namespace in our KCC GKE cluster where the namespace is annotated using cnrm.cloud.google.com/organization-id, and then just have all the kind: Project resources in this namespace. Will this be broken if we upgrade to the latest KCC version? Note that we don't apply the annotation directly to Projects, just the namespace that they belong to (though from docs I believe that means the Projects no longer inherit the annotation from the namespace after v1.43.0)

"To maintain consistency across our resources, we intend to roll out support for projectRef, folderRef, and organizationRef in place of the project-id, folder-id, and organization-id annotations for our other resources over the next few weeks."

Doesn't that mean that the entire doc section in https://cloud.google.com/config-connector/docs/how-to/organizing-resources/overview will be obsolete in a few weeks?

[jcanseco]

Hi @tonybenchsci. Your set-up should continue to work just fine. We made sure to keep backwards compatibility in mind when developing this feature.

Here are the important points:

• Creating new resources with an annotation and without a reference will continue to work. We will just default a reference value for you based on the annotation. For example, if you create a folder-scoped resource with a folder-id: 12345 annotation, we will default a spec.folderRef.external: 12345 field for you.
• Creating new resources without an annotation or reference in a namespace that has an annotation will continue to work. We will just default a reference value for you based on the annotation. This will hold true even for future resources that never supported the annotations to begin with but do support the references.
• Resources that already exist in your cluster which have an annotation and no reference will be given a reference value based on the annotation. This means there should be no issues for existing resources.

The only real behavioral change is the following:

• If you want to migrate a Project/Folder resource across folders/orgs, you must update the folderRef/organizationRef field instead of the folder-id/organization-id annotation since annotations are ignored once a resource has a reference.
• The above also means that you can choose to remove the annotations from your resources once they have a reference if desired.

Doesn't that mean that the entire doc section in https://cloud.google.com/config-connector/docs/how-to/organizing-resources/overview will be obsolete in a few weeks?

Sort of, and we do plan to update it. Most of it still holds true though. The key difference is that the source of truth for which project/folder/organization a resource belongs to is now the resource's projectRef/folderRef/organizationRef rather than the resource's project-id/folder-id/organization-id annotation.

Before, namespace-level annotations were used to default resource-level annotations. Now, namespace-level annotations will be used to default references.

Resources which already support resource-level annotations will continue to support them as a legacy feature, and they will be converted to references. Future resources will only support references. This is what I meant when I said that annotations will be deprecated.

@tonybenchsci I hope that alleviates the concerns somewhat. Please let me know if you have any other questions.

[tonybenchsci]

@jcanseco Thank you for the excellent explanation. We'll be upgrading now with the assurance of backwards compatibility; also acknowledging that migrating these resources in the future requires updating the new source-of-truth resourceRef

[jcanseco]

Hi @tonybenchsci, we recommend delaying an upgrade for now. We just detected an issue where the defaulting of references doesn't quite work when applying resources with --server-side (the default in K8s 1.18+).

You should not have the problem if you're not applying resources with --server-side and if you're using an older version of K8s than 1.18, but that may be unlikely since K8s 1.18 is now the default for the REGULAR channel in GKE.

We're working on a fix at the moment and will put out an update soon. Note that the only resources affected are Project and Folder as we have not yet rolled out references support for other resources.

Workaround: There is no problem if you explicitly specify folderRef / organizationRef yourself, but this obviously is not an easy option for users who are relying on the references being defaulted from namespace-level or resource-level folder-id / organization-id annotations.

[tonybenchsci]

@jcanseco Thanks but I already upgraded at the time of my last comment :)
We're on 1.17.17-gke.1101 in STABLE channel, so hopefully your patch timing works out.

[jcanseco]

Hi @tonybenchsci, yep we've put out a fix in KCC 1.45.0.

[tonybenchsci]

Thanks @jcanseco
BTW, the documentation (e.g. https://cloud.google.com/config-connector/docs/reference/resource-docs/bigquery/bigquerydataset) says projectRef is "Required". Shouldn't it be optional? (because "Creating new resources with an annotation and without a reference will continue to work. ")

Another point: While we rely on cnrm.cloud.google.com/organization-id annotation for kind: Project, we actually mostly rely on the default behaviour of KCC namespace == GCP project_id which has neither annotations nor projectRef for resources we create in a GCP-project-named namespace. I assume this will also continue to work?

[jcanseco]

Shouldn't it be optional?

Yes this is docs improvement we can consider. I was thinking it'd make sense to link to the docs on how to set up default projects/folders/orgs via namespace annotations, once we get around to updating those docs :)

we actually mostly rely on the default behaviour of KCC namespace == GCP project_id which has neither annotations nor projectRef for resources we create in a GCP-project-named namespace. I assume this will also continue to work?

Yes that will continue to work (and is something we test).

[tonybenchsci]

@jcanseco We've upgraded to v1.47.0, but now get spec.projectRef: Required value For BigQueryDataset resources.
Is this intended? It wasn't required before because it was created in a namespace that is equal to the GCP project_id that the BQ dataset was created in.

So it seems like the defaulting behaviours are broken, at least for BigQueryDataset resources, have changed.

[jcanseco]

Hey @tonybenchsci, to confirm, are you saying you cannot create BigQueryDataset resources (without either the projectRef reference or project-id annotation) in namespaces without the project-id annotation?

I just tried to reproduce locally, but was able to create a BigQueryDataset just fine (it picked up the namespace name as the project ID).

Could you share some more debugging info:

1. Your YAML.
2. The full error message.
3. Where is it coming from -- presumably from the API server the moment the YAML is kubectl apply-ied ?

[tonybenchsci]

Thanks for getting back to me @jcanseco @spew . I should've given more information in my last comment :-)

1. YAML

apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
kind: BigQueryDataset
metadata:
  name: bsmining
  namespace: project-id1  # This is what we use to make sure the resource ends up in the right GCP Project
  owner: datarun
spec:
  description: "Data Extraction Pipeline Dataset"
  friendlyName: bsmining
  location: US

1. Error message is The BigQueryDataset "bsmining" is invalid: spec.projectRef: Required value
2. Coming from kubectl apply --dry-run=server (edit: any kubectl command!)

This resource has been created months ago where kubectl apply would always return unchanged in our GitOps CICD pipeline, which deploys entire namespaces with tens of resources. Now we're getting the error, which means backwards-compatibility is not satisfied (in this sense the documentation "mistake" is actually not a mistake). Is the only workaround to add spec.projectRef?

EDIT: This seems to be only an issue for BigQueryDatasets that don't have spec.resourceID and default to using metadata.name for the GCP resource naming. Weird.

EDIT2: kubectl -n=project-id1 annotate BigQueryDataset bsmining cnrm.cloud.google.com/deletion-policy=abandon returns The BigQueryDataset "bsmining" is invalid: spec.projectRef: Required value. Looks like this one resource is kaput.

EDIT3: kubectl edit or kubectl label also don't work! If I add the annotation in edit mode, I still get invalid: spec.projectRef: Required value, and if I add spec.projectRef: project-id1, I get immutable field(s): [projectRef]

EDIT4: Server-side dry-run for pruning (via kubectl apply --prune-whitelist) didn't give an error.

There might be something pretty flawed with the new projectRef implementions from this experience... Currently this resource is completely broken, because it cannot be abandoned. I cannot delete this resource because it contains critical data, and I suspect kubectl delete will also just return invalid: spec.projectRef: Required value

Is there another way to abandon? Maybe remove managed-by-cnrm: true label in the GCP console? EDIT6: Didn't work.

EDIT5: kubectl -n=project-id1 delete BigQueryDataset bsmining --force --grace-period=0 and kubectl -n=project-id1 delete BigQueryDataset bsmining --now Also don't work.

[jcanseco]

Hi @tonybenchsci, apologies that this is causing such a big problem.

I wasn't able to reproduce the error when I ran kubectl apply --dry-run=server (I saw unchanged as expected).

Can you share the following with us to help with debugging?

1. The current state of your BigQueryDataset:

   kubectl get bigquerydataset bsmining -n project-id1 -o yaml
   

2. Your K8s version:

   kubectl version --short
   

3. Can you ensure that all pods are READY (including the webhooks):

   kubectl get pods -n cnrm-system
   

[tonybenchsci]

3.

kubectl get pods -n cnrm-system
NAME                                            READY   STATUS    RESTARTS   AGE
cnrm-controller-manager-0                       2/2     Running   0          13h
cnrm-deletiondefender-0                         1/1     Running   0          22h
cnrm-resource-stats-recorder-55b65f9559-wfdbk   2/2     Running   0          22h
cnrm-webhook-manager-547bbf4cfb-cq9xl           1/1     Running   0          22h
cnrm-webhook-manager-547bbf4cfb-gd758           1/1     Running   0          22h

2.

Client Version: v1.18.18
Server Version: v1.18.16-gke.302

KCC is v1.47.0

1.

apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
kind: BigQueryDataset
metadata:
  annotations:
    cnrm.cloud.google.com/management-conflict-prevention-policy: resource
    cnrm.cloud.google.com/project-id: project-id1
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"bigquery.cnrm.cloud.google.com/v1beta1","kind":"BigQueryDataset","metadata":{"annotations":{},"name":"bsmining","namespace":"project-id1","owner":"datarun"},"spec":{"description":"Data Extraction Pipeline Dataset","friendlyName":"bsmining","location":"US"}}
  creationTimestamp: "2020-03-05T21:55:43Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2021-04-27T16:33:33Z"
  finalizers:
  - cnrm.cloud.google.com/finalizer
  - cnrm.cloud.google.com/deletion-defender
  generation: 51466
  managedFields:
  - apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cnrm.cloud.google.com/management-conflict-prevention-policy: {}
          f:cnrm.cloud.google.com/project-id: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:finalizers:
          v:"cnrm.cloud.google.com/deletion-defender": {}
          v:"cnrm.cloud.google.com/finalizer": {}
      f:spec:
        f:access: {}
        f:description: {}
        f:friendlyName: {}
        f:location: {}
      f:status:
        f:creationTime: {}
        f:selfLink: {}
    manager: before-first-apply
    operation: Update
  - apiVersion: bigquery.cnrm.cloud.google.com/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:cnrm-lease-expiration: {}
          f:cnrm-lease-holder-id: {}
          f:managed-by-cnrm: {}
      f:spec:
        f:resourceID: {}
      f:status:
        f:conditions: {}
        f:etag: {}
        f:lastModifiedTime: {}
    manager: cnrm-controller-manager
    operation: Update
    time: "2021-04-20T15:05:22Z"
  name: bsmining
  namespace: project-id1
  resourceVersion: "190279473"
  selfLink: /apis/bigquery.cnrm.cloud.google.com/v1beta1/namespaces/project-id1bigquerydatasets/bsmining
  uid: 0f88c058-5f2c-11ea-9811-42010a9601bc
spec:
  access:
  - role: OWNER
    userByEmail: [email protected]
  - role: OWNER
    specialGroup: projectOwners
  - role: READER
    specialGroup: projectReaders
  - role: WRITER
    specialGroup: projectWriters
  description: Data Extraction Pipeline Dataset
  friendlyName: bsmining
  location: US
  resourceID: bsmining
status:
  conditions:
  - lastTransitionTime: "2021-04-26T19:37:31Z"
    message: The resource is up to date
    reason: UpToDate
    status: "True"
    type: Ready
  creationTime: 1583445344540
  etag: Mrbo54kWrRf9TbSgGFyF5A==
  lastModifiedTime: 1619465851115
  selfLink: https://bigquery.googleapis.com/bigquery/v2/projects/project-id1/datasets/bsmining

If I do kubectl describe I sometimes get Updating for a long time.

[jcanseco]

Thanks @tonybenchsci . It's interesting that the BigQueryDataset on the API server didn't have a spec.projectRef defaulted.

Can you scan through the controller's logs and see if there are any errors?

kubectl logs -f -n cnrm-system cnrm-controller-manager-0

[tonybenchsci]

Well we have 100s of resources across many projects. So if I do kubectl logs -n cnrm-system cnrm-controller-manager-0 -c manager | grep bigquerydataset I'm now seeing way more failures than I anticipated.

Many of them are invalid: spec.projectRef: Required value while some are successfully finished reconcile
Note that we've never specified spec.projectRef on any of our BigQueryDatasets

[jcanseco]

Gotcha, could you search for any of the following strings:

1. error applying changes to resource
2. error ensuring resource
3. error with status update call to API server
4. error with update call to API server

My suspicion is that the controller is failing to add the spec.projectRef to the existing BigQueryDataset resources as would be expected.

[jcanseco]

To unblock yourself (i.e. abandon all your BigQueryDataset resources), you can follow these steps:

1. Disable the operator temporarily (so that it doesn't reset the changes we're about to make to the BigQueryDataset CRD):

   kubectl edit statefulset -n configconnector-operator-system configconnector-operator
   # Set spec.replicas to 0, then save and quit.
   

2. Modify the BigQueryDataset CRD to not require spec.projectRef:

   kubectl edit crd bigquerydatasets.bigquery.cnrm.cloud.google.com
   # Delete the field `spec.versions[].schema.openAPIV3Schema.properties.spec.required`
   # Note that there is only one element under `versions[]`, so you only need to remove one `required` field.
   

3. Verify that BigQueryDataset no longer requires projectRef:
   shell kubectl get crd bigquerydatasets.bigquery.cnrm.cloud.google.com -o jsonpath="{.spec.versions[].schema.openAPIV3Schema.properties.spec.required}" # Should return nothing

4. Try abandoning your BigQueryDataset resources. If that doesn't work, try deleting the finalizers.

5. Re-enable the operator:

   kubectl edit statefulset -n configconnector-operator-system configconnector-operator
   # Set spec.replicas to 1, then save and quit.
   

6. Confirm that BigQueryDataset requires spec.projectRef again (after a few minutes):

   kubectl get crd bigquerydatasets.bigquery.cnrm.cloud.google.com -o jsonpath="{.spec.versions[].schema.openAPIV3Schema.properties.spec.required}"
   # Should return:
   # ["projectRef"]
   

[tonybenchsci]

Thanks @jcanseco , but doesn't this just abandon all the BigQueryDatasets? We want to manage them.
Is this a real bug for which a fix will be released?

[jcanseco]

Ah yes, I provided steps for abandonment in case you needed them (e.g. if you're getting spammed by errors and just wish for them to stop).

Is this a real bug for which a fix will be released?

Yes, it could be a real bug so we're currently trying to reproduce the issue, but so far we haven't had any luck, so we're continuing to investigate.

If we can reproduce it, we will try to get a fix out ASAP since this would be considered a high-priority issue.

[tonybenchsci]

Gotcha, could you search for any of the following strings:

1. `error applying changes to resource`

2. `error ensuring resource`

3. `error with status update call to API server`

4. `error with update call to API server`

My suspicion is that the controller is failing to add the spec.projectRef to the existing BigQueryDataset resources as would be expected.

All of them are of the type 3:

{"severity":"error","logger":"controller-runtime.manager.controller.bigquerydataset-controller","msg":"Reconciler error","reconciler group":"bigquery.cnrm.cloud.google.com","reconciler kind":"BigQueryDataset","name":"REDACTED","namespace":"REDACTED","error":"error with status update call to API server: BigQueryDataset.bigquery.cnrm.cloud.google.com \"REDACTED\" is invalid: spec.projectRef: Required value"}

[jcanseco]

Thanks @tonybenchsci! That last comment helped us reproduce the issue. We'll work on a fix and update this thread accordingly.

[tonybenchsci]

I'm guessing 1.48.0 doesn't have a fix, right?

[jcanseco]

Hi @tonybenchsci, unfortunately no. We're working on a fix at the moment and will put out a patch release if we can get it in this week. We'll update you when that happens.

[jcanseco]

Hi @tonybenchsci, we should have a fix for you in 1.49.0. Could you try it out once it's out and let us know if it works for you? It should be out by EOD.

[tonybenchsci]

Great thanks @jcanseco
I see 1.49.0 on the releases page, but it seems incomplete, and it's not seen in gsutil ls gs://configconnector-operator yet.

[caieo]

@tonybenchsci, sorry for the delay in updating our release documentation and our operator. We'll have the operator storage bucket updated in an hour or so!

[caieo]

@tonybenchsci, 1.49.0 should now be up in gs://configconnector-operator/, please let us know if the fix works for you.

[tonybenchsci]

@caieo @jcanseco It worked after 1.49.0 thanks.
We're now stuck in a deletion state because of previous attempts that got unblocked as a result of the fix.

message: 'Delete call failed: error deleting resource: [{0 Error when reading
      or editing Dataset: googleapi: Error 400: Dataset project-id1:bsmining is still
      in use, resourceInUse  []}]'
    reason: DeleteFailed
    status: "False"

There must be a kubectl command to cancel a delete...
Nvm, now I can annotate with abandon policy and re-acquire

EDIT: This is now resolved for us. Thanks again for such a prompt turnaround!

[jcanseco]

Hi @tonybenchsci, great, happy to hear your issue has been resolved!

[tonybenchsci]

@jcanseco Is this expected? admission webhook "deny-immutable-field-updates.cnrm.cloud.google.com" denied the request: error validating container annotations: cannot add/change container annotation cnrm.cloud.google.com/folder-id as it is no longer supported by the resource; set one of [spec.folderRef, spec.organizationRef] instead.

I was expecting the folder-id annotation on a "kind: Project" to still be able to override the namespace level org-id annotation. And then have it do the automatic defaulting of folderRef, just like how the organizationRef is defaulted via the org-id annotation.

[jcanseco]

Hi @tonybenchsci, yes that is expected behavior. Unfortunately, certain cases become quite complex and confusing from a UX perspective by continuing to allow mutations of container annotations, so it was decided to go for a "stop the bleeding" approach by making container annotations immutable. Note that container annotations had been mutable for only Project and Folder since they could be migrated across parent resources.

As a side-note: we do allow _removal_ of container annotations once the resource has a hierarchical reference. This is done to allow users to prevent other people from potentially being confused by seeing both an annotation and a reference on a resource.