K8s-config-connector: `client-go` Code Generation

[axisofentropy]

We're planning to use Config Connector through the Go Kubernetes Client and it's pretty difficult to use with Custom Resource Definitions like those used by Config Connector. Is there an accompanying extra library to make this easier? Maybe y'all can use code generation to supply a library?

[spew]

Hi @axisofentropy, can you describe a bit more about your application which requires it to talk to the K8s APIs directly? Is this simply so you can avoid calling out to kubectl within a go application that creates infrastructure or something different?

[axisofentropy]

Yeah lots of automation, provisioning resources for each of our customers.
And coordinating with a database, so we're keeping tight transactions.

[caieo]

Hi @axisofentropy , thanks for putting this on our radar and providing us the context in which you will be using this. We don't have immediate plans to supply an extra library for interacting directing with K8s API yet, but it is a feature that has been requested by other customers and is important for us to get to. We'll make sure to update this thread when we add it to Config Connector!

[n1koo]

As is, this tool is pretty useless for any end-to-end automation like provisioning a database and letting a pod use it directly.

Deployment etc resources can't use any data in CRs so even though we have Host address etc in the Status field of the CR that cannot be consumed directly by Pods.

I was thinking of building a tool to bridge this gap and output these things that Pods need to ConfigMaps and Secrets but since libraries aren't available its pretty clumsy.

Right now best option is to dynamically recreate these in your own code with eg. dynamic support ( https://soggy.space/namespaced-crds-dynamic-client/ ) but since we have no idea how the API is gonna change this is not only error prone but likely to break in the future.

[spew]

Hi @n1koo ,

Thanks for sharing the limitations you are running into. Referencing the fields & values in a config connector resource is something we want to provide help with and are experimenting with some solutions right now.

[n1koo]

@spew awesome! I understand config-connector is in beta and that you mentioned that you are only just testing, but any idea of a timeline where we might be able to take it for a drive?

[spew]

Unfortunately I cannot share more in this forum, but if you would like we can schedule a video-meeting and go over what we are thinking.

[AeroNotix]

Being able to create google resources via kubectl is very useful and a very welcome addition to using kubernetes on GCP. That said, there are definite use-cases where people would like to create their own operators which include creating Google resources. Our team is developing operators which already create Google resources, but we use a different library for this - https://github.com/paulczar/gcp-cloud-compute-operator - since Google has released k8s-config-connector the ultimate win would be if Google was the provider of a similar library.

Having access to the source code simply to reference the types in our own operators makes life much simpler.

[derekperkins]

We're in the same boat as @AeroNotix, using kube-builder for an internal operator that provisions all GCP resources for dev applications.
cc @whizard

[whizard]

Any update for this? Without this it is very difficult to create operators to use GCP resources. Could someone at least give us options on how to do this?

[spew]

Hello @whizard we are prioritizing this for the team as this is an important ask from multiple customers. I apologize for the delay, at this time I do not have a date to share, but will update when we do.

[whizard]

Thanks for the response. We are eagerly waiting for your update.

[errordeveloper]

I am also very keen to see this, I am going to try something in the meantime and post an update if I find a feasible solution.

[errordeveloper]

I did find that it was quite easy to use turn JSON manifests into unstructured.UnstructuredList and then I was able to pass those to k8s.io/controller-runtime client. The only trick I've had to implement was scheme registration, I basically have this:

// Package  API Schema definitions for the CNRM API
package cnrm

import (
        "k8s.io/apimachinery/pkg/runtime/schema"
        "sigs.k8s.io/controller-runtime/pkg/scheme"
)

var (
        GroupVersion  = schema.GroupVersion{Group: "container.cnrm.cloud.google.com", Version: "v1beta1"}
        SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}
        AddToScheme   = SchemeBuilder.AddToScheme
)

And in main.go, the init function looks like this:

func init() {
        _ = clientgoscheme.AddToScheme(scheme)

        _ = myapiv1alpha1.AddToScheme(scheme)
        // +kubebuilder:scaffold:scheme

        // register Config Connector scheme, so that the client can access its objects
        _ = cnrm.AddToScheme(scheme)
}

[jonnylangefeld]

I tried that too for a while and was happy with it at first, but then ran into issues with indexes.
For instance, this sets up an index to filter by owner:

// SetupOwnerIndexer creates an index on an object to quickly filter by owner
func SetupOwnerIndexer(mgr ctrl.Manager, obj runtime.Object, key string, apiVersion string, kind string) error {
    return mgr.GetFieldIndexer().IndexField(obj, key, func(rawObj runtime.Object) []string {
        o, err := runtime.DefaultUnstructuredConverter.ToUnstructured(&rawObj)
        if err != nil {
            return nil
        }
        u := &unstructured.Unstructured{Object: o}
        owner := metav1.GetControllerOf(u)
        if owner == nil {
            return nil
        }
        if owner.APIVersion != apiVersion || owner.Kind != kind {
            return nil
        }

        return []string{owner.Name}
    })
}

This function works perfectly fine on structured runtime object. I can run something like

    ccl := &containerv1beta1.ContainerClusterList{}
    if err := r.List(ctx, ccl, client.MatchingFields{ownerKey, "my-controller"}); err != nil {
        return err
    }

and get all controlled objects.
But I couldn't figure out how to make this work with unstructed objects. It would always tell me that the index doesn't exist.

[spew]

Hello, we will release generated go clients in the very near future.

[evalsocket]

@spew Do you have any date for release ?

[caieo]

Hi @evalsocket, we're aiming to release this in next week's release.

[caieo]

Update: we ran into a blocker that delayed this release but we have all the go clients for our CRDs staged for next week's release.

[aw185176]

Any update on this?

[kibbles-n-bytes]

Our latest release, 1.39.0, includes the initial release of the Go structs and client libraries: https://github.com/GoogleCloudPlatform/k8s-config-connector/tree/master/generated/pkg . Please give things a try and let us know if you have any issues.

[toumorokoshi]

@caieo can we close this issue? since the go clients are not available.

[caieo]

Yep, we can close this issue now that go-clients are available.