Operator-sdk: Clientset generation broken after 0.0.7

[benceszikora]

I am trying to generate a clientset for my CRD so that I can use it outside the operator. Is there a way to do this with the current version of the SDK?

I used this hack previously on 0.0.7:

vendor/k8s.io/code-generator/generate-groups.sh all github.com/user/memcached-operator/pkg/generated github.com/user/memcached-operator/pkg/apis memcached:v1alpha1

This however seems to have broken on master (ab09a3d). The problem seems to be that the generated client expects to have the AddToScheme function in the api imported from: github.com/user/memcached-operator/pkg/apis/memcached/v1alpha1
However, in the latest version of the operator-sdk, it's one package up in apis.go:

└── memcached
    ├── addtoscheme_memcached_v1alpha1.go
    ├── apis.go
    └── v1alpha1
        ├── doc.go
        ├── register.go
        ├── memcached_types.go
        └── zz_generated.deepcopy.go

Am I missing something here? Or should the function be somehow moved back into v1alpha1?

Thanks!

Environment

• operator-sdk version: ab09a3d

[lilic]

@benceszikora Thanks for opening this. The operator-sdk has gone through API changes, have you seen the migration-guide? Hopefully following that it should make things more clear, if not feel free to come back to the issue and we can answer any further questions!

[benceszikora]

Hey @LiliC ! That's exactly what I followed. Generating the deepcopy functions still works perfectly fine, it's generating the clientset that I am having issues with. Maybe I am missing something, but it seems like the new version breaks assumptions that https://github.com/kubernetes/code-generator/tree/master/cmd/client-gen makes.

[paulcostinean]

Worth mentioning that in order for the client to be generated at all, the // +genclient tag has to be added here (directly above the type wrapped in the API).

[lilic]

@benceszikora You are supposed to aim for the following directory structure. It seems like your apis.go is not in the api directory.

You can still use the old generator if all you need is to generate the clientset, just avoid using master, and switch to one of the release branches instead. https://github.com/operator-framework/operator-sdk/tree/v0.0.7

[hasbro17]

@benceszikora Since the SDK only supports deepcopy-gen all the other generators probably won't work on pkg/apis for 0.1.0. We could add support for those if it makes sense.

In the meanwhile you can still hack around it by adding the // +genclient tag in your memcached_types.go.

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// Memcached is the Schema for the memcacheds API
// +k8s:openapi-gen=true
type Memcached struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MemcachedSpec   `json:"spec,omitempty"`
    Status MemcachedStatus `json:"status,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// MemcachedList contains a list of Memcached
type MemcachedList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []Memcached `json:"items"`
}

And as you pointed out we're missing AddToScheme in register.go. So you'll need to add that in:

AddToScheme = SchemeBuilder.AddToScheme

And then try:

vendor/k8s.io/code-generator/generate-groups.sh all github.com/user/memcached-operator/pkg/generated github.com/user/memcached-operator/pkg/apis memcached:v1alpha1

Other generators like lister-gen, defaulter-gen might need other tags and methods to be present. We'll look into supporting all generator types by default.

[indradhanush]

@hasbro17 I was running into the same issue and tried your fix. I was able to generate the client set, but some bits of it still looks incorrect to me. For eg: https://github.com/indradhanush/debug-operator/blob/7ad260664fcb9626dc7a764e9974d7abab8b248e/pkg/client/clientset/versioned/typed/foo/v1alpha1/debuglist.go#L43

I created the above repo to easily reproduce the issue. Am I missing something?

[indradhanush]

My apologies. It appears that I had the tags placed above the wrong struct. This commit seems to have fixed it for me: https://github.com/indradhanush/debug-operator/commit/0cccb2a426ffdf336815f308d6ce237a55f5eda3

[lilic]

I think this can be closed as it's outdated.

[wreed4]

@LiliC What is the best practice for generating a golang client for CRDs produced with the operatorSDK?? It appears no client is automatically generated at this point.

[hasbro17]

@wreed4 The SDK will most likely not support generating typed clients(nor listers,informers) for your CRD since the expectation is to use the controller-runtime's dynamic client to interact with your CRD objects.
https://github.com/operator-framework/operator-sdk/blob/master/doc/user/client.md

If you do have a use case that can't be fulfilled by the dynamic client then please feel free to follow up at #1646.

In the meanwhile if you do need to generate a client-set for some reason then you can use the upstream k8s code-generator. There's a generate-groups.sh helper script that you can run against your pkg/apis/... to generate client/lister/informer for your CRD.

[wreed4]

@hasbro17 I should have been more explicit I think, our use-case is to interact with these CRDs from various other repositories/microservices that are not themselves operators. Is requiring the controller-runtime as a dependency still the best-practice you suggest?

Put slightly differently, we have developed an operator and now we are developing several microservices that are themselves _clients_ of that operator interacting with the CRs it manages (CRUD operations).

[hasbro17]

@wreed4 If you need other projects outside your operator to use to be clients of your operator's API then you can use still the controller-runtime's client except you want to instantiate a new one that talks directly to the API server.
And also make sure you add the CRD type that you want to the client's scheme as you instantiate it.
Here's a quick example of a non-operator project listing out Pods and an AppService type objects using the dynamic client.

import (
    "context"
    "fmt"
    "os"

    crdapi "github.com/hasbro17/app-operator/pkg/apis"
    appv1alpha1 "github.com/hasbro17/app-operator/pkg/apis/app/v1alpha1"

    corev1 "k8s.io/api/core/v1"
    runtime "k8s.io/apimachinery/pkg/runtime"
    clientgoscheme "k8s.io/client-go/kubernetes/scheme"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/client/config"
)

func main() {
    scheme := runtime.NewScheme()
    clientgoscheme.AddToScheme(scheme)
    crdapi.AddToScheme(scheme)

    cl, err := client.New(config.GetConfigOrDie(), client.Options{Scheme: scheme})
    if err != nil {
        fmt.Println("failed to create client")
        os.Exit(1)
    }

    podList := &corev1.PodList{}
    appList := &appv1alpha1.AppServiceList{}

    err = cl.List(context.Background(), client.InNamespace("default"), podList)
    if err != nil {
        fmt.Printf("failed to list pods in namespace default: %v\n", err)
        os.Exit(1)
    }

    fmt.Printf("PodList: %+v", podList)

    err = cl.List(context.Background(), client.InNamespace("default"), appList)
    if err != nil {
        fmt.Printf("failed to list appservices in namespace default: %v\n", err)
        os.Exit(1)
    }

    fmt.Printf("AppServiceList: %+v", appList)
}

You can read more at:
https://github.com/operator-framework/operator-sdk/blob/master/doc/user/client.md#non-default-client
https://godoc.org/github.com/kubernetes-sigs/controller-runtime/pkg/client#New

Hopefully that should cover your use case for typed clients.