Operator-sdk: Deepcopy generation fails due to "unsupported type invalid type for invalid type"

Created on 22 Aug 2019 · 30Comments · Source: operator-framework/operator-sdk

Bug Report

When migrating an Operator that works with the SDK 0.8.2 to 0.10.0, the generate k8s command fails with a message like Hit an unsupported type invalid type for invalid type, from github.com/jaegertracing/jaeger-operator/pkg/apis/io/v1alpha1.Jaeger. It's unclear which property has an invalid type, so, I commented out all custom types, ending up with a type like this:

type Jaeger struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata"`
    // Spec              JaegerSpec   `json:"spec"`
    // Status            JaegerStatus `json:"status,omitempty"`
}

Even for this type, the generate k8s fails with the same message. The code for this can be seen on this branch: https://github.com/jpkrohling/jaeger-operator/tree/607-Update-Operator-SDK-to-0.10.0

What did you do?
Clone the branch mentioned above and run:

$ GO111MODULE=on operator-sdk generate k8s
INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [io:[v1alpha1], jaegertracing:[v1], ] 
F0822 11:01:11.492974   28811 deepcopy.go:885] Hit an unsupported type invalid type for invalid type, from github.com/jaegertracing/jaeger-operator/pkg/apis/io/v1alpha1.Jaeger
goroutine 1 [running]:
k8s.io/klog.stacks(0xc0003caf00, 0xc0006d4500, 0xb0, 0x100)
    pkg/mod/k8s.io/[email protected]/klog.go:855 +0xb1
k8s.io/klog.(*loggingT).output(0x33f8420, 0xc000000003, 0xc00030abd0, 0x323b24c, 0xb, 0x375, 0x0)
    pkg/mod/k8s.io/[email protected]/klog.go:806 +0x2d9
k8s.io/klog.(*loggingT).printf(0x33f8420, 0x3, 0x1dc88a2, 0x2a, 0xc0006d2e88, 0x3, 0x3)
    pkg/mod/k8s.io/[email protected]/klog.go:705 +0x14e
k8s.io/klog.Fatalf(...)
    pkg/mod/k8s.io/[email protected]/klog.go:1256
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).doStruct(0xc00002f900, 0xc0007e2790, 0xc0007fa280)
    pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:885 +0x7bf
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).generateFor(0xc00002f900, 0xc0007e2790, 0xc0007fa280)
    pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:695 +0xc5
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).GenerateType(0xc00002f900, 0xc0007896e0, 0xc0007e2790, 0x209c0c0, 0xc0008143e0, 0x0, 0x0)
    pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:608 +0xe86
k8s.io/gengo/generator.(*Context).executeBody(0xc0007896e0, 0x2098d80, 0xc00080e3e0, 0x2111580, 0xc00002f900, 0x60, 0x10)
    pkg/mod/k8s.io/[email protected]/generator/execute.go:304 +0x11d
k8s.io/gengo/generator.(*Context).ExecutePackage(0xc0007894a0, 0xc0004652c0, 0x25, 0x20e9e00, 0xc00002f500, 0x0, 0x0)
    pkg/mod/k8s.io/[email protected]/generator/execute.go:265 +0xbf7
k8s.io/gengo/generator.(*Context).ExecutePackages(0xc0007894a0, 0xc0004652c0, 0x25, 0xc0004af120, 0x1, 0x1, 0x0, 0xc000465320)
    pkg/mod/k8s.io/[email protected]/generator/execute.go:51 +0xc5
k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc00047b860, 0xc0006d3810, 0x1d6e857, 0x6, 0x1e62798, 0xc0005b7540, 0x16)
    pkg/mod/k8s.io/[email protected]/args/args.go:194 +0x2d7
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.deepcopyGen(0xc0005b4700, 0xe, 0xc0002e5a40, 0x2, 0x2, 0x18, 0xc0006d39b0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:94 +0x4d2
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen.func1(0xc0005b4700, 0xe, 0xc0005b4700, 0xe)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:53 +0x50
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.generateWithHeaderFile(0xc0002e5aa0, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/genutil.go:104 +0x178
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen(0xc0005af180, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:54 +0x4a5
github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate.k8sFunc(0xc0005af180, 0x341bfc0, 0x0, 0x0, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate/k8s.go:56 +0x156
github.com/spf13/cobra.(*Command).execute(0xc0005af180, 0x341bfc0, 0x0, 0x0, 0xc0005af180, 0x341bfc0)
    pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x465
github.com/spf13/cobra.(*Command).ExecuteC(0xc000546f00, 0x209d6c0, 0xc0003d0600, 0x0)
    pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
    pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:85 +0x50f

Environment

operator-sdk version: operator-sdk version: v0.10.0, commit: ff80b17737a6a0aade663e4827e8af3ab5a21170
go version: go version go1.12.5 linux/amd64

Additional context
Found while working on jaegertracing/jaeger-operator#607

kinbug

Source

jpkrohling

Most helpful comment

Alright, I _think_ I finally have this figured out.

When the operator-sdk binary is built from source, one of the transitive dependencies that gets pulled in by controller-tools for code generation is kubernetes/gengo.

kubernetes/gengo's AddDirRecursive function searches a set of paths, including GOROOT, when parsing Go code. It determines GOROOT using go/build.Default.GOROOT.

go/build.Default.GOROOT is initialized using runtime.GOROOT(), which first checks at runtime for the GOROOT environment variable and falls back to the compile-time GOROOT of the binary if GOROOT is not explicitly set in the environment. go env GOROOT does not appear to be used at runtime.

So when using the operator-sdk, it is necessary to either (a) set GOROOT in your environment, or (b) use the same GOROOT that was used to build the operator-sdk binary.

Obviously (b) is undesirable because you have to know what GOROOT was used and then force your environment to match it, so (a) setting GOROOT explicitly seems like the best workaround.

It appears that kubernetes/gengo is no longer used in controller-tools v0.2.0, so hopefully controller-tools's new mechanism for Go code parsing does not have this same problem. We should be switching to controller-tools v0.2.0 very soon.

/cc @estroz @hasbro17 @jpkrohling @mandymchu @kdwodc

TL;DR: If you run into strange problems with code generation, try setting GOROOT explicitly

joelanford on 27 Aug 2019

👍14 🎉10

All 30 comments

1810 might be relevant, but I can't test the same steps on a new project because of #1853.

jpkrohling on 22 Aug 2019

I was having the same issue yesterday.

go version go1.12.9 darwin/amd64
operator-sdk version: v0.10.0, commit: ff80b17737a6a0aade663e4827e8af3ab5a21170

This script will blow up when running the 'generate k8s' command:

!/usr/bin/env bash
export GO111MODULE=on
cd ~/go/src
mkdir example.com
cd example.com
operator-sdk new tester
cd tester
operator-sdk add api --api-version=example.com/v1 --kind=Tester
operator-sdk generate k8s
operator-sdk add controller --api-version=example.com/v1 --kind=Tester

and the error:

INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [example:[v1], ]
F0821 16:07:22.238693    3233 deepcopy.go:885] Hit an unsupported type invalid type for invalid type, from example.com/tester/pkg/apis/example/v1.Tester
goroutine 1 [running]:
k8s.io/klog.stacks(0xc000266200, 0xc0006fc0f0, 0x99, 0xe9)
        pkg/mod/k8s.io/[email protected]/klog.go:855 +0xb1
k8s.io/klog.(*loggingT).output(0x40010a0, 0xc000000003, 0xc0005979d0, 0x3e405a6, 0xb, 0x375, 0x0)
        pkg/mod/k8s.io/[email protected]/klog.go:806 +0x2d9
k8s.io/klog.(*loggingT).printf(0x40010a0, 0x3, 0x29c79f6, 0x2a, 0xc0006a0e88, 0x3, 0x3)
        pkg/mod/k8s.io/[email protected]/klog.go:705 +0x14e
k8s.io/klog.Fatalf(...)
        pkg/mod/k8s.io/[email protected]/klog.go:1256
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).doStruct(0xc000350300, 0xc00031f1e0, 0xc000160dc0)
        pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:885 +0x7bf
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).generateFor(0xc000350300, 0xc00031f1e0, 0xc000160dc0)
        pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:695 +0xc5
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).GenerateType(0xc000350300, 0xc0006f80c0, 0xc00031f1e0, 0x2c9c200, 0xc0000df3e0, 0x0, 0x299d142
)
        pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:608 +0xe86
k8s.io/gengo/generator.(*Context).executeBody(0xc0006f80c0, 0x2c98d80, 0xc0002a8cd0, 0x2d11ac0, 0xc000350300, 0x60, 0x10)
        pkg/mod/k8s.io/[email protected]/generator/execute.go:304 +0x11d
k8s.io/gengo/generator.(*Context).ExecutePackage(0xc00014f980, 0xc000052800, 0x1e, 0x2cea060, 0xc000350100, 0x0, 0x0)
        pkg/mod/k8s.io/[email protected]/generator/execute.go:265 +0xbf7
k8s.io/gengo/generator.(*Context).ExecutePackages(0xc00014f980, 0xc000052800, 0x1e, 0xc0005c4c30, 0x1, 0x1, 0x0, 0xc0005cc520)
        pkg/mod/k8s.io/[email protected]/generator/execute.go:51 +0xc5
k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc0001666e0, 0xc0006a1810, 0x296ddc9, 0x6, 0x2a61928, 0xc000052760, 0x15)
        pkg/mod/k8s.io/[email protected]/args/args.go:194 +0x2d7
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.deepcopyGen(0xc00034f080, 0x3a, 0xc000033410, 0x1, 0x1, 0x18, 0xc0006a19b0)
        src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:94 +0x4d2
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen.func1(0xc00034f080, 0x3a, 0xc00034f080, 0x3a)
        src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:53 +0x50
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.generateWithHeaderFile(0xc0002e7580, 0x0, 0x0)
        src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/genutil.go:104 +0x178
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen(0xc0000b4f00, 0x0)
        src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:54 +0x4a5
github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate.k8sFunc(0xc0000b4f00, 0x4024ca0, 0x0, 0x0, 0x0, 0x0)
        src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate/k8s.go:56 +0x156
github.com/spf13/cobra.(*Command).execute(0xc0000b4f00, 0x4024ca0, 0x0, 0x0, 0xc0000b4f00, 0x4024ca0)
        pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x465
github.com/spf13/cobra.(*Command).ExecuteC(0xc0002d7b80, 0x2c9d7a0, 0xc00067a400, 0x0)
        pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
        pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
        src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:85 +0x50f

kdwodc on 22 Aug 2019

@jpkrohling Are you still having this issue or did you update the branch?

I checked out your branch locally and was able to run generate k8s:

$ git remote -v
jpkrohling  https://github.com/jpkrohling/jaeger-operator.git (fetch)
jpkrohling  https://github.com/jpkrohling/jaeger-operator.git (push)
origin  https://github.com/jaegertracing/jaeger-operator.git (fetch)
origin  https://github.com/jaegertracing/jaeger-operator.git (push)

$ git branch -vv
* 607-Update-Operator-SDK-to-0.10.0 a73ec90 [jpkrohling/607-Update-Operator-SDK-to-0.10.0] Updated operator-sdk to 0.10.0
  master                            be83935 [origin/master: behind 161] Add Elasticsearch image to CR and flag (#289)
  test-openapi                      9cdd514 Get rid of finalizer, clean sidecars when no jaeger instance found (#575)

$ operator-sdk version
operator-sdk version: v0.10.0, commit: ff80b17737a6a0aade663e4827e8af3ab5a21170

$ operator-sdk generate k8s
INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [io:[v1alpha1], jaegertracing:[v1], ]
INFO[0024] Code-generation complete.

hasbro17 on 22 Aug 2019

It still happens, but I just found out that it works when I'm outside of the GOPATH. When I'm in the GOPATH, it fails even with GO111MODULE=on.

jpkrohling on 23 Aug 2019

That's weird. It's working fine for me inside $GOPATH:

$ echo $GOPATH
/Users/haseeb/work/go-space

$ pwd
/Users/haseeb/work/go-space/src/github.com/jaegertracing/jaeger-operator

$ GO111MODULE=on operator-sdk generate k8s
INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [io:[v1alpha1], jaegertracing:[v1], ]
INFO[0022] Code-generation complete.

$ GO111MODULE=off operator-sdk generate k8s
FATA[0000] dependency manager "modules" requires working directory to be in $GOPATH/src and GO111MODULE=on, or outside of $GOPATH/src and GO111MODULE="on", "auto", or unset. More info: https://github.com/operator-framework/operator-sdk/blob/master/doc/user-guide.md#go-modules

This behavior inside $GOPATH makes me think this might be a modules related issue to #1853 and #1759.
@jpkrohling Are you using the v0.10.0 release tag binary or did you build from source.
And can you try and build from the master and double check if you still hit this and #1853 ?

hasbro17 on 23 Aug 2019

I'm using the binaries available under the Releases tab in the Operator SDK's repository, and it yields the same SHA as yours. For the record, here's what I get when I run the same commands as you:

$ operator-sdk version
operator-sdk version: v0.10.0, commit: ff80b17737a6a0aade663e4827e8af3ab5a21170

$ git st
On branch 607-Update-Operator-SDK-to-0.10.0
nothing to commit, working tree clean

$ git log -1
commit a73ec9006395578aa301c6827bafaae286b50f13 (HEAD -> 607-Update-Operator-SDK-to-0.10.0, origin/607-Update-Operator-SDK-to-0.10.0)
Author: Juraci Paixão Kröhling <[email protected]>
Date:   Wed Aug 21 18:27:04 2019 +0200

    Updated operator-sdk to 0.10.0

    Signed-off-by: Juraci Paixão Kröhling <[email protected]>

$ echo $GOPATH
/mnt/storage/jpkroehling/Projects

$ pwd
/home/jpkroehling/Projects/src/github.com/jaegertracing/jaeger-operator

$ GO111MODULE=on operator-sdk generate k8s
INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [jaegertracing:[v1], io:[v1alpha1], ] 
F0826 09:22:48.291885   28597 deepcopy.go:885] Hit an unsupported type invalid type for invalid type, from github.com/jaegertracing/jaeger-operator/pkg/apis/io/v1alpha1.ElasticsearchSpec
goroutine 1 [running]:
k8s.io/klog.stacks(0xc000860100, 0xc0002d27e0, 0xbb, 0x10b)
    pkg/mod/k8s.io/[email protected]/klog.go:855 +0xb1
k8s.io/klog.(*loggingT).output(0x33f8420, 0xc000000003, 0xc0004b6540, 0x323b24c, 0xb, 0x375, 0x0)
    pkg/mod/k8s.io/[email protected]/klog.go:806 +0x2d9
k8s.io/klog.(*loggingT).printf(0x33f8420, 0x3, 0x1dc88a2, 0x2a, 0xc000730e88, 0x3, 0x3)
    pkg/mod/k8s.io/[email protected]/klog.go:705 +0x14e
k8s.io/klog.Fatalf(...)
    pkg/mod/k8s.io/[email protected]/klog.go:1256
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).doStruct(0xc0001f5e80, 0xc00026c420, 0xc000838cd0)
    pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:885 +0x7bf
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).generateFor(0xc0001f5e80, 0xc00026c420, 0xc000838cd0)
    pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:695 +0xc5
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).GenerateType(0xc0001f5e80, 0xc000812c60, 0xc00026c420, 0x209c0c0, 0xc00083d120, 0x0, 0x1d9dde4)
    pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:608 +0xe86
k8s.io/gengo/generator.(*Context).executeBody(0xc000812c60, 0x2098d80, 0xc00085e650, 0x2111580, 0xc0001f5e80, 0x60, 0x10)
    pkg/mod/k8s.io/[email protected]/generator/execute.go:304 +0x11d
k8s.io/gengo/generator.(*Context).ExecutePackage(0xc000812a20, 0x0, 0x0, 0x20e9e00, 0xc0001f5a80, 0x0, 0x0)
    pkg/mod/k8s.io/[email protected]/generator/execute.go:265 +0xbf7
k8s.io/gengo/generator.(*Context).ExecutePackages(0xc000812a20, 0x0, 0x0, 0xc0003d5910, 0x1, 0x1, 0x0, 0xc0000593b0)
    pkg/mod/k8s.io/[email protected]/generator/execute.go:51 +0xc5
k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc0001badc0, 0xc000731810, 0x1d6e857, 0x6, 0x1e62798, 0xc0000f77a0, 0x16)
    pkg/mod/k8s.io/[email protected]/args/args.go:194 +0x2d7
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.deepcopyGen(0xc0006449c0, 0xe, 0xc000404bc0, 0x2, 0x2, 0x18, 0xc0007319b0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:94 +0x4d2
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen.func1(0xc0006449c0, 0xe, 0xc0006449c0, 0xe)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:53 +0x50
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.generateWithHeaderFile(0xc000404c00, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/genutil.go:104 +0x178
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen(0xc000384500, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:54 +0x4a5
github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate.k8sFunc(0xc000384500, 0x341bfc0, 0x0, 0x0, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate/k8s.go:56 +0x156
github.com/spf13/cobra.(*Command).execute(0xc000384500, 0x341bfc0, 0x0, 0x0, 0xc000384500, 0x341bfc0)
    pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x465
github.com/spf13/cobra.(*Command).ExecuteC(0xc00039c000, 0x209d6c0, 0xc0002bcd00, 0x0)
    pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
    pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:85 +0x50f

jpkrohling on 26 Aug 2019

I also tried the latest master (abac23c) and it seems that this is fixed there:

$ operator-sdk version
operator-sdk version: v0.10.0-19-gabac23c8, commit: abac23c897b898e6fecfb2c02fe318843ae9347f

$ git st
On branch 607-Update-Operator-SDK-to-0.10.0
nothing to commit, working tree clean

$ git log -1
commit a73ec9006395578aa301c6827bafaae286b50f13 (HEAD -> 607-Update-Operator-SDK-to-0.10.0, origin/607-Update-Operator-SDK-to-0.10.0)
Author: Juraci Paixão Kröhling <[email protected]>
Date:   Wed Aug 21 18:27:04 2019 +0200

    Updated operator-sdk to 0.10.0

    Signed-off-by: Juraci Paixão Kröhling <[email protected]>

$ echo $GOPATH
/mnt/storage/jpkroehling/Projects

$ GO111MODULE=on operator-sdk generate k8s
INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [io:[v1alpha1], jaegertracing:[v1], ] 
INFO[0010] Code-generation complete.

I'm therefore closing this one. Do you have plans for a new release already?

jpkrohling on 26 Aug 2019

So, a fix is coming in the next release? Do we know when that will be?

kdwodc on 26 Aug 2019

@jpkrohling I'm not convinced this is something that's fixed on master. Can you try to rebuild one of the tags from source (e.g. git checkout v0.10.0 && make install), and see if that works.

My hypothesis is that building from source makes the problem go away because you end up using the same GOROOT when you build the SDK and when you run the SDK. When the build env and run env are different seems to be when these (and related) issues pop up.

joelanford on 26 Aug 2019

You are absolutely correct: I can reproduce the problem with the binary at 0.10.0, but not with the self-built 0.10.0.

jpkrohling on 26 Aug 2019

I hit this error when not exporting GOROOT, it worked for me after exporting that.

mandymchu on 26 Aug 2019

👍9

Alright, I _think_ I finally have this figured out.

When the operator-sdk binary is built from source, one of the transitive dependencies that gets pulled in by controller-tools for code generation is kubernetes/gengo.

kubernetes/gengo's AddDirRecursive function searches a set of paths, including GOROOT, when parsing Go code. It determines GOROOT using go/build.Default.GOROOT.

So when using the operator-sdk, it is necessary to either (a) set GOROOT in your environment, or (b) use the same GOROOT that was used to build the operator-sdk binary.

Obviously (b) is undesirable because you have to know what GOROOT was used and then force your environment to match it, so (a) setting GOROOT explicitly seems like the best workaround.

/cc @estroz @hasbro17 @jpkrohling @mandymchu @kdwodc

TL;DR: If you run into strange problems with code generation, try setting GOROOT explicitly

joelanford on 27 Aug 2019

👍14 🎉10

Wonderful, thanks for digging into this! I confirm that setting GOROOT does make it work for me (Fedora 30, latest updates as of today):

$ export GOROOT=/usr/lib/golang
$ GO111MODULE=on operator-sdk generate k8s
INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [io:[v1alpha1], jaegertracing:[v1], ] 
INFO[0009] Code-generation complete.                    

$ unset GOROOT
$ GO111MODULE=on operator-sdk generate k8s
INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [io:[v1alpha1], jaegertracing:[v1], ] 
F0827 09:05:57.950321   12347 deepcopy.go:885] Hit an unsupported type invalid type for invalid type, from github.com/jaegertracing/jaeger-operator/pkg/apis/io/v1alpha1.ElasticsearchSpec
goroutine 1 [running]:
k8s.io/klog.stacks(0xc000814700, 0xc0002987e0, 0xbb, 0x10b)
    pkg/mod/k8s.io/[email protected]/klog.go:855 +0xb1
k8s.io/klog.(*loggingT).output(0x33f8420, 0xc000000003, 0xc0002b1260, 0x323b24c, 0xb, 0x375, 0x0)
    pkg/mod/k8s.io/[email protected]/klog.go:806 +0x2d9
k8s.io/klog.(*loggingT).printf(0x33f8420, 0x3, 0x1dc88a2, 0x2a, 0xc0006cae88, 0x3, 0x3)
    pkg/mod/k8s.io/[email protected]/klog.go:705 +0x14e
k8s.io/klog.Fatalf(...)
    pkg/mod/k8s.io/[email protected]/klog.go:1256
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).doStruct(0xc000562200, 0xc000464f20, 0xc0007e3c20)
    pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:885 +0x7bf
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).generateFor(0xc000562200, 0xc000464f20, 0xc0007e3c20)
    pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:695 +0xc5
k8s.io/gengo/examples/deepcopy-gen/generators.(*genDeepCopy).GenerateType(0xc000562200, 0xc0004f6f00, 0xc000464f20, 0x209c0c0, 0xc0007d1c40, 0x0, 0x1d9dde4)
    pkg/mod/k8s.io/[email protected]/examples/deepcopy-gen/generators/deepcopy.go:608 +0xe86
k8s.io/gengo/generator.(*Context).executeBody(0xc0004f6f00, 0x2098d80, 0xc000806720, 0x2111580, 0xc000562200, 0x60, 0x10)
    pkg/mod/k8s.io/[email protected]/generator/execute.go:304 +0x11d
k8s.io/gengo/generator.(*Context).ExecutePackage(0xc0004f6cc0, 0x0, 0x0, 0x20e9e00, 0xc0003a7e80, 0x0, 0x0)
    pkg/mod/k8s.io/[email protected]/generator/execute.go:265 +0xbf7
k8s.io/gengo/generator.(*Context).ExecutePackages(0xc0004f6cc0, 0x0, 0x0, 0xc00044a170, 0x1, 0x1, 0x0, 0xc0005868d0)
    pkg/mod/k8s.io/[email protected]/generator/execute.go:51 +0xc5
k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc00049b360, 0xc0006cb810, 0x1d6e857, 0x6, 0x1e62798, 0xc00039a980, 0x16)
    pkg/mod/k8s.io/[email protected]/args/args.go:194 +0x2d7
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.deepcopyGen(0xc00057ac90, 0xe, 0xc000568fc0, 0x2, 0x2, 0x18, 0xc0006cb9b0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:94 +0x4d2
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen.func1(0xc00057ac90, 0xe, 0xc00057ac90, 0xe)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:53 +0x50
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.generateWithHeaderFile(0xc000569000, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/genutil.go:104 +0x178
github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil.K8sCodegen(0xc00036c000, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/internal/genutil/k8s.go:54 +0x4a5
github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate.k8sFunc(0xc00036c000, 0x341bfc0, 0x0, 0x0, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/generate/k8s.go:56 +0x156
github.com/spf13/cobra.(*Command).execute(0xc00036c000, 0x341bfc0, 0x0, 0x0, 0xc00036c000, 0x341bfc0)
    pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x465
github.com/spf13/cobra.(*Command).ExecuteC(0xc000368a00, 0x209d6c0, 0xc0003ac900, 0x0)
    pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
    pkg/mod/github.com/spf13/[email protected]/command.go:800
main.main()
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:85 +0x50f

jpkrohling on 27 Aug 2019

👍4

TL;DR: If you run into strange problems with code generation with operator-sdk<=v0.10.0, try setting GOROOT explicitly

If you always reproduce this issue, although go env GOROOT display the right path, you should export GOROOT= again. Then it can works fine.

$ go env GOROOT
/usr/local/Cellar/go/1.12.9/libexec
$ operator-sdk add api --api-version=app.example.com/v1alpha1 --kind=AppService
INFO[0000] Generating api version app.example.com/v1alpha1 for kind AppService.
INFO[0000] Created pkg/apis/app/group.go
INFO[0000] Created pkg/apis/app/v1alpha1/appservice1_types.go
INFO[0000] Created pkg/apis/addtoscheme_app1_v1alpha1.go
INFO[0000] Created pkg/apis/app/v1alpha1/register.go
INFO[0000] Created pkg/apis/app/v1alpha1/doc.go
INFO[0000] Created deploy/crds/app1_v1alpha1_appservice1_cr.yaml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x18e8815]

goroutine 1 [running]:
k8s.io/gengo/parser.(*Builder).AddDirRecursive(0xc000560d70, 0xc0002af830, 0x2c, 0x2, 0x2)
    pkg/mod/k8s.io/[email protected]/parser/parse.go:229 +0xb5
$ export GOROOT='/usr/local/Cellar/go/1.12.9/libexec'
$ operator-sdk add api --api-version=app1.example.com/v1alpha1 --kind=AppService1
INFO[0000] Generating api version app1.example.com/v1alpha1 for kind AppService1.
INFO[0000] Created pkg/apis/app1/group.go
INFO[0001] Created pkg/apis/app1/v1alpha1/appservice_types.go
INFO[0001] Created pkg/apis/addtoscheme_app_v1alpha1.go
INFO[0001] Created pkg/apis/app1/v1alpha1/register.go
INFO[0001] Created pkg/apis/app1/v1alpha1/doc.go
INFO[0001] Created deploy/crds/app_v1alpha1_appservice_cr.yaml
INFO[0007] Created deploy/crds/app_v1alpha1_appservice_crd.yaml
INFO[0007] Running deepcopy code-generation for Custom Resource group versions: [app1:[v1alpha1], ]
INFO[0013] Code-generation complete.
INFO[0013] Running OpenAPI code-generation for Custom Resource group versions: [app1:[v1alpha1], ]
INFO[0026] Created deploy/crds/app_v1alpha1_appservice_crd.yaml
INFO[0026] Code-generation complete.
INFO[0026] API generation complete.

Thanks @joelanford gave the detailed answers.

jiwq on 27 Aug 2019

👍4

Soo I am running 0.11 + Modules and am still seeing this error.

abergmeier on 12 Oct 2019

👍3

I have tried with 0.11.0 release of operator-sdk cli from homebrew, download using curl, built from source (needed to downgrade go version to 1.12.12 from 1.13.1 for building from source to work successfully due to https://github.com/operator-framework/operator-sdk/issues/2055 ) .
have setup GOROOT explicitly.

$ env  | grep GO
GO111MODULE=on
GOROOT=/usr/local/Cellar/[email protected]/1.12.12/libexec
GOPATH=/Users/gupta/.go

have put my operator in $GOPATH/src/github.com/himanshug/my-operator

but still getting same error.

himanshug on 21 Oct 2019

@himanshug There's a couple of things going on possibly.

I just installed SDK using homebrew, and it installed go1.13. operator-sdk version reports it was compiled with go1.13.1. However, SDK does not yet totally support go1.13. So the homebrew install may be broken at the moment.
Given 1), I'm wondering if your GOROOT is set correctly. Can you post the output from the following commands:
$ which -a go $ go version $ go env $ env | grep GO $ ls -lH $GOROOT $ which -a operator-sdk $ operator-sdk version

I never went back and checked that v0.11.0 resolved the issue with setting GOROOT. It appears that, via transitive dependencies, controller-tools does actually still depend on k8s.io/gengo, which is where the root cause of this issue lies.

joelanford on 21 Oct 2019

@joelanford thanks for looking into it.

~$ which -a go
/usr/local/bin/go
~$ ls -l /usr/local/bin/go
lrwxr-xr-x  1 hgupta  admin  32 Oct 21 12:50 /usr/local/bin/go -> ../Cellar/[email protected]/1.12.12/bin/go
~$ go version
go version go1.12.12 darwin/amd64
~$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/hgupta/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/hgupta/.go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/[email protected]/1.12.12/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/[email protected]/1.12.12/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/n6/c43tqbkx2011jm66dz0yvjzr0000gn/T/go-build393580094=/tmp/go-build -gno-record-gcc-switches -fno-common"
~$ env | grep GO
GO111MODULE=on
GOROOT=/usr/local/Cellar/[email protected]/1.12.12/libexec
GOPATH=/Users/hgupta/.go
~$ ls -lH $GOROOT
total 208
-rw-r--r--    1 hgupta  staff   1339 Oct 17 14:55 CONTRIBUTING.md
-rw-r--r--    1 hgupta  staff  78132 Oct 17 14:55 CONTRIBUTORS
-rw-r--r--    1 hgupta  staff   1303 Oct 17 14:55 PATENTS
-rw-r--r--    1 hgupta  staff      9 Oct 17 14:55 VERSION
drwxr-xr-x   18 hgupta  staff    576 Oct 17 14:55 api
drwxr-xr-x    5 hgupta  staff    160 Oct 17 14:55 bin
drwxr-xr-x   49 hgupta  staff   1568 Oct 17 14:55 doc
-rw-r--r--    1 hgupta  staff   5686 Oct 17 14:55 favicon.ico
drwxr-xr-x    3 hgupta  staff     96 Oct 17 14:55 lib
drwxr-xr-x   15 hgupta  staff    480 Oct 17 14:55 misc
drwxr-xr-x    7 hgupta  staff    224 Oct 17 14:55 pkg
-rw-r--r--    1 hgupta  staff     26 Oct 17 14:55 robots.txt
drwxr-xr-x   68 hgupta  staff   2176 Oct 17 14:55 src
drwxr-xr-x  309 hgupta  staff   9888 Oct 17 14:55 test
~$ which -a operator-sdk
/Users/hgupta/.go/bin/operator-sdk
/Users/hgupta/.go/bin/operator-sdk
/usr/local/bin/operator-sdk
~$ operator-sdk version
operator-sdk version: "v0.11.0", commit: "39c65c36159a9c249e5f3c178205cc6e86c16f8d", go version: "go1.13.1 darwin/amd64"
~$
~$ $GOPATH/bin/operator-sdk version
operator-sdk version: "v0.11.0", commit: "39c65c36159a9c249e5f3c178205cc6e86c16f8d", go version: "go1.12.12 darwin/amd64"

~/.go/src/github.com/himanshug/my-operator[master]$ GO111MODULE=on $GOPATH/bin/operator-sdk generate k8s
INFO[0000] Running deepcopy code-generation for Custom Resource group versions: [xxobfuscatedxx:[v1alpha1], ]
F1021 13:49:48.680272   92981 deepcopy.go:813] Hit an unsupported type invalid type for []invalid type
goroutine 1 [running]:
k8s.io/klog.stacks(0xc008a6e000, 0xc0021a5140, 0x67, 0xb7)
    pkg/mod/k8s.io/[email protected]/klog.go:855 +0xb1

himanshug on 21 Oct 2019

Thanks!

F1021 13:49:48.680272 92981 deepcopy.go:813] Hit an unsupported type invalid type for []invalid type

This error might be legitimate (though obviously not a great error message). Do you have any types in your objects that are interfaces (or that contain interfaces)?

To confirm, it may be worth generating a project from scratch to see if you get the same error on a simple type scaffolded by the SDK:

$ operator-sdk new test-operator --repo=example.com/test/test-operator
$ cd test-operator
$ operator-sdk add api --api-version=example.com/v1alpha1 --kind=Test

Another potential problem I see is that it looks like you have a couple of SDK binaries in different places, built with different versions of Go. I don't know for sure, but I wouldn't be surprised if it was necessary to align the go versions of GOROOT and the SDK binary.

~$ operator-sdk version
operator-sdk version: "v0.11.0", commit: "39c65c36159a9c249e5f3c178205cc6e86c16f8d", go version: "go1.13.1 darwin/amd64"

GOROOT=/usr/local/Cellar/[email protected]/1.12.12/libexec

It might be worth trying to uninstall operator-sdk and go using homebrew, deleting $GOPATH/bin/operator-sdk, cleaning out any other Go installations, and trying again from scratch.

As you can probably tell, this is a notoriously nasty bug and it can get even more difficult to solve the further down the rabbit hole you go, changing your environment in different ways to try to get things working.

The most foolproof way that I've found to solve this is to install Go from https://golang.org/dl/ and compile the SDK from source by cloning it, checking out the tag (e.g. v0.11.0), and running make install. That way the SDK binary is always built from your local Go installation and has the correct GOROOT built in.

joelanford on 21 Oct 2019

@joelanford the foolproof method isn't foolproof :)
I spent some time today removing operator-sdk and go deployed by Homebrew , downloaded go 1.12.12 from https://golang.org/dl/ and manually built operator-sdk from source from v0.11.0 tag. But, still getting same error.

that said, I do have types which are interfaces or contain interfaces , so I tried the test operator

~/.go/src/github.com/test$ operator-sdk new test-operator --repo=github.com/test/test-operator
...
~/.go/src/github.com/test$ cd test-operator/
~/.go/src/github.com/test/test-operator$ operator-sdk add api --api-version=example.com/v1alpha1 --kind=Test
~/.go/src/github.com/test/test-operator$ operator-sdk generate k8s

above test worked, so it must be something in my changes to types file which has a whole lot of things. I will do a binary search by removing stuff one by one to see what causes this. thanks.

~$ which -a go
/Users/hgupta/installations/go1.12.12/bin/go
~$ go version
go version go1.12.12 darwin/amd64
~$ go version
go version go1.12.12 darwin/amd64
~$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/hgupta/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/hgupta/.go"
GOPROXY=""
GORACE=""
GOROOT="/Users/hgupta/installations/go1.12.12"
GOTMPDIR=""
GOTOOLDIR="/Users/hgupta/installations/go1.12.12/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/n6/c43tqbkx2011jm66dz0yvjzr0000gn/T/go-build580793539=/tmp/go-build -gno-record-gcc-switches -fno-common"
~$ env | grep GO
GO111MODULE=on
GOROOT=/Users/hgupta/installations/go1.12.12
GOPATH=/Users/hgupta/.go
~$ ls -lH $GOROOT
total 336
-rw-r--r--@   1 hgupta  staff  55358 Oct 17 14:54 AUTHORS
-rw-r--r--@   1 hgupta  staff   1339 Oct 17 14:54 CONTRIBUTING.md
-rw-r--r--@   1 hgupta  staff  78132 Oct 17 14:54 CONTRIBUTORS
-rw-r--r--@   1 hgupta  staff   1479 Oct 17 14:54 LICENSE
-rw-r--r--@   1 hgupta  staff   1303 Oct 17 14:54 PATENTS
-rw-r--r--@   1 hgupta  staff   1607 Oct 17 14:54 README.md
-rw-r--r--@   1 hgupta  staff      9 Oct 17 14:55 VERSION
drwxr-xr-x@  18 hgupta  staff    576 Oct 17 14:54 api
drwxr-xr-x@   5 hgupta  staff    160 Oct 17 14:58 bin
drwxr-xr-x@  49 hgupta  staff   1568 Oct 17 14:54 doc
-rw-r--r--@   1 hgupta  staff   5686 Oct 17 14:54 favicon.ico
drwxr-xr-x@   3 hgupta  staff     96 Oct 17 14:54 lib
drwxr-xr-x@  15 hgupta  staff    480 Oct 17 14:54 misc
drwxr-xr-x@   6 hgupta  staff    192 Oct 17 14:58 pkg
-rw-r--r--@   1 hgupta  staff     26 Oct 17 14:54 robots.txt
drwxr-xr-x@  68 hgupta  staff   2176 Oct 17 14:54 src
drwxr-xr-x@ 309 hgupta  staff   9888 Oct 17 14:55 test
~$ which -a operator-sdk
/Users/hgupta/.go/bin/operator-sdk
~$ operator-sdk version
operator-sdk version: "v0.11.0", commit: "39c65c36159a9c249e5f3c178205cc6e86c16f8d", go version: "go1.12.12 darwin/amd64"

himanshug on 22 Oct 2019

that said, I do have types which are interfaces or contain interfaces

Yeah, that's definitely a problem and not supported. As I understand it, the code generator needs to know the concrete types to be able to generate the CRD schemas, and the kubernetes client needs to know what concrete types to use when decoding the API objects (e.g. when doing a GET from the API server).

joelanford on 22 Oct 2019

@joelanford thanks for the help. I did get everything working after various updates to my type file. that same error message happens from so many different reasons, one needs to figure out what specifically is the cause in their setup. I hope the discussion here would help future operator writers.

himanshug on 24 Oct 2019

So to summarise

export GOROOT=$(go env GOROOT)

does the trick

ytsarev on 27 Dec 2019

🎉3 ❤1 👍1

So to summarise
export GOROOT=$(go env GOROOT)
does the trick

~~Not for me~~

Workaround is now working.

abergmeier on 28 Dec 2019

@abergmeier can you open another issue with your specific environment specs? Thanks.

estroz on 28 Dec 2019

This is still an ongoing issue. Someone should post this on the operator-sdk install guide, as I spent atleast 30 minutes trying to figure out what was causing it. Installed go with snap package manager

Nik-Novak on 24 Feb 2020

👍2

HI @Nik-Novak,

This error is because of the local env which is not configured properly. I am not sure where and how to fit it in the docs. However, please feel free to raise PR with your idea/suggestion for we are able to check it. Really thank you.

camilamacedo86 on 24 Feb 2020

HI @Nik-Novak,

This error is because of the local env which is not configured properly. I am not sure where and how to fit it in the docs. However, please feel free to raise PR with your idea/suggestion for we are able to check it. Really thank you.

I question why setting GOROOT should be be seen as _local env configured properly_. IMO it should fallback to go env GOROOT if it really needs to.

abergmeier on 24 Feb 2020

HI @abergmeier,

The GOROOT environment variable point to the directory where the go which should be used is installed. In this way, I understand that usually, you don’t need to set $GOROOT variable. However, you may face issues in some scenarios. For example; if you have more than one go version installed locally, or if you installed it in different locally from where was recommend or if you use homebrew to install. Just to illustrate the homebrew scenario see, for example, https://coderwall.com/p/c00m1g/set-goroot-variable-with-homebre

I hope that it can help you with.

camilamacedo86 on 24 Feb 2020

Still seeing this issue on operator-sdk v0.16.0, go 1.13.8, Fedora 30.