Operator-sdk: sigsegv if GOROOT is not set

[rhuss]

Bug Report

When running an

operator-sdk add api --api-version=hello.k8spatterns.io/v1alpha1 --kind=Hello

in a module enabled project below $GOPATH, and $GOROOT is not set, then the following errors occurs:

operator-sdk add api --api-version=hello.k8spatterns.io/v1alpha1 --kind=Hello
INFO[0000] Generating api version hello.k8spatterns.io/v1alpha1 for kind Hello.
INFO[0001] Created pkg/apis/hello/v1alpha1/hello_types.go
INFO[0001] Created pkg/apis/addtoscheme_hello_v1alpha1.go
INFO[0001] Created pkg/apis/hello/v1alpha1/register.go
INFO[0001] Created pkg/apis/hello/v1alpha1/doc.go
INFO[0001] Created deploy/crds/hello_v1alpha1_hello_cr.yaml
W0611 10:07:38.859903   39295 parse.go:224] Ignoring directory github.com/k8spatterns/hello-operator/pkg/apis: unable to import "github.com/k8spatterns/hello-operator/pkg/apis": go/build: importGo github.com/k8spatterns/hello-operator/pkg/apis: exit status 2
go: cannot find GOROOT directory: /usr/local/go

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1760595]

goroutine 1 [running]:
github.com/operator-framework/operator-sdk/vendor/k8s.io/gengo/parser.(*Builder).AddDirRecursive(0xc000962230, 0xc0006b71d0, 0x2e, 0x2, 0x2)
    src/github.com/operator-framework/operator-sdk/vendor/k8s.io/gengo/parser/parse.go:229 +0xb5
github.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-tools/pkg/crd/generator.(*Generator).Do(0xc00092d760, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-tools/pkg/crd/generator/generator.go:126 +0x3ac
github.com/operator-framework/operator-sdk/internal/pkg/scaffold.(*CRD).CustomRender(0xc0001b11e0, 0x2cca880, 0x41d87c0, 0x4f814e8, 0xc0001b11e0, 0x1)
    src/github.com/operator-framework/operator-sdk/internal/pkg/scaffold/crd.go:122 +0x868
github.com/operator-framework/operator-sdk/internal/pkg/scaffold.(*Scaffold).doRender(0xc00092dcf8, 0xc0006b7110, 0x29, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0003e4db1, 0x25, ...)
    src/github.com/operator-framework/operator-sdk/internal/pkg/scaffold/scaffold.go:228 +0x16c
github.com/operator-framework/operator-sdk/internal/pkg/scaffold.(*Scaffold).doFile(0xc00092dcf8, 0x2c4f2c0, 0xc0001b11e0, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/internal/pkg/scaffold/scaffold.go:198 +0x230
github.com/operator-framework/operator-sdk/internal/pkg/scaffold.(*Scaffold).Execute(0xc00092dcf8, 0xc000925c68, 0xc00092dc98, 0x6, 0x6, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/internal/pkg/scaffold/scaffold.go:163 +0x103
github.com/operator-framework/operator-sdk/cmd/operator-sdk/add.apiRun(0xc0003b8000, 0xc00000d020, 0x0, 0x2, 0x0, 0x0)
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/add/api.go:107 +0x4d2
github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).execute(0xc0003b8000, 0xc00000cfc0, 0x2, 0x2, 0xc0003b8000, 0xc00000cfc0)
    src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:762 +0x465
github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000361b80, 0x2c53720, 0xc0000f1a00, 0x0)
    src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:852 +0x2ec
github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).Execute(...)
    src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:800
main.main()
    src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:80 +0x4ce

I would expect either to work out of the box (i'm using modules anyway) or a clear simple pre-run check the GOROOT needs to be set.

(ideally I would prefer to run outside of $GOPATH but this is prevented by #1457 )

Environment

• operator-sdk version:

operator-sdk version: v0.8.1, commit: 33b3bfe10176f8647f5354516fff29dea42b6342

env | grep GO
GOPATH=/Users/roland/Development/go/workspace
GO111MODULE=on

• go version:

go version go1.12.4 darwin/amd64

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-19T22:12:47Z", GoVersion:"go1.12.4", Compiler:"gc", Platform:"darwin/amd64"}

• Kubernetes cluster kind: ----

• Are you writing your operator in ansible, helm, or go?

go

Possible Solution

Pre-check with clear error message indicating what is wrong or missing.

[joelanford]

This appears to be a duplicate of #1480.

@rhuss Is your go installation in /usr/local/go? If not, you probably need to set GOROOT (or move your installation to /usr/local/go).

On my system, I have go installed in the default location (/usr/local/go), and I'm able to run operator-sdk add api successfully without setting GOROOT.

When I move my go installation and retry, I get the error you included in your description and I can fix it by setting GOROOT.

This issue is also moot due to https://github.com/operator-framework/operator-sdk/pull/1470, which has been merged into master and will be included in the next release. That PR updates the add api command to no longer require go builds for the code generators.

[rhuss]

Yes, I could fix it by setting GOROOT. I just think that its not a nice UX to get a SIGSEGV when not setting GOROOT in my case. If there would have been an error message like 'Please set your GOROOT to your go installation' that would be totally fine.

[joelanford]

@rhuss There is a line in your log with go: cannot find GOROOT directory: /usr/local/go, but I agree that it isn't nice to panic.

Unfortunately, it looks like the k8s.io/gengo library is the source of this bug, and I would hesitate to add extra checks in the operator-sdk to guard against misconfigured Go environments.

I submitted an issue upstream to see if this can be improved: https://github.com/kubernetes/gengo/issues/146

[yashbhutwala]

I agree with @rhuss here. For example, for MacOS users, if a user is using homebrew to install go, operator-sdk, etc., these binaries will reside in /usr/local/bin/go, etc. by default. If this is not fixed here, at least documentation clarifying this to add GOROOT should be added.

[joelanford]

@rhuss and @yashbhutwala could you try reproducing your issue with a build from master? I think this should be fixed now because we no longer try to use the local Go environment to build the generator binaries.

[joelanford]

To satisfy my own curiosity, I did a little more digging about how GOROOT works for distributions from package managers, and it appears that packagers should be building the correct GOROOT into the go binary to match where Go is installed using the GOROOT_FINAL environment variable when building from source.

See:

• https://dave.cheney.net/2013/06/14/you-dont-need-to-set-goroot-really
• https://github.com/golang/go/blob/4c2ffd26ec3fe2fb9552b5a38c2b731e86a85abf/src/make.bash#L10-L11

I confirmed that this is the case for the binaries produced from brew-installed Go:

https://github.com/Homebrew/homebrew-core/blob/255b8e25d596eeaf7aa3f1af65e1dfefc9932a37/Formula/go.rb#L44-L48

$ brew install go
$ which go
/usr/local/bin/go
$ go env | grep GOROOT
GOROOT="/usr/local/Cellar/go/1.12.5/libexec"

@yashbhutwala Is this what you're go env output looks like?
@rhuss Did you also install Go using homebrew?

TL;DR if your Go installation is from homebrew (or any other OS package manager), it is unlikely that you need to set GOROOT since the correct GOROOT should be automatically built into the Go binary.

If you downloaded the zip or tarball binary distribution from https://golang.org/dl/ and didn't install it at /usr/local/go, then explicitly setting GOROOT is required. This is documented in the official Go installation instructions.

I'm going to go ahead and close this issue since I believe #1470 refactors the SDK to remove the requirement for a working Go environment for operator-sdk add api, and because it seems like an environment issue anyway.

I'm still curious how your environments look though, to make sure I'm not missing something!

[yashbhutwala]

@joelanford this is my env config:

$ which -a go
/usr/local/bin/go
$ go version
go version go1.12.6 darwin/amd64
$ go env | grep GOROOT
GOROOT="/usr/local/Cellar/go/1.12.6/libexec"
$ operator-sdk version
operator-sdk version: v0.8.1, commit: 33b3bfe10176f8647f5354516fff29dea42b6342

But upon execution of operator-sdk add api --api-version=cache.example.com/v1alpha1 --kind=Memcached, in the error I see this:

W0614 14:27:43.883919   52596 parse.go:224] Ignoring directory github.com/yashbhutwala/memcached-operator/pkg/apis: unable to import "github.com/yashbhutwala/memcached-operator/pkg/apis": go/build: importGo github.com/yashbhutwala/memcached-operator/pkg/apis: exit status 2
go: cannot find GOROOT directory: /usr/local/Cellar/go/1.12.5/libexec

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1760595]

So somehow, it's caching /usr/local/Cellar/go/1.12.5/libexec vs using /usr/local/Cellar/go/1.12.6/libexec

Also I'm running into this error, trying to compile on master:

go: launchpad.net/[email protected]: bzr branch --use-existing-dir https://launchpad.net/~niemeyer/gocheck/trunk . in /Users/bhutwala/gocode/pkg/mod/cache/vcs/f46ce2ae80d31f9b0a29099baa203e3b6d269dace4e5357a2cf74bd109e13339: exec: "bzr": executable file not found in $PATH
go: error loading module requirements
make: *** [install] Error 1

[joelanford]

So somehow, it's caching /usr/local/Cellar/go/1.12.5/libexec vs using /usr/local/Cellar/go/1.12.6/libexec

Interesting! I'll see if I can reproduce that somehow.

Also I'm running into this error, trying to compile on master:

For your SDK compile error, make sure to set GO111MODULE=off. We haven't switched over to go modules quite yet.

[yashbhutwala]

Interesting! I'll see if I can reproduce that somehow.

Yeah, it's weird and it may just be my system, but I doubt it since I don't think I have configured anything funky on there 😄

For your SDK compile error, make sure to set GO111MODULE=off. We haven't switched over to go modules quite yet.

Gotcha, cool, so it works on master.

[joelanford]

I'm not able to reproduce on my Mac. I don't typically use the brew go package, so I cleaned out my existing go installation and environment, ran brew install go, verified the which go and the go env GOROOT and then successfully ran operator-sdk add api successfully.

Gotcha, cool, so it works on master.

👍

[rhuss]

I meant that I had no GOROOT set at all. I would expect an error message pointing me to set my GOROOT env var, but not a SIGSEGV. That's all.

[mogren]

Thanks a lot for the replies. For me on OS X, setting GOROOT worked.

export GOROOT=/usr/local/Cellar/go/1.12.6/libexec

[ChristianKreuzberger]

I've installed Go using snap on Ubuntu 18.04, for that I checked

go env | grep GOROOT

and then I set

export GOROOT="/snap/go/4098"

I'm not very familiar with how Go works, but this just seems to be a sub-optimal experience for everyone involved.

I hope that this can be addressed with a better error message in the future.

[wreed4]

@ChristianKreuzberger this is my experience as well. Feels entirely unnecessary if go itself knows what GOROOT to use, why should I have to duplicate it in my env.

[joelanford]

@ChristianKreuzberger @wreed4 @mogren @rhuss @yashbhutwala Just to satisfy everyone's curiousity, I think I've gotten to the bottom of this. See https://github.com/operator-framework/operator-sdk/issues/1854#issuecomment-525132306

[yashbhutwala]

Thanks @joelanford for tracking this down!