Operator-sdk: Kubebuilder is not listed in prerequisites

[deepak1725]

Bug Report

Kubebuilder is Tightly coupled in requirements, which is fine but...

1. kubebuilder is not listed in Prerequisites
2. make docker-build ... expects kubebuilder in usr/local/kubebuilder only otherwise it raises an error, which I don't think is a good idea. User may have kubebuilder anywhere in PATH.

What did you do?
make docker-build $IMG

What did you expect to see?
resolves kubebuilder from $PATH not from any Specific Location.

What did you see instead? Under which circumstances?

Failure [0.002 seconds]
[BeforeSuite] BeforeSuite 
/home/deepak/projects/go/src/operator/newoperator3/final/crda-operator/controllers/suite_test.go:52

  Unexpected error:
      <*fmt.wrapError | 0xc00046c2a0>: {
          msg: "failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory",
          err: {
              Op: "fork/exec",
              Path: "/usr/local/kubebuilder/bin/etcd",
              Err: 0x2,
          },
      }
      failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory
  occurred

  /home/deepak/projects/go/src/operator/newoperator3/final/crda-operator/controllers/suite_test.go:62
------------------------------


Ran 0 of 0 Specs in 0.002 seconds
FAIL! -- 0 Passed | 0 Failed | 0 Pending | 0 Skipped
--- FAIL: TestAPIs (0.00s)
FAIL
coverage: 0.0% of statements
FAIL    github.com/deepak1725/crda-operator/controllers 0.010s
FAIL
make: *** [Makefile:30: test] Error 1

Environment

• operator-sdk version: 0.19

• go version: 0.14

• Kubernetes version information: 1.18

• Kubernetes cluster kind: minikube

• Are you writing your operator in ansible, helm, or go? go

Possible Solution

Additional context
Add any other context about the problem here.

[camilamacedo86]

Hi @deepak1725,

I am unable to reproduce this issue. Kubebuilder should not be a pre-requirement and shows that your project is asking its binary to setup the Env Test.

Could you please let us know the steps performed to generate this issue? Could you please let me know if you have kubebuilder installed when you create the project?

[deepak1725]

@camilamacedo86
My System didn't had kubebuilder

Steps:

• I tried make docker-build $IMG
• Got Error: As Specified above.
• then I installed it in the separate Path than specified in Official Kubebuilder docs.
  ie. I installed in /usr/local/bin/kubebuilder instead of /usr/local/kubebuilder/
• Exported Kubebuilder and it is recognized from every end of PC.
• I tried again make docker-build $IMG, but the error was still not resolved.
• Finally I have moved binaries from my custom path to officially specified Kubebuilder path, which finally Resolves Error.

• Conclusion acc. to me:
  SDK has Hard-Coded Kubebuilder path which relies solely on Official specified Path.

[camilamacedo86]

Hi @deepak1725,

Really thank you for raise that. As you know, SDK is in a process to be integrated with KB which means that its project layouts will be aligned. More info : Integrating Kubebuilder and Operator SDK. But, IMHO kubebuilder cannot be set as a pre-requirement because the kubebuilder version used by SDK as dependency can be diff of the version installed.

The problem faced by you is because Env Test requires the apiserver, kubectl and etcd binaries to run and these are shipped with kubebuilder binary. You can customize its set up to use them from another location as well. To understand it further as to see how you can change its conjugation see: https://github.com/kubernetes-sigs/kubebuilder/issues/686#issuecomment-647376924.

We will discuss this scenario in the bug triage meeting on Monday. Feel free to attend if you wish.

[camilamacedo86]

PR to fix it for v3 in Kubebuilder; https://github.com/kubernetes-sigs/kubebuilder/pull/1600

[camilamacedo86]

Hi @deepak1725,

See that we added a script to upstream to do this setup and we have a pr against master https://github.com/operator-framework/operator-sdk/pull/3510 and against 0.19.x https://github.com/operator-framework/operator-sdk/pull/3517 to update the docs.

[camilamacedo86]

All done. We can improve the docs in the future as well. Really tks for raise that.

[deepak1725]

@camilamacedo86 are we going to cherry-pick this in 0.19 later..?

[camilamacedo86]

It is applied to the 0.19 already. See; https://v0-19-x.sdk.operatorframework.io/docs/golang/references/env-test-setup/ and the PR: https://github.com/operator-framework/operator-sdk/pull/3517

[camilamacedo86]

We are re-open this one. After the kubebuilder/controller-runtime meeting was decided that we should not re-build the bins and instead of that get them from https://storage.googleapis.com/kubebuilder-tools. @joelanford has been working in an script against controller-runtime: https://github.com/kubernetes-sigs/controller-runtime/pull/1092

And then, we imagine as a final solution something such as:

• setup_envtest.sh:

ENVTEST_ASSETS_DIR=$(pwd)/test/assets
fetch_envtest_assets() {
    OS=$(uname -s | tr '[:upper:]' '[:lower:]')
    ARCH=$(uname -m | sed 's/x86_64/amd64/')
    mkdir -p $ENVTEST_ASSETS_DIR
    curl -fsL "https://storage.googleapis.com/kubebuilder-tools/kubebuilder-tools-1.16.4-${OS}-${ARCH}.tar.gz" | tar zx --strip-components=1 -C $ENVTEST_ASSETS_DIR
}
setup_envtest_env() {
    export TEST_ASSET_KUBE_APISERVER=$ENVTEST_ASSETS_DIR/bin/kube-apiserver
    export TEST_ASSET_ETCD=$ENVTEST_ASSETS_DIR/bin/etcd
    export TEST_ASSET_KUBECTL=$ENVTEST_ASSETS_DIR/bin/kubectl
}

Then in a makefile (or wherever)
source setup_envtest.sh; fetch_envtest_assets; setup_envtest_env

Also, in the PRs https://github.com/operator-framework/operator-sdk/pull/3551 and https://github.com/operator-framework/operator-sdk/pull/3543 we are marking the Env Test doc as TBD as the step to setup it until we have a better solution.

[estroz]

This shouldn't block v1.0 since the steps to get envtest tools are encoded in a script and documented. Moving this to v1.1.0.

[jimdickinson]

I just hit this on 1.0, and I don't see this explained in the docs. Is this the right page? https://sdk.operatorframework.io/docs/building-operators/golang/references/envtest-setup/

Might want to weave a link to that into the tutorial, although it's quite possible I missed it.

[camilamacedo86]

Hi @jimdickinson,

In the quickstart, you will see that a step was added to solve this problem for now. See: https://sdk.operatorframework.io/docs/building-operators/golang/quickstart/#configuring-your-test-environment

[zonnie]

@camilamacedo86 - the link you sent leads to the Quickstart page but the section is missing.
I tried to look for the workaround anyway but was unable to find it...any quick way to avoid this issue ?

[camilamacedo86]

The above link shows the latest version (1.1). For this version, the setup is no longer required. See that your project is built with the target as: https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/memcached-operator/Makefile#L28-L33.

However, if you see any problem could you please raise a new issue.