Kubebuilder: CI is broken

[camilamacedo86]

Following the error, when runs e2e tests for v2.

STEP: building image
running: make docker-build IMG=e2e-test/controller-manager:krii
STEP: clean up created API objects during test process
running: kustomize build config/default
running: kubectl delete -f -
warning: error when running kubectl delete -f -: kubectl delete -f - failed with error: Error from server (NotFound): error when deleting "STDIN": namespaces "e2e-krii-system" not found
Error from server (NotFound): error when deleting "STDIN": customresourcedefinitions.apiextensions.k8s.io "fookriis.barkrii.example.comkrii" not found
Error from server (NotFound): error when deleting "STDIN": mutatingwebhookconfigurations.admissionregistration.k8s.io "e2e-krii-mutating-webhook-configuration" not found
Error from server (NotFound): error when deleting "STDIN": roles.rbac.authorization.k8s.io "e2e-krii-leader-election-role" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "e2e-krii-manager-role" not found
Error from server (NotFound): error when deleting "STDIN": clusterroles.rbac.authorization.k8s.io "e2e-krii-proxy-role" not found
Error from server (NotFound): error when deleting "STDIN": rolebindings.rbac.authorization.k8s.io "e2e-krii-leader-election-rolebinding" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "e2e-krii-manager-rolebinding" not found
Error from server (NotFound): error when deleting "STDIN": clusterrolebindings.rbac.authorization.k8s.io "e2e-krii-proxy-rolebinding" not found
Error from server (NotFound): error when deleting "STDIN": services "e2e-krii-controller-manager-metrics-service" not found
Error from server (NotFound): error when deleting "STDIN": services "e2e-krii-webhook-service" not found
Error from server (NotFound): error when deleting "STDIN": deployments.apps "e2e-krii-controller-manager" not found
Error from server (NotFound): error when deleting "STDIN": certificates.cert-manager.io "e2e-krii-serving-cert" not found
Error from server (NotFound): error when deleting "STDIN": issuers.cert-manager.io "e2e-krii-selfsigned-issuer" not found
Error from server (NotFound): error when deleting "STDIN": servicemonitors.monitoring.coreos.com "e2e-krii-controller-manager-metrics-monitor" not found
Error from server (NotFound): error when deleting "STDIN": validatingwebhookconfigurations.admissionregistration.k8s.io "e2e-krii-validating-webhook-configuration" not found
STEP: uninstalling prometheus manager bundle
running: kubectl delete -f https://github.com/coreos/prometheus-operator/blob/release-0.33/bundle.yaml
error when running kubectl delete during cleaning up prometheus bundle: kubectl delete -f https://github.com/coreos/prometheus-operator/blob/release-0.33/bundle.yaml failed with error: error: error parsing https://github.com/coreos/prometheus-operator/blob/release-0.33/bundle.yaml: error converting YAML to JSON: yaml: line 114: mapping values are not allowed in this context
STEP: uninstalling cert manager bundle
running: kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/v0.11.0/cert-manager.yaml
running: kubectl delete namespace cert-manager
warning: error when cleaning up the cert manager namespace: kubectl delete namespace cert-manager failed with error: Error from server (NotFound): namespaces "cert-manager" not found
STEP: remove container image and work dir
running: docker rmi -f e2e-test/controller-manager:krii
• Failure [90.799 seconds]
kubebuilder
/home/travis/gopath/src/sigs.k8s.io/kubebuilder/test/e2e/v2/e2e_suite.go:33
  with v2 scaffolding
  /home/travis/gopath/src/sigs.k8s.io/kubebuilder/test/e2e/v2/e2e_suite.go:34
    should generate a runnable project [It]
    /home/travis/gopath/src/sigs.k8s.io/kubebuilder/test/e2e/v2/e2e_suite.go:63
    Expected success, but got an error:
        <*errors.errorString | 0xc00003e330>: {
            s: "make docker-build IMG=e2e-test/controller-manager:krii failed with error: /home/travis/gopath/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths=\"./...\"\ngo fmt ./...\ngo vet ./...\n/home/travis/gopath/bin/controller-gen \"crd:trivialVersions=true\" rbac:roleName=manager-role webhook paths=\"./...\" output:crd:artifacts:config=config/crd/bases\ngo test ./... -coverprofile cover.out\n?   \tsigs.k8s.io/kubebuilder\t[no test files]\n?   \tsigs.k8s.io/kubebuilder/api/v1alpha1\t[no test files]\nok  \tsigs.k8s.io/kubebuilder/controllers\t7.478s\tcoverage: 0.0% of statements\ndocker build . -t e2e-test/controller-manager:krii\nSending build context to Docker daemon  105.5kB\r\r\nStep 1/14 : FROM golang:1.12.5 as builder\n1.12.5: Pulling from library/golang\n6f2f362378c5: Pulling fs layer\n494c27a8a6b8: Pulling fs layer\n7596bb83081b: Pulling fs layer\n372744b62d49: Pulling fs layer\n0658c6765517: Pulling fs layer\ndbd5b1749350: Pulling fs layer\nca642e59c356: Pulling fs layer\n0658c6765517: Waiting\n372744b62d49: Waiting\ndbd5b1749350: Waiting\nca642e59c356: Waiting\n7596bb83081b: Verifying Checksum\n7596bb83081b: Download complete\n494c27a8a6b8: Verifying Checksum\n494c27a8a6b8: Download complete\n6f2f362378c5: Verifying Checksum\n6f2f362378c5: Download complete\n372744b62d49: Verifying Checksum\n372744b62d49: Download complete\n0658c6765517: Verifying Checksum\n0658c6765517: Download complete\nca642e59c356: Verifying Checksum\nca642e59c356: Download complete\ndbd5b1749350: Verifying Checksum\ndbd5b1749350: Download complete\n6f2f362378c5: Pull complete\n494c27a8a6b8: Pull complete\n7596bb83081b: Pull complete\n372744b62d49: Pull complete\n0658c6765517: Pull complete\ndbd5b1749350: Pull complete\nca642e59c356: Pull complete\nDigest: sha256:5bc207d20bd9e97c57dc2ad013f2b02d982cc6e411d48bdc79d90bb601696650\nStatus: Downloaded newer image for golang:1.12.5\n ---> 1ef078f0da9e\nStep 2/14 : WORKDIR /workspace\n ---> Running in 181f268b30fc\nRemoving intermediate container 181f268b30fc\n ---> 636713142316\nStep 3/14 : COPY go.mod go.mod\n ---> 23e468b7320f\nStep 4/14 : COPY go.sum go.sum\n ---> e42554350810\nStep 5/14 : RUN go mod download\n ---> Running in 1173980644b1\n\x1b[91mgo: finding github.com/onsi/ginkgo v1.6.0\n\x1b[0m\x1b[91mgo: finding github.com/onsi/gomega v1.4.2\n\x1b[0m\x1b[91mgo: finding github.com/go-logr/logr v0.1.0\n\x1b[0m\x1b[91mgo: finding sigs.k8s.io/controller-runtime v0.3.0\n\x1b[0m\x1b[91mgo: finding k8s.io/apimachinery v0.0.0-20190817020851-f2f3a405f61d\n\x1b[0m\x1b[91mgo: finding k8s.io/client-go v0.0.0-20190918200256-06eb1244587a\n\x1b[0m\x1b[91mgo: finding github.com/golang/protobuf v1.2.0\n\x1b[0m\x1b[91mgo: finding github.com/fsnotify/fsnotify v1.4.7\n\x1b[0m\x1b[91mgo: finding github.com/hpcloud/tail v1.0.0\n\x1b[0m\x1b[91mgo: finding golang.org/x/net v0.0.0-20180906233101-161cd47e91fd\n\x1b[0m\x1b[91mgo: finding golang.org/x/text v0.3.0\n\x1b[0m\x1b[91mgo: finding golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e\n\x1b[0m\x1b[91mgo: finding k8s.io/api v0.0.0-20190918195907-bd6ac527cfd2\n\x1b[0m\x1b[91mverifying k8s.io/[email protected]/go.mod: checksum mismatch\n\tdownloaded: h1:nEP/6rwhzfljWYGVS6pfyES3ipZTR19vzMnSM+ur3ho=\n\tgo.sum:     h1:3jediapYqJ2w1BFw7lAZPCx7scubsTfosqHkhXCWJKw=\n\x1b[0mThe command '/bin/sh -c go mod download' returned a non-zero code: 1\nMakefile:59: recipe for target 'docker-build' failed\nmake: *** [docker-build] Error 1\n",
        }
        make docker-build IMG=e2e-test/controller-manager:krii failed with error: /home/travis/gopath/bin/controller-gen object:headerFile=./hack/boilerplate.go.txt paths="./..."
        go fmt ./...
        go vet ./...

The first time that it has broken as . after a merge in the quickstart which shows not be the root cause at all.

[camilamacedo86]

/kind bug

[camilamacedo86]

/assign @camilamacedo86

[akutz]

Hi @camilamacedo86,

We're seeing this issue as well. Did you happen to find any solution that didn't involve updating to Go 1.13? Why is the cause rooted in Go 1.12? Has the checksum logic changed?

[camilamacedo86]

Hi @akutz, the k8s projects have been upgraded to use 1.13 so if would like to keep using go 1.12 you need to ensure that you are using fixed versions for the projects and modules an etc to get the versions before these changes. I hope that it is what you were working for.

[akutz]

Hi @camilamacedo86,

Thank you so much for the rapid response!

[kayrus]

Are you sure that the checksum error relates to a newer go version?

verifying k8s.io/[email protected]/go.mod: checksum mismatch
        downloaded: h1:nEP/6rwhzfljWYGVS6pfyES3ipZTR19vzMnSM+ur3ho=
        go.sum:     h1:3jediapYqJ2w1BFw7lAZPCx7scubsTfosqHkhXCWJKw=

It worked fine within the same go 1.13.4 before previous weekend.

[akutz]

Hi @camilamacedo86,

We're having some internal discussion trying to understand how the version of Go has altered the checksum. That shouldn't happen. Do you have any leads on the root cause to do with the versions?

[akutz]

Hi @kayrus,

Exactly. It looks way more like someone force-pushed an existing tag or branch.

[camilamacedo86]

HI @akutz and @kayrus,

This error for this project was solved in the PR https://github.com/kubernetes-sigs/kubebuilder/pull/1168 by replacing the go docker image version 1.12 to 1.13 since all rest is using go 1.13 already.

It can be because of the GOPROXY as well since in the go 1.13 it no longer requires the usage ofexport GOPROXY?=https://proxy.golang.org/. The checksum mismatch means that the hash of the module downloaded in not == the hash expected. See https://proxy.golang.org/.

[akutz]

FWIW, it's because of the new version selection rules in Go 1.13. See the section on _version selection_. If you disable GOPROXY=direct and GOSUMDB=off and do go mod download you'll get the following error:

go mod tidy
go: sigs.k8s.io/[email protected] requires
    k8s.io/[email protected]: invalid pseudo-version: does not match short name of revision (c18f71bf2947)
make: *** [Makefile:198: modules] Error 1

The _invalid pseudo version_ is because of the aforementioned version selection rules. This is because a transitive dependency is using Go 1.12 to create their own go.mod and go.sum files, and those files have information no longer valid for Go 1.13.

[akutz]

By the way @kayrus, in addition to @camilamacedo86's efforts, we should also thank @ncdc, he may have just root-caused it:

someone created a BRANCH named f2f3a405f61d and that’s why this is messed up

See https://github.com/kubernetes/apimachinery/commit/f2f3a405f61d. That's the GH branch icon :(

[akutz]

Hi @camilamacedo86 / @kayrus,

Please see this Slack discussion with @liggitt and @ncdc. They're talking about deleting the branch -- it should fix this issue without needing to update Go.

[Shashankft9]

I am facing this issue currently, whats the resolution?

go build -o main
verifying k8s.io/[email protected]/go.mod: checksum mismatch
        downloaded: h1:3jediapYqJ2w1BFw7lAZPCx7scubsTfosqHkhXCWJKw=
        go.sum:     h1:nEP/6rwhzfljWYGVS6pfyES3ipZTR19vzMnSM+ur3ho=

go version
go version go1.14.2 linux/amd64