Dashboard: Switch to Go modules breaks ko apply under certain conditions

Created on 9 Jun 2020 · 19Comments · Source: tektoncd/dashboard

Describe the bug

Since the switch, I (anyone else?) can't ko resolve/ko apply Dashboard resources.

Adams-MBP:dashboard aroberts$ echo $GOPATH
/Users/aroberts/go

Adams-MBP:dashboard aroberts$ pwd
/Users/aroberts/go/src/github.com/tektoncd/dashboard

Adams-MBP:dashboard aroberts$ echo $KO_DOCKER_REPO
docker.io/adamroberts

Adams-MBP:dashboard aroberts$ docker login
Authenticating with existing credentials...
Login Succeeded

Expected behaviour

I'd love for ko apply to work again

Steps to reproduce the bug

Tried with ko 0.2, 0.5. Go 1.13.x, Go 1.14.0.

Checkout master of the dashboard. Do the usual command pointing to Docker Desktop or OpenShift (so kustomize build overlays/dev | ko apply -f - or kustomize build overlays/dev-openshift --load_restrictor=LoadRestrictionsNone | ko resolve -f - | kubectl apply -f - --validate=false).

Environment details

Kubernetes Platform:

Docker Desktop
Kubernetes or OpenShift version:
1.16.6 (latest Docker Desktop)

Install mode (if on OpenShift):
Cloud-provider/provisioner:
Versions:
- Tekton Dashboard:
  
  master

Tekton Pipelines:
Tekton Triggers:
- Install namespaces:
Tekton Dashboard:
tekton-pipelines or openshift-pipelines
Tekton Pipelines:
Tekton Triggers:

Additional Info

(base) Adams-MBP:dashboard aroberts$ ko version
0.5.0

(base) Adams-MBP:dashboard aroberts$ kustomize build overlays/dev | ko apply -f -customresourcedefinition.apiextensions.k8s.io/extensions.dashboard.tekton.dev unchanged
serviceaccount/tekton-dashboard unchanged
clusterrole.rbac.authorization.k8s.io/tekton-dashboard-backend unchanged
clusterrole.rbac.authorization.k8s.io/tekton-dashboard-extensions unchanged
clusterrole.rbac.authorization.k8s.io/tekton-dashboard-pipelines unchanged
clusterrole.rbac.authorization.k8s.io/tekton-dashboard-tenant unchanged
clusterrole.rbac.authorization.k8s.io/tekton-dashboard-triggers unchanged
rolebinding.rbac.authorization.k8s.io/tekton-dashboard-pipelines unchanged
rolebinding.rbac.authorization.k8s.io/tekton-dashboard-triggers unchanged
clusterrolebinding.rbac.authorization.k8s.io/tekton-dashboard-backend unchanged
clusterrolebinding.rbac.authorization.k8s.io/tekton-dashboard-extensions unchanged
clusterrolebinding.rbac.authorization.k8s.io/tekton-dashboard-tenant unchanged
service/tekton-dashboard unchanged
deployment.apps/tekton-dashboard configured

(image coordinates remain github.com ones, nothing built)

Checkout commit immediately before Go modules change

(base) Adams-MBP:dashboard aroberts$ git checkout d7ed18a609f7f4c93f6c8c5667078838c8a8a099
Note: checking out 'd7ed18a609f7f4c93f6c8c5667078838c8a8a099'.

HEAD is now at d7ed18a Add frontend support for single namespace visibility support

(base) Adams-MBP:dashboard aroberts$ kustomize build overlays/dev | ko apply -f -2020/06/09 10:46:22 NOTICE!
-----------------------------------------------------------------
We will start requiring ko:// in a coming release.  Please prefix
the following import path for things to continue working:

   github.com/tektoncd/dashboard/cmd/dashboard

For more information see:

   https://github.com/google/ko/issues/158

-----------------------------------------------------------------
2020/06/09 10:46:22 Using base gcr.io/distroless/base:nonroot for github.com/tektoncd/dashboard/cmd/dashboard

This errors too with

error: no objects passed to apply
2020/06/09 10:48:56 error processing import paths in "-": error resolving image references: Get https://gcr.io/v2/: dial tcp: i/o timeout

but at least it tried. 🤔

kinbug

Source

a-roberts

All 19 comments

tried it with ko 0.4.0 and had no issue kustomize build overlays/dev | ko resolve -f -

eddycharly on 9 Jun 2020

gonna try with v0.5.0 now

eddycharly on 9 Jun 2020

@eddycharly yeah I've even tried with a go clean --modcache, my Go set up looks great, the ko resolve just does not switch out the image coordinates any more (which happened when my GOPATH was incorrectly configured or I hadn't set KO_DOCKER_REPO and authenticated) - all are still the same

a-roberts on 9 Jun 2020

tested it with a fresh clone out of GOAPH, ko 0.5.0 and ko 0.5.1, both work fine. investigating further...

eddycharly on 9 Jun 2020

tested it with a fresh clone out of GOAPH, ko 0.5.0 and ko 0.5.1, both work fine. investigating further...

Lemme know if you want anything (e.g. env output etc), feels like there's something interfering with the normal dev flow but not sure what atm - thanks for taking a look, I realise it's awkward when you're trying to make it break...wondering what other comparisons we can do.

a-roberts on 9 Jun 2020

ok, i can repro the issue if i kustomize build overlays/dev | GO111MODULE=off ko resolve -f -
i think something forces GO111MODULE=off on your side, can you try with kustomize build overlays/dev | GO111MODULE=on ko resolve -f - to force it on ?

eddycharly on 9 Jun 2020

@eddycharly same problem (I actually have GO111MODULE=on exported in my shell anyway)

This is my Kubernetes version specifically, running on the latest Docker Desktop

(base) Adams-MBP:dashboard aroberts$ k version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:18:29Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

@mattmoor this is a long shot but when I thought of ko...I thought of you. Any pointers would be great (I'm wondering under what conditions does ko resolve not replace the image coordinates correctly, particularly with a project using Go modules - which shouldn't factor into the equation AFAIK).

a-roberts on 9 Jun 2020

FWIW I'm not seeing any issues either, ko 0.5.1, running it outside of GOPATH 🤷

$ kubectl version --short
Client Version: v1.18.2
Server Version: v1.16.6-beta.0

$ go version
go version go1.13.4 darwin/amd6

AlanGreene on 9 Jun 2020

except when GO111MODULE=off it seems to always work for me...
my Go version is 1.14.2, my ko version is 0.5.0, and i have no GOPATH env variable defined.

i don't think it is related to your kubernetes version, it looks like ko doesn't realize the github.com/tektoncd/dashboard/cmd/dashboard image is the current repository

eddycharly on 9 Jun 2020

Unsetting GOPATH and moving things around doesn't help either - with and without GO111MODULE being on or off

I'm on

commit b86276045745f3ba536605ab4f50db5fdee15256 (HEAD, origin/master, origin/HEAD, master)
Author: Alan Greene <[email protected]>
Date:   Mon Jun 8 18:22:19 2020 +0100

    Remove unnecessary use of redux in tests

commit f0a4ac4e4489af77286951dfadabc92830c59faf
Author: charles-edouard.breteche <[email protected]>
Date:   Sat Jun 6 11:25:34 2020 +0200

    Separate bug report and feature request issue templates

tried with a fresh Docker Desktop cluster as well (it's the same against my OpenShift cluster mind you). My Docker creds are still good and I'm still authorised...

a-roberts on 9 Jun 2020

There are some strange ko issues like this one https://github.com/google/ko/issues/106

When updating Go

eddycharly on 9 Jun 2020

👍1

There are some strange ko issues like this one google/ko#106

When updating Go

Nice find, go version please? 1.14 and the latest 1.13 doesn't cut it for me (untransformed)...

a-roberts on 9 Jun 2020

mine: go version go1.14.2 darwin/amd64

eddycharly on 9 Jun 2020

1.14.4 is the latest I tried, same problem.
1.14.2 is the same, again this is all experimenting with GO111MODULE being on or off, and with directory experiments (in and outside of gopath, with gopath set and not). At this point I'm thinking I'm looking at instrumenting ko or looking for debug goodies.

a-roberts on 9 Jun 2020

Did you try rebuilding ko ?

eddycharly on 9 Jun 2020

Did you try rebuilding ko ?

From source? So I've been grabbing ko binaries since the go get fails, I could build it from source but it feels weird and unnecessary to do so.

Potentially useful

Adams-MBP:dashboard aroberts$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/aroberts/Library/Caches/go-build"
GOENV="/Users/aroberts/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/aroberts/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/aroberts/go-installs/1131/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/aroberts/go-installs/1131/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/aroberts/go/src/github.com/tektoncd/dashboard/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/f4/l7d_4zt53b59hgs8bv8szynm0000gn/T/go-build125344897=/tmp/go-build -gno-record-gcc-switches -fno-common"

a-roberts on 9 Jun 2020

Go get ko but outside of the dashboard checkout

eddycharly on 9 Jun 2020

Go get ko but outside of the dashboard checkout

That was it, thank you! For anyone else who may find this - that means I end up using ko 0.5.1. Outside of the Dashboard repository meant I no longer had any errors with the

 GO111MODULE=on go get github.com/google/ko/cmd/ko

command.

kustomize build overlays/dev | ko apply -f - for this project now works.

/meow party

a-roberts on 9 Jun 2020

🎉1

@a-roberts: cat image

In response to this:

Go get ko but outside of the dashboard checkout

That was it, thank you! For anyone else who may find this - that means I end up using ko 0.5.1. Outside of the Dashboard repository meant I no longer had any errors with the
GO111MODULE=on go get github.com/google/ko/cmd/ko
command.

kustomize build overlays/dev | ko apply -f - for this project now works.

/meow party

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.