Serving: E2E tests fail on Prow with newer ko versions

[adrcunha]

When making Prow use go 1.11.1 and Cloud SDK 225, E2E tests started failing with different errrors:

https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2468/pull-knative-serving-integration-tests/1062426675620352001/

W1113 19:37:16.237] 2018/11/13 19:37:16 error processing import paths in "config/webhook.yaml": BLOB_UNKNOWN: "Manifest \"sha256:e392d2a37aeba7425e33c18b57a1bf4a4663e35f5d67445727cafe2527edca6f\" blob \"sha256:f39c3f3ec4ec5beb7fbe9400562f4b186b5e784501d5d79f537d1ad376057b61\" unknown to registry."
...
I1113 19:37:40.400] >> Patching Istio
W1113 19:37:40.511] Error from server (NotFound): horizontalpodautoscalers.autoscaling "knative-ingressgateway" not found
I1113 19:37:40.612] ERROR: Knative Serving installation failed

https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2437/pull-knative-serving-integration-tests/1062427808908709890/

W1113 19:41:33.420] 2018/11/13 19:41:33 Unexpected error running "go build": exit status 2
W1113 19:41:33.421] # github.com/knative/serving/pkg/apis/serving/v1alpha1
W1113 19:41:33.422] pkg/apis/serving/v1alpha1/revision_validation.go:66:11: undefined: apis.ErrOutOfBoundsValue
W1113 19:41:33.422] 2018/11/13 19:41:33 error processing import paths in "config/activator.yaml": exit status 2
...
I1113 19:41:57.178] >> Patching Istio
W1113 19:41:57.282] Error from server (NotFound): horizontalpodautoscalers.autoscaling "knative-ingressgateway" not found
I1113 19:41:57.383] ERROR: Knative Serving installation failed

https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-serving-continuous/1062420259731935234

W1113 19:13:15.335] 2018/11/13 19:13:15 error processing import paths in "config/999-cache.yaml": BLOB_UNKNOWN: "Manifest \"sha256:496427954201b3d3bb709160cc9bb44d87f0946ab6526e52cdd4745321c89e0f\" blob \"sha256:2992b18231ba9f736eadad4889e6cdcb1cc5e80b1646fc4d61a5d39afe09d391\" unknown to registry."
...
I1113 19:13:39.642] >> Patching Istio
W1113 19:13:39.758] Error from server (NotFound): horizontalpodautoscalers.autoscaling "knative-ingressgateway" not found
I1113 19:13:39.859] ERROR: Knative Serving installation failed

To reproduce it, you must start a Prow test job using prow-tests:v20181113-07b942c-dirty.

[adrcunha]

Actually, the last link points to a Prow job using go 1.10 and cloud SDK 221. The root cause for these errors might be unrelated to Prow or the go/cloud version.

[adrcunha]

The "error processing import paths" issue still happpens with go 1.11 and Cloud SDK 228:

W1214 23:25:15.835] 2018/12/14 23:25:15 error processing import paths in "config/config-controller.yaml": BLOB_UNKNOWN: "Manifest \"sha256:49f2e2fc674322ac1aa1385f19bdacd26f079a4b7036a5ffd34a503d83ea9e06\" blob \"sha256:85cffe608e625a1a4f494ad319d83f9b6117e6f57dd838c25fcc52f10c3d03e2\" unknown to registry."

https://storage.googleapis.com/knative-prow/pr-logs/pull/knative_serving/2725/pull-knative-serving-integration-tests/1073718706254647296/build-log.txt

[srinivashegde86]

this is now happening with go 1.10 as well - https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2725/pull-knative-serving-upgrade-tests/1074733769555972099/

[adrcunha]

This is actually unrelated to Cloud SDK and go version. It failed for the same reason with Cloud SDK 221 and go 1.10:

https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2725/pull-knative-serving-integration-tests/1074773657357651968/

I'm now suspicious of ko.

[adrcunha]

prow-tests image v20181025-0fcaba1 works fine. It uses Cloud SDK 221 and go 1.10:

https://prow.knative.dev/log?job=pull-knative-serving-integration-tests&id=1074778816099913728

[jonjohnsonjr]

It looks like this is happening with github.com/knative/serving/cmd/queue a lot.

Two things I've noticed:

1. That package has symlinks in kodata. We saw another failure in build-pipeline with the same for the webhook package.
2. That package is being built and published twice for serving: here and here.

I think there might be several issues, but one of them is probably symlink related + a race.

Also, it seems that invoking go build concurrently isn't safe, which is exactly what ko does, but I would be surprised if we were downloading modules because we vendor everything... right?

@mattmoor any thoughts?

[adrcunha]

Nailed down the culprit to ko. Details and logs in https://github.com/knative/serving/pull/2745. The culprit seems to be https://github.com/google/go-containerregistry/pull/297

Jon, you authored that PR and has way more context than me, so I'm assigning this issue to you.

/assign @jonjohnsonjr

[jonjohnsonjr]

@adrcunha thank you for narrowing it down to that commit, that's really helpful. This is interesting because I've suspected something about the test environment might be causing an issue (which is why I can't reproduce it locally). Is it possible to get a dump of the environment variables?

I suspect there's a bug/race somewhere that involves all of this + the symlinks stuff in my comment:

• https://github.com/google/go-containerregistry/issues/112
• https://github.com/google/go-containerregistry/blob/e0f538712eb0767ae6959531febe6041fa466602/pkg/v1/tarball/layer.go#L81
• https://github.com/google/go-containerregistry/blob/e0f538712eb0767ae6959531febe6041fa466602/pkg/ko/build/gobuild.go#L190

And I suspect there's an environment difference between {prow, me, that commit} that triggers the bug/race in our test environment but not when run locally...

[adrcunha]

@jonjohnsonjr environment variables are dumped here:

https://prow.knative.dev/log?job=pull-knative-serving-integration-tests&id=1075118053097738241

Search for @@@, they're right after the first occurrence.

[jonjohnsonjr]

I have a theory that https://github.com/google/go-containerregistry/pull/297 has a hard requirement on a later version of kubectl than we're using, perhaps?

I1218 19:58:36.888] >> kubectl version
I1218 19:58:36.985] Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
I1218 19:58:36.985] Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.11", GitCommit:"dc4f6dda6a08aae2108d7a7fdc2a44fa23900f4c", GitTreeState:"clean", BuildDate:"2018-11-10T20:22:02Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Might be related to https://github.com/google/go-containerregistry/pull/328 as well.

[jonjohnsonjr]

I believe this was fixed by https://github.com/google/go-containerregistry/pull/330

[jonjohnsonjr]

/assign @adrcunha

[adrcunha]

@jonjohnsonjr the newest ko doesn't throw "error processing import paths" anymore, but the E2E tests are failing consistently with

W1222 02:12:32.821] unable to recognize "/tmp/tmp.HqUwlDylgM/release-no-mon.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"

e.g. https://gubernator.knative.dev/build/knative-prow/pr-logs/pull/knative_serving/2745/pull-knative-serving-integration-tests/1076297323824287744/

Any clues?

[tcnghia]

@adrcunha this can be fixed by retrying. It's a race condition of kubectl.

[srinivashegde86]

We should add the retry even to our current prow job setup. I sometimes see this error during the current e2e setup as well.

[adrcunha]

I'm not convinced that this is a race condition of kubectl. The E2E tests failed 3 times in a row for the same reason. It doesn't happen with the older version of ko, as you can see in the CI flow.

[adrcunha]

Actually, this is the real culprit:

I1218 06:48:00.380] >> Creating test resources (test/config/)
I1218 06:48:00.391] ERROR: Knative Serving installation failed

[adrcunha]

2745 confirmed that the issues was fixed by google/go-containerregistry#330.

[adrcunha]

Sample job using the latest ko:

https://prow.knative.dev/log?job=pull-knative-serving-integration-tests&id=1079514393449861120