Pipeline: Tekton controller goes in crashloopbackoff in EKS

[hprateek43]

Expected Behavior

The pipelines controller should work on EKS 1.15 with KUBERNETES_MIN_VERSION=1.15.11

Actual Behavior

The pod controller crashes due to the SHA appended at the end of version string

kubectl logs tekton-pipelines-controller-795b6fc6c7-8wz4h -n tekton-pipelines

2020/10/06 05:08:25 Registering 4 clients
2020/10/06 05:08:25 Registering 4 informer factories
2020/10/06 05:08:25 Registering 8 informers
2020/10/06 05:08:25 Registering 2 controllers
{"level":"info","caller":"logging/config.go:110","msg":"Successfully created the logger."}
{"level":"info","caller":"logging/config.go:111","msg":"Logging level set to info"}
{"level":"info","logger":"tekton","caller":"profiling/server.go:59","msg":"Profiling enabled: false","commit":"61e4167"}
{"level":"fatal","logger":"tekton","caller":"sharedmain/main.go:345","msg":"Version check failed","commit":"61e4167","error":"kubernetes version \"1.15.11-eks-065dce\" is not compatible, need at least \"1.15.11\" (this can be overridden with the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"github.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain.CheckK8sClientMinimumVersionOrDie\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain/main.go:345\ngithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain.MainWithConfig\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain/main.go:168\ngithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain.MainWithContext\n\tgithub.com/tektoncd/pipeline/vendor/knative.dev/pkg/injection/sharedmain/main.go:131\nmain.main\n\tgithub.com/tektoncd/pipeline/cmd/controller/main.go:68\nruntime.main\n\truntime/proc.go:203"}

Steps to Reproduce the Problem

1. Spin up EKS 1.15
2. Deploy Tekton release yaml
3. Check logs for controller in crashloopbackoff

Additional Info

• Kubernetes version:

  Output of kubectl version:

  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-065dce", GitCommit:"065dcecfcd2a91bd68a17ee0b5e895088430bd05", GitTreeState:"clean", BuildDate:"2020-07-16T01:44:47Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

v0.16.3

[bobcatfish]

eek thanks for reporting this @hprateek43 !

[hprateek43]

A solution can be to use the contains check for the min version rather than exact match. If you want I can pick this up as a contribution.

[danielhelfand]

Similar to https://github.com/tektoncd/triggers/issues/627. I think this would need to be a pr to knative/pkg?

[bobcatfish]

Opened https://github.com/knative/pkg/issues/1883 in knative/pkg

One weird thing is that our docs for pipelines 0.16.3 seem to indicate you need kubernetes 0.16 (https://github.com/tektoncd/pipeline/blob/v0.16.3/docs/install.md#before-you-begin) but the error looks like it's checking for 0.15 🤔

[vdemeester]

One weird thing is that our docs for pipelines 0.16.3 seem to indicate you need kubernetes 0.16 (https://github.com/tektoncd/pipeline/blob/v0.16.3/docs/install.md#before-you-begin) but the error looks like it's checking for 0.15 thinking

This is because @hprateek43 is using the KUBERNETES_MIN_VERSION to override the version (as documented in the error message if you are running on 1.15). The problem is that setting KUBERNETES_MIN_VERSION=1.15.11 fails because EKS provides a slightly different version scheme that doesn't seem compatible with what knative/pkg uses for checking the version

[vdemeester]

Looking at knative/pkg#1883, in order to fix this error @hprateek43 you should use KUBERNETES_MIN_VERSION=1.15.11-0 :sweat_smile:

[vdemeester]

Note, this happens for GKE too :sweat_smile: on our CI when updating to knative 0.19 (https://github.com/tektoncd/pipeline/pull/3605)

[vdemeester]

Question is then :

• Should we override this env var in our deployment with the -0 to make sure it works ?
• Should we "try" to fix that upstream as seems like it is gonna happen for all cloud provider ? cc @mattmoor @markusthoemmes @vagababov

[markusthoemmes]

@vdemeester We have since added a more informative log in these cases, but yeah, it sounds like it would be useful to just ignore the pre-version part as it feels kinda unintuitive and I can't think of a case where ignoring it would actually do the wrong thing.

[vdemeester]

@vdemeester We have since added a more informative log in these cases, but yeah, it sounds like it would be useful to just ignore the pre-version part as it feels kinda unintuitive and I can't think of a case where ignoring it would actually do the wrong thing.

Yeah that's more or less what I think — mainly because if all user of knative.dev/pkg/version need to override the env var to get it work correctly on each cloud provider, I think we should just ignore the pre-version part. I also don't see where it would do the wrong thing :)

[vdemeester]

Would be fixed by https://github.com/knative/pkg/pull/1944 (@markusthoemmes :heart:)

[vdemeester]

:tada:
This will be automatically fix when we bump to release-0.20 of knative/pkg, so probably in 0.20.

[afrittoli]

We agreed on the WG to decrease the priority of this item because there is a workaround available now.
The fix will be available in the next release.