Serving: CrashLoopBackOff due to 'Failed to get k8s version'

[jansauer]

What version of Knative?

0.10.x

Expected Behavior

'webhook' and 'activator' pods are in 'CrashLoopBackOff' and won't run.

Actual Behavior

'webhook' and 'activator' pods should be in status running.

Steps to Reproduce the Problem

Follow the Knative Install using Gloo on a Kubernetes Cluster guide with a fresh cluster.

$ glooctl version
Client: {"version":"1.3.1"}
Server: {"type":"Knative","kubernetes":{"containers":[{"Tag":"1.3.1","Name":"ingress","Registry":"quay.io/solo-io"},{"Tag":"1.3.1","Name":"discovery","Registry":"quay.io/solo-io"},{"Tag":"1.3.1","Name":"gloo-envoy-wrapper","Registry":"quay.io/solo-io"},{"Tag":"1.3.1","Name":"gloo-envoy-wrapper","Registry":"quay.io/solo-io"},{"Tag":"1.3.1","Name":"gloo","Registry":"quay.io/solo-io"}],"namespace":"gloo-system"}}

$ glooctl install knative --install-eventing --install-knative
installing Knative CRDs...
installing Knative...
Knative successfully installed!
Creating namespace gloo-system... Done.
Starting Gloo installation...

Gloo was successfully installed!

$ kubectl get pods --namespace knative-serving
NAME                              READY   STATUS             RESTARTS   AGE
autoscaler-hpa-5cfd4f6845-44crv   1/1     Running            0          47m
controller-7fd74c8f67-472g8       1/1     Running            0          47m
autoscaler-5c98b7c9b6-vk4sg       1/1     Running            0          47m
webhook-74847bb77c-g7n68          0/1     CrashLoopBackOff   13         47m
activator-77fc555665-72gk7        0/1     CrashLoopBackOff   19         47m

Both log se same error cause. 'Failed to get k8s version'

...
{"level":"info","ts":"2020-01-26T15:44:33.400Z","logger":"activator","caller":"activator/main.go:168","msg":"Starting the knative activator","commit":"a1a35b1","knative.dev/controller":"activator","knative.dev/pod":"activator-77fc555665-72gk7"}
{"level":"error","ts":"2020-01-26T15:44:33.404Z","logger":"activator","caller":"activator/main.go:173","msg":"Failed to get k8s version","commit":"a1a35b1","knative.dev/controller":"activator","knative.dev/pod":"activator-77fc555665-72gk7","error":"kubernetes version \"\" is not compatible, need at l ││ east \"v1.14.0\" (this can be overridden with the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"main.main.func1\n\t/home/prow/go/src/knative.dev/serving/cmd/activator/main.go:173\nknative.dev/serving/vendor/k8s.io/apimachinery/pkg/util/wait.pollImmediateInternal\n\t/home/prow/go/src/knative.dev/serving/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:330\nknative.dev/serving/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/home/prow/go/src/knative.dev/serving/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:326\nmain.main\n\t/home/prow/go/src/knative.dev/serving/cmd/activator/main.go:171\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

I tried searching the code to lern how the version is determined but only gut till 'knative.dev/pkg/client/injection/kube/client' :-/

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-23T14:21:54Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0+k3s.1", GitCommit:"0f644650f5d8e9f091629f860b342f221c46f6d7", GitTreeState:"clean", BuildDate:"2020-01-06T23:22:38Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

[markusthoemmes]

We use the GitVersion part to figure out what version you're using, see

https://github.com/knative/pkg/blob/5ff923b836abcbeb6ac3ff2784db0d5b5786ade2/version/version.go#L61

// String returns info as a human-friendly version string.
func (info Info) String() string {
    return info.GitVersion
}

My guess would be that the build tag is throwing the parser off or something 🤔

[markusthoemmes]

Indeed, the semver parser is the culprit. We should upstream a fix. Meanwhile: Should we change our version comparison to be based on Major + Minor + ".0" maybe? That sounds safer and wouldn't be subject to such parsing errors.

[markusthoemmes]

Opened https://github.com/knative/pkg/pull/1012 and https://github.com/knative/pkg/pull/1013 respectively to solve this issue one way or the other.

[jansauer]

I managed to work around it be adding the following env var to the effected pods but fixing it upstream would be awesome.

        - name: KUBERNETES_MIN_VERSION
          value: v1.17.0+k3s.1

@markusthoemmes thx for the info about the used version string as setting the var to the git version and not the normal version string was the missing info the get it working.

[knative-housekeeping-robot]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale

[markusthoemmes]

This is fixed via https://github.com/knative/pkg/pull/1013

[abvijaykumar]

HI Guyus what is the final fix for this? I am not getting it. I am having the same issue with knative-serving activator

Failed to get k8s version Get "https://10.96.0.1:443/version?timeout=32s"

[vagababov]

Does it really take you 32s to make a call to the master?

[keshamin]

@vagababov The line that @abvijaykumar provided is from Knative activator logs. I have the same crash loop in activator pod:
2020/12/28 11:33:09 Failed to get k8s version Get "https://10.46.128.1:443/version?timeout=32s": EOF