K3s: Resolve errors with installing Knative
Describe the bug
Errors running Knative
To Reproduce
Steps to reproduce the behavior:
https://www.knative.dev/docs/install/knative-with-iks/#installing-istio
Screenshots
Error Step 1:
unable to recognize "https://github.com/knative/serving/releases/download/v0.4.0/istio.yaml": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta1"
And when installing Knative Step 2:
unable to recognize "https://github.com/knative/eventing/releases/download/v0.4.0/release.yaml": no matches for kind "ClusterChannelProvisioner" in version "eventing.knative.dev/v1alpha1"
root@erik-k3s-dev:~# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
istio-system cluster-local-gateway-547467ccf6-7xbxw 1/1 Running 0 7m58s
istio-system istio-citadel-7d64db8bcf-vvvc5 1/1 Running 0 7m59s
istio-system istio-cleanup-secrets-hfwww 0/1 Completed 0 8m
istio-system istio-egressgateway-6ddf4c8bd6-k8l94 1/1 Running 0 8m
istio-system istio-galley-7dd996474-kdxj2 1/1 Running 0 8m
istio-system istio-ingressgateway-84b89d647f-th8bk 1/1 Running 0 8m
istio-system istio-pilot-86bb4fcbbd-p7s6c 2/2 Running 0 7m59s
istio-system istio-policy-5c4d9ff96b-xzkbl 2/2 Running 0 7m59s
istio-system istio-sidecar-injector-6977b5cf5b-g85j4 1/1 Running 0 7m59s
istio-system istio-statsd-prom-bridge-b44b96d7b-fzk8l 1/1 Running 0 8m
istio-system istio-telemetry-7676df547f-5xmpd 2/2 Running 0 7m59s
istio-system zipkin-5495cf4cf-jxf2k 1/1 Running 0 6m54s
knative-build build-controller-7b8987d675-mvtkv 1/1 Running 0 6m56s
knative-build build-webhook-74795c8696-564zr 1/1 Running 0 6m56s
knative-eventing eventing-controller-864657d8d4-wqnch 1/1 Running 0 6m56s
knative-eventing in-memory-channel-controller-f794cc9d8-g2qv4 1/1 Running 0 6m55s
knative-eventing in-memory-channel-dispatcher-8595c7f8d7-wtvjs 1/2 CrashLoopBackOff 6 6m55s
knative-eventing webhook-5d76776d55-z9wpm 1/1 Running 0 6m56s
knative-monitoring elasticsearch-logging-0 1/1 Running 0 6m55s
knative-monitoring elasticsearch-logging-1 1/1 Running 0 5m50s
knative-monitoring grafana-568674f4f9-p257v 1/1 Running 0 6m54s
knative-monitoring kibana-logging-7698db4f94-frdmb 1/1 Running 0 6m55s
knative-monitoring kube-state-metrics-58b9779756-bbvsd 4/4 Running 0 5m46s
knative-monitoring node-exporter-ftfzq 2/2 Running 0 6m55s
knative-monitoring prometheus-system-0 1/1 Running 0 6m54s
knative-monitoring prometheus-system-1 1/1 Running 0 6m54s
knative-serving activator-7c8b59d78-gkhnw 1/2 CrashLoopBackOff 6 6m57s
knative-serving autoscaler-666c9bfcc6-5q6z7 1/2 CrashLoopBackOff 6 6m57s
knative-serving controller-799cd5c6dc-sr9md 1/1 Running 0 6m57s
knative-serving webhook-5b66fdf6b9-ssg84 1/1 Running 0 6m57s
knative-sources controller-manager-0 1/1 Running 0 6m55s
kube-system coredns-7748f7f6df-m7jm9 1/1 Running 0 8m42s
kube-system helm-install-traefik-d8j6d 0/1 Completed 0 8m42s
kube-system svclb-traefik-8f7c97697-s5wqd 2/2 Running 1 8m31s
kube-system traefik-6876857645-dl88l 1/1 Running 0 8m31s
root@erik-k3s-dev:~# kubectl logs -n knative-eventing in-memory-channel-dispatcher-8595c7f8d7-wtvjs dispatcher
{"level":"fatal","ts":1552001051.4944558,"caller":"fanoutsidecar/main.go:91","msg":"Unable to create configMap noticer.","error":"Get https://10.43.0.1:443/api?timeout=32s: EOF","stacktrace":"main.main\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:91\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"}
root@erik-k3s-dev:~# kubectl logs -n knative-serving activator-7c8b59d78-gkhnw activator
{"level":"info","ts":"2019-03-07T23:25:49.551Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n \"level\": \"info\",\n \"development\": false,\n \"outputPaths\": [\"stdout\"],\n \"errorOutputPaths\": [\"stderr\"],\n \"encoding\": \"json\",\n \"encoderConfig\": {\n \"timeKey\": \"ts\",\n \"levelKey\": \"level\",\n \"nameKey\": \"logger\",\n \"callerKey\": \"caller\",\n \"messageKey\": \"msg\",\n \"stacktraceKey\": \"stacktrace\",\n \"lineEnding\": \"\",\n \"levelEncoder\": \"\",\n \"timeEncoder\": \"iso8601\",\n \"durationEncoder\": \"\",\n \"callerEncoder\": \"\"\n }\n}"}
{"level":"info","ts":"2019-03-07T23:25:49.551Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-03-07T23:25:49.551Z","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.4\" is not a valid GitHub commit ID"}
{"level":"info","ts":"2019-03-07T23:25:49.551Z","logger":"activator","caller":"activator/main.go:131","msg":"Starting the knative activator","knative.dev/controller":"activator"}
{"level":"fatal","ts":"2019-03-07T23:25:51.644Z","logger":"activator","caller":"activator/main.go:147","msg":"Version check failed: Get https://10.43.0.1:443/version?timeout=32s: EOF","knative.dev/controller":"activator","stacktrace":"main.main\n\t/go/src/github.com/knative/serving/cmd/activator/main.go:147\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"}
root@erik-k3s-dev:~# kubectl logs -n knative-serving autoscaler-666c9bfcc6-5q6z7 autoscaler
{"level":"info","ts":"2019-03-07T23:25:24.499Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n \"level\": \"info\",\n \"development\": false,\n \"outputPaths\": [\"stdout\"],\n \"errorOutputPaths\": [\"stderr\"],\n \"encoding\": \"json\",\n \"encoderConfig\": {\n \"timeKey\": \"ts\",\n \"levelKey\": \"level\",\n \"nameKey\": \"logger\",\n \"callerKey\": \"caller\",\n \"messageKey\": \"msg\",\n \"stacktraceKey\": \"stacktrace\",\n \"lineEnding\": \"\",\n \"levelEncoder\": \"\",\n \"timeEncoder\": \"iso8601\",\n \"durationEncoder\": \"\",\n \"callerEncoder\": \"\"\n }\n}"}
{"level":"info","ts":"2019-03-07T23:25:24.499Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-03-07T23:25:24.499Z","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.4\" is not a valid GitHub commit ID"}
{"level":"fatal","ts":"2019-03-07T23:25:33.647Z","logger":"autoscaler","caller":"autoscaler/main.go:95","msg":"Version check failed: an error on the server (\"\") has prevented the request from succeeding","stacktrace":"main.main\n\t/go/src/github.com/knative/serving/cmd/autoscaler/main.go:95\nruntime.main\n\t/root/sdk/go1.12rc1/src/runtime/proc.go:200"}
root@erik-k3s-dev:~# head -3 /proc/meminfo
MemTotal: 32939856 kB
MemFree: 16705668 kB
MemAvailable: 23599904 kB
Additional context
rancher/k3s/issues/90
rancher/k3s/issues/285
kinbug
erikwilson
👍1
All 14 comments
I don't see any errors any more with the latest k3s, for the release error it seems to be a race condition happens because crds for knative wasn't initialized, here is the original issue https://github.com/knative/eventing/issues/680
Running the command again works just fine.
galal-hussein
on 27 Apr 2019
I'm still seeing some issues with certain pods, running with knative 0.5.0 and k3s v0.5.0-rc1:
[ 2019-04-26 16:37:28 ]
root馃惍k3s:~$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
istio-system cluster-local-gateway-5bf5488bb-f5x42 1/1 Running 0 6m59s
istio-system istio-citadel-78fd647cf-cb2b4 1/1 Running 0 7m
istio-system istio-cleanup-secrets-gfqtd 0/1 Completed 0 6m59s
istio-system istio-egressgateway-5547468b8d-6p4mt 1/1 Running 0 7m
istio-system istio-galley-64ddbbff8b-cjjbk 1/1 Running 0 7m
istio-system istio-ingressgateway-686d54b4bf-6qsdl 1/1 Running 0 7m
istio-system istio-pilot-7bd454bf5c-g2cvr 2/2 Running 0 7m
istio-system istio-pilot-7bd454bf5c-lgfg8 2/2 Running 0 6m45s
istio-system istio-pilot-7bd454bf5c-vztzx 2/2 Running 0 6m45s
istio-system istio-policy-64b5ff6bd8-2dtsn 2/2 Running 0 7m
istio-system istio-security-post-install-2t84k 0/1 Completed 0 6m59s
istio-system istio-sidecar-injector-6956c47586-t2phg 1/1 Running 0 7m
istio-system istio-telemetry-55b97c59c4-w25kj 2/2 Running 0 7m
istio-system svclb-istio-ingressgateway-68sqr 0/8 Pending 0 7m1s
istio-system zipkin-57869dbb48-k7ld2 1/1 Running 0 6m16s
knative-build build-controller-77cbbc9888-dw2j9 1/1 Running 0 5m37s
knative-build build-webhook-64f9bbc7c9-qtzqx 1/1 Running 0 5m28s
knative-eventing eventing-controller-f747584b7-zwb9c 1/1 Running 0 5m28s
knative-eventing in-memory-channel-controller-5f587cc68c-x4q5g 1/1 Running 0 5m37s
knative-eventing in-memory-channel-dispatcher-9f7d664d4-8k9k7 1/2 CrashLoopBackOff 5 5m37s
knative-eventing webhook-5fd955f896-msszp 1/1 Running 0 5m37s
knative-monitoring elasticsearch-logging-0 0/1 CrashLoopBackOff 3 5m37s
knative-monitoring elasticsearch-logging-1 0/1 CrashLoopBackOff 3 3m40s
knative-monitoring grafana-69f88c5544-ggrdn 1/1 Running 0 5m37s
knative-monitoring kibana-logging-7cb6b64bff-tvsw7 1/1 Running 0 5m37s
knative-monitoring kube-state-metrics-75fdd67ffc-7jpph 4/4 Running 0 4m21s
knative-monitoring node-exporter-zqn67 2/2 Running 0 5m20s
knative-monitoring prometheus-system-0 1/1 Running 0 5m37s
knative-monitoring prometheus-system-1 1/1 Running 0 5m37s
knative-serving activator-654b696b84-zmkkq 1/2 CrashLoopBackOff 5 5m28s
knative-serving autoscaler-585b674986-lrvsk 2/2 Running 0 5m28s
knative-serving controller-54b89c944c-qh6fc 1/1 Running 0 5m37s
knative-serving webhook-5cc857b9b8-vps79 1/1 Running 0 5m28s
knative-sources controller-manager-0 1/1 Running 0 5m34s
kube-system coredns-695688789-xbkb5 1/1 Running 0 7m33s
kube-system helm-install-traefik-5mv4v 0/1 Completed 0 7m33s
kube-system svclb-traefik-4b7g9 2/2 Running 0 7m14s
kube-system traefik-55bd9646fc-bxgqj 1/1 Running 0 7m14s
[ 2019-04-26 16:37:43 ]
root馃惍k3s:~$ kubectl logs -n knative-eventing service/in-memory-channel-dispatcher dispatcher
{"level":"error","ts":"2019-04-26T23:36:38.015Z","caller":"fanoutsidecar/main.go:131","msg":"Error starting manager.","error":"Get https://10.43.0.1:443/api?timeout=32s: EOF","stacktrace":"main.setupConfigMapNoticer\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:131\nmain.main\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:92\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"fatal","ts":"2019-04-26T23:36:38.015Z","caller":"fanoutsidecar/main.go:94","msg":"Unable to create configMap noticer.","error":"Get https://10.43.0.1:443/api?timeout=32s: EOF","stacktrace":"main.main\n\t/go/src/github.com/knative/eventing/cmd/fanoutsidecar/main.go:94\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
[ 2019-04-26 16:37:46 ]
root馃惍k3s:~$ kubectl logs -n knative-serving deployment.apps/activator activator | head -5
{"level":"info","ts":"2019-04-26T23:36:24.823Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n \"level\": \"info\",\n \"development\": false,\n \"outputPaths\": [\"stdout\"],\n \"errorOutputPaths\": [\"stderr\"],\n \"encoding\": \"json\",\n \"encoderConfig\": {\n \"timeKey\": \"ts\",\n \"levelKey\": \"level\",\n \"nameKey\": \"logger\",\n \"callerKey\": \"caller\",\n \"messageKey\": \"msg\",\n \"stacktraceKey\": \"stacktrace\",\n \"lineEnding\": \"\",\n \"levelEncoder\": \"\",\n \"timeEncoder\": \"iso8601\",\n \"durationEncoder\": \"\",\n \"callerEncoder\": \"\"\n }\n}"}
{"level":"info","ts":"2019-04-26T23:36:24.823Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-04-26T23:36:24.823Z","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/upstream/release-0.5\" is not a valid GitHub commit ID"}
{"level":"info","ts":"2019-04-26T23:36:24.823Z","logger":"activator","caller":"activator/main.go:134","msg":"Starting the knative activator","knative.dev/controller":"activator"}
{"level":"error","ts":"2019-04-26T23:36:26.827Z","logger":"activator","caller":"activator/main.go:152","msg":"Failed to get k8s version","knative.dev/controller":"activator","error":"Get https://10.43.0.1:443/version?timeout=32s: EOF","stacktrace":"main.main.func1\n\t/go/src/github.com/knative/serving/cmd/activator/main.go:152\ngithub.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait.pollImmediateInternal\n\t/go/src/github.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:245\ngithub.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/go/src/github.com/knative/serving/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:241\nmain.main\n\t/go/src/github.com/knative/serving/cmd/activator/main.go:150\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
Should be fixed with #359 where the advertise ip is set to the node ip.
erikwilson
on 12 Jul 2019
🎉1
k3s version: v0.7.0-rc6
I was able to verify that knative and istio installation worked successfully:
# kubectl get pods --namespace knative-serving
NAME READY STATUS RESTARTS AGE
activator-5ccf59c56-z85fc 1/1 Running 0 96s
autoscaler-7f55589d4b-77pb9 1/1 Running 0 96s
controller-77cd5ffdc4-qgdz7 1/1 Running 0 96s
networking-istio-5476685c6f-4xtcf 1/1 Running 0 96s
webhook-5f668cf9f8-zpwng 1/1 Running 0 96s
# kubectl get pods --namespace knative-build
NAME READY STATUS RESTARTS AGE
build-controller-f969bf74d-7rgg6 1/1 Running 0 96s
build-webhook-6f58484688-l4994 1/1 Running 0 96s
# kubectl get pods --namespace knative-eventing
NAME READY STATUS RESTARTS AGE
eventing-controller-569448686c-t4s9f 1/1 Running 0 95s
eventing-webhook-6d6795d497-2rv87 1/1 Running 0 95s
imc-controller-77f4776c99-mn4z8 1/1 Running 0 95s
imc-dispatcher-7b69ccd5d8-mqgkx 1/1 Running 0 95s
in-memory-channel-controller-85dc669f69-vjqrm 1/1 Running 0 95s
in-memory-channel-dispatcher-7f896c7d85-4tmsh 1/1 Running 0 95s
sources-controller-7b7579bc94-9kgr7 1/1 Running 0 95s
# kubectl get pods --namespace knative-monitoring
NAME READY STATUS RESTARTS AGE
elasticsearch-logging-0 1/1 Running 0 95s
elasticsearch-logging-1 1/1 Running 0 22s
grafana-5b8fcf6dbd-79982 1/1 Running 0 93s
kibana-logging-7cb6b64bff-26brf 1/1 Running 0 94s
kube-state-metrics-75fdd67ffc-zwmb8 4/4 Running 0 45s
node-exporter-wftcq 2/2 Running 0 94s
prometheus-system-0 1/1 Running 0 93s
prometheus-system-1 1/1 Running 0 93s
🎉3
Can you share more details on the steps you took to install Istio (and Knative?) I've been running into issues where the Istio pods are stuck in pending. I've started a thread in the Istio Slack.
Note: I'm running on a Raspberry Pi (ARM64) cluster set up with K3sup.
beriberikix
on 29 Oct 2019
It looks like istio may be broken with k8s 1.16 from the knative instructions, where some of the beta APIs have been removed.
erikwilson
on 29 Oct 2019
Oh interesting, you mean these instructions? Any proposed changes?
FWIW, it seems like the latest k3sup doesn't use 1.16.
jonathanberi:istio-1.3.3 馃殌 $ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:51:13Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4-k3s.1", GitCommit:"a7531b1ab3fd5ff987b074472ddfb84a2f5326bc", GitTreeState:"clean", BuildDate:"2019-09-19T22:36Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/arm"}
It looks like istio has node selectors for amd64, ppc64le, or s390x architectures (quick look at docker hub seems they only publish for amd64), so makes sense that arm64 or arm would remain in pending. :(
erikwilson
on 29 Oct 2019
I believe many of us willing to try out Knative since it's a very active project. I'll be great if we have a documentation section or wiki on this as a tutorial.
I'm also stucking installing knative using glooctl. I remove the traefik from system namespace, and the installation of Knative+Gloo goes fine, but there is one blocking issue with gloo proxy that I can't connect to. There is some missing points but I don't know where to start to research.
I will try with Istio to see everything going fine.
xgenvn
on 4 Nov 2019
I'v encountered same problem with knative 0.13.0:
$ kubectl apply --filename https://github.com/knative/serving/releases/download/v0.13.0/serving-istio.yaml 15:03:18
clusterrole.rbac.authorization.k8s.io/knative-serving-istio unchanged
configmap/config-istio unchanged
deployment.apps/networking-istio unchanged
unable to recognize "https://github.com/knative/serving/releases/download/v0.13.0/serving-istio.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
unable to recognize "https://github.com/knative/serving/releases/download/v0.13.0/serving-istio.yaml": no matches for kind "Gateway" in version "networking.istio.io/v1alpha3"
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4+k3s1", GitCommit:"3eee8ac3a1cf0a216c8a660571329d4bda3bdf77", GitTreeState:"clean", BuildDate:"2020-03-25T16:13:25Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4+k3s1", GitCommit:"3eee8ac3a1cf0a216c8a660571329d4bda3bdf77", GitTreeState:"clean", BuildDate:"2020-03-25T16:13:25Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
$ docker -v
Docker version 17.06.2-ce, build 0e7f239
$ k3s --version
k3s version v1.17.4+k3s1 (3eee8ac3)
It's a two nodes cluster:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
n227-160-185 Ready master 28h v1.17.4+k3s1
n227-160-033 Ready master 26h v1.17.4+k3s1
amio
on 21 Apr 2020
@amio sounds like your Istio CRDs are missing or incompatible. Which version of Istio did you install?
Possibly related: https://github.com/knative/serving/issues/2195
brandond
on 21 Apr 2020
@brandond I was using this to install it:
$ kubectl apply --filename https://github.com/knative/serving/releases/download/v0.13.0/serving-crds.yaml 14:58:14
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
@amio that doesn't include any of the istio CRDs, just the knative CRDs. Did you maybe need to upgrade istio before upgrading knative? It looks like it wants 1.4.6: https://knative.dev/development/install/installing-istio/
brandond
on 22 Apr 2020
@brondond Thanks for pointing out! I did miss the step of installing Istio 馃槅
amio
on 23 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
giezi
路
3Comments
pierreozoux
路
4Comments
davidnuzik
路
3Comments
joakimr-axis
路
3Comments
wpwoodjr
路
3Comments
Most helpful comment
k3s version: v0.7.0-rc6
I was able to verify that knative and istio installation worked successfully: