Serving: installing knative results in the `server could not find the requested resource (post gatewaies.networking.istio.io)`

[cppforlife]

/area test-and-release
/kind bug

Expected Behavior

following installation instructions from docs repo should result in working isitio+knative setup.

Actual Behavior

kubectl apply of knative ymls fails with following error:

Error from server (NotFound): error when creating "https://storage.googleapis.com/knative-releases/latest/release.yaml": the server could not find the requested resource (post gatewaies.networking.istio.io)

Steps to Reproduce the Problem

1. setup default GKE with 1.10.5-gke.0
2. install istio successfully
3. fail to install knative

[cppforlife]

cc @maximilien

[sukhil-suresh]

@cppforlife EXCITED to see you on Knative!!

Tried just now with a fresh GKE cluster, by following Knative the easy way docs and was unable to reproduce the problem. I was also able to deploy the Hello World - Go sample on Knative.

Sorry, I know that ^ is not very helpful. My best (not great) guess is to wonder if you confirmed that the Istio components were Running or Completed, before proceeding with the Knative deploy?

Ref: As part of Step 2: Install Istio

Monitor the Istio components, until all of the components report Running or Completed

Also, confused as to why your error message refers to gatewaies.networking.istio.io. The closest match I see in the output when installing istio are the references to:

...
gateway.networking.istio.io/knative-shared-gateway created
...
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io created
...

Not sure if there were any changes to the repos since you last tried. But, could consider trying again since it worked on my machine? Horrible! :)

[sukhil-suresh]

just an FYI... my cluster zone was northamerica-northeast1-a

[tcnghia]

@cppforlife do you still have a copy of your YAML file? I'd love to take a look to see why any reference to 'gatewaies' is still there.

[cppforlife]

ill give another try later today, will report back.

[maximilien]

Attempted with a new cluster on GKE running:

 $ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T19:32:09Z", GoVersion:"go1.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.5-gke.0", GitCommit:"f4c74e18e57148052c59cc0467bb7e99dcc46399", GitTreeState:"clean", BuildDate:"2018-06-21T14:11:26Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Same problem. Output:

 $ kubectl get node
NAME                                       STATUS    AGE       VERSION
gke-dk-jul-12-default-pool-d49794d7-rjj6   Ready     3h        v1.10.5-gke.0
gke-dk-jul-12-default-pool-d49794d7-t1rw   Ready     3h        v1.10.5-gke.0
gke-dk-jul-12-default-pool-d49794d7-zk24   Ready     3h        v1.10.5-gke.0
 $ ./perms
clusterrolebinding "cluster-admin-binding" created
 $ wget -O- https://storage.googleapis.com/knative-releases/latest/istio.yaml > istio.yml
--2018-07-12 14:06:24--  https://storage.googleapis.com/knative-releases/latest/istio.yaml
Resolving storage.googleapis.com... 216.58.194.176, 2607:f8b0:4005:804::2010
Connecting to storage.googleapis.com|216.58.194.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 84026 (82K) [application/octet-stream]
Saving to: 'STDOUT'

100%[==================================================================================================================>] 84,026      --.-K/s   in 0.008s

2018-07-12 14:06:24 (10.6 MB/s) - written to stdout [84026/84026]

 $ kubectl apply -f istio.yml
namespace "istio-system" created
configmap "istio-statsd-prom-bridge" created
configmap "istio-mixer-custom-resources" created
configmap "istio" created
configmap "istio-sidecar-injector" created
serviceaccount "istio-egressgateway-service-account" created
serviceaccount "istio-ingress-service-account" created
serviceaccount "istio-ingressgateway-service-account" created
serviceaccount "istio-mixer-post-install-account" created
clusterrole "istio-mixer-post-install-istio-system" created
clusterrolebinding "istio-mixer-post-install-role-binding-istio-system" created
job "istio-mixer-post-install" created
serviceaccount "istio-mixer-service-account" created
serviceaccount "istio-pilot-service-account" created
serviceaccount "istio-citadel-service-account" created
serviceaccount "istio-cleanup-old-ca-service-account" created
serviceaccount "istio-sidecar-injector-service-account" created
customresourcedefinition "rules.config.istio.io" created
customresourcedefinition "attributemanifests.config.istio.io" created
customresourcedefinition "circonuses.config.istio.io" created
customresourcedefinition "deniers.config.istio.io" created
customresourcedefinition "fluentds.config.istio.io" created
customresourcedefinition "kubernetesenvs.config.istio.io" created
customresourcedefinition "listcheckers.config.istio.io" created
customresourcedefinition "memquotas.config.istio.io" created
customresourcedefinition "noops.config.istio.io" created
customresourcedefinition "opas.config.istio.io" created
customresourcedefinition "prometheuses.config.istio.io" created
customresourcedefinition "rbacs.config.istio.io" created
customresourcedefinition "servicecontrols.config.istio.io" created
customresourcedefinition "solarwindses.config.istio.io" created
customresourcedefinition "stackdrivers.config.istio.io" created
customresourcedefinition "statsds.config.istio.io" created
customresourcedefinition "stdios.config.istio.io" created
customresourcedefinition "apikeys.config.istio.io" created
customresourcedefinition "authorizations.config.istio.io" created
customresourcedefinition "checknothings.config.istio.io" created
customresourcedefinition "kuberneteses.config.istio.io" created
customresourcedefinition "listentries.config.istio.io" created
customresourcedefinition "logentries.config.istio.io" created
customresourcedefinition "metrics.config.istio.io" created
customresourcedefinition "quotas.config.istio.io" created
customresourcedefinition "reportnothings.config.istio.io" created
customresourcedefinition "servicecontrolreports.config.istio.io" created
customresourcedefinition "tracespans.config.istio.io" created
customresourcedefinition "serviceroles.config.istio.io" created
customresourcedefinition "servicerolebindings.config.istio.io" created
customresourcedefinition "destinationpolicies.config.istio.io" created
customresourcedefinition "egressrules.config.istio.io" created
customresourcedefinition "routerules.config.istio.io" created
customresourcedefinition "virtualservices.networking.istio.io" created
customresourcedefinition "destinationrules.networking.istio.io" created
customresourcedefinition "serviceentries.networking.istio.io" created
customresourcedefinition "gateways.networking.istio.io" created
customresourcedefinition "policies.authentication.istio.io" created
customresourcedefinition "httpapispecbindings.config.istio.io" created
customresourcedefinition "httpapispecs.config.istio.io" created
customresourcedefinition "quotaspecbindings.config.istio.io" created
customresourcedefinition "quotaspecs.config.istio.io" created
clusterrole "istio-ingress-istio-system" created
clusterrole "istio-mixer-istio-system" created
clusterrole "istio-pilot-istio-system" created
clusterrole "istio-citadel-istio-system" created
role "istio-cleanup-old-ca-istio-system" created
clusterrole "istio-sidecar-injector-istio-system" created
clusterrolebinding "istio-ingress-istio-system" created
clusterrolebinding "istio-mixer-admin-role-binding-istio-system" created
clusterrolebinding "istio-pilot-istio-system" created
clusterrolebinding "istio-citadel-istio-system" created
rolebinding "istio-cleanup-old-ca-istio-system" created
clusterrolebinding "istio-sidecar-injector-admin-role-binding-istio-system" created
service "istio-egressgateway" created
service "istio-ingress" created
service "istio-ingressgateway" created
service "istio-policy" created
service "istio-telemetry" created
service "istio-statsd-prom-bridge" created
deployment "istio-statsd-prom-bridge" created
service "istio-pilot" created
service "istio-citadel" created
service "istio-sidecar-injector" created
deployment "istio-egressgateway" created
deployment "istio-ingress" created
deployment "istio-ingressgateway" created
deployment "istio-policy" created
deployment "istio-telemetry" created
deployment "istio-pilot" created
deployment "istio-citadel" created
deployment "istio-sidecar-injector" created
job "istio-cleanup-old-ca" created
horizontalpodautoscaler "istio-egressgateway" created
horizontalpodautoscaler "istio-ingress" created
horizontalpodautoscaler "istio-ingressgateway" created
mutatingwebhookconfiguration "istio-sidecar-injector" created
 $ kubectl label namespace default istio-injection=enabled
namespace "default" labeled
 $ kubectl get pods -n istio-system --watch
...
snip
...
 $ wget -O- https://storage.googleapis.com/knative-releases/latest/release.yaml > knative.yml
--2018-07-12 14:08:57--  https://storage.googleapis.com/knative-releases/latest/release.yaml
Resolving storage.googleapis.com... 216.58.194.176, 2607:f8b0:4005:804::2010
Connecting to storage.googleapis.com|216.58.194.176|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 493176 (482K) [application/octet-stream]
Saving to: 'STDOUT'

100%[==================================================================================================================>] 493,176     --.-K/s   in 0.03s

2018-07-12 14:08:57 (17.2 MB/s) - written to stdout [493176/493176]

 $ kubectl apply -f knative.yml
namespace "knative-build" created
clusterrole "knative-build-admin" created
serviceaccount "build-controller" created
clusterrolebinding "build-controller-admin" created
customresourcedefinition "builds.build.knative.dev" created
customresourcedefinition "buildtemplates.build.knative.dev" created
service "build-controller" created
service "build-webhook" created
configmap "config-logging" created
deployment "build-controller" created
deployment "build-webhook" created
namespace "knative-serving" created
clusterrole "knative-serving-admin" created
clusterrole "knative-serving-write" created
serviceaccount "controller" created
serviceaccount "autoscaler" created
clusterrolebinding "knative-serving-controller-admin" created
clusterrolebinding "knative-serving-autoscaler-write" created
service "knative-ingressgateway" created
deployment "knative-ingressgateway" created
horizontalpodautoscaler "knative-ingressgateway" created
customresourcedefinition "configurations.serving.knative.dev" created
customresourcedefinition "revisions.serving.knative.dev" created
customresourcedefinition "routes.serving.knative.dev" created
customresourcedefinition "services.serving.knative.dev" created
service "activator-service" created
service "controller" created
service "webhook" created
deployment "activator" created
configmap "config-autoscaler" created
configmap "config-controller" created
configmap "config-domain" created
configmap "config-logging" created
configmap "config-network" created
configmap "config-observability" created
deployment "controller" created
deployment "webhook" created
namespace "monitoring" created
configmap "fluentd-ds-config" created
configmap "scaling-config" created
configmap "grafana-dashboard-definition-istio" created
configmap "grafana-dashboard-definition-mixer" created
configmap "grafana-dashboard-definition-pilot" created
serviceaccount "fluentd-ds" created
clusterrole "fluentd-ds" created
clusterrolebinding "fluentd-ds" created
daemonset "fluentd-ds" created
serviceaccount "kube-state-metrics" created
role "kube-state-metrics-resizer" created
rolebinding "kube-state-metrics" created
clusterrole "kube-state-metrics" created
clusterrolebinding "kube-state-metrics" created
deployment "kube-state-metrics" created
service "kube-state-metrics" created
configmap "grafana-dashboard-definition-kubernetes-deployment" created
configmap "grafana-dashboard-definition-kubernetes-capacity-planning" created
configmap "grafana-dashboard-definition-kubernetes-cluster-health" created
configmap "grafana-dashboard-definition-kubernetes-cluster-status" created
configmap "grafana-dashboard-definition-kubernetes-control-plane-status" created
configmap "grafana-dashboard-definition-kubernetes-resource-requests" created
configmap "grafana-dashboard-definition-kubernetes-nodes" created
configmap "grafana-dashboard-definition-kubernetes-pods" created
configmap "grafana-dashboard-definition-kubernetes-statefulset" created
serviceaccount "node-exporter" created
clusterrole "node-exporter" created
clusterrolebinding "node-exporter" created
daemonset "node-exporter" created
service "node-exporter" created
service "elasticsearch-logging" created
serviceaccount "elasticsearch-logging" created
clusterrole "elasticsearch-logging" created
clusterrolebinding "elasticsearch-logging" created
statefulset "elasticsearch-logging" created
deployment "kibana-logging" created
service "kibana-logging" created
service "fluentd-ds" created
configmap "grafana-dashboard-definition-knative-efficiency" created
configmap "grafana-dashboard-definition-knative" created
configmap "grafana-datasources" created
configmap "grafana-dashboards" created
service "grafana" created
deployment "grafana" created
logentry "requestlog" created
fluentd "requestloghandler" created
rule "requestlogtofluentd" created
metric "revisionrequestcount" created
metric "revisionrequestduration" created
metric "revisionrequestsize" created
metric "revisionresponsesize" created
prometheus "revisionpromhandler" created
rule "revisionpromhttp" created
service "zipkin" created
deployment "zipkin" created
service "kube-controller-manager" created
service "prometheus-system-discovery" created
configmap "prometheus-scrape-config" created
serviceaccount "prometheus-system" created
role "prometheus-system" created
role "prometheus-system" created
role "prometheus-system" created
role "prometheus-system" created
clusterrole "prometheus-system" created
rolebinding "prometheus-system" created
rolebinding "prometheus-system" created
rolebinding "prometheus-system" created
rolebinding "prometheus-system" created
clusterrolebinding "prometheus-system" created
service "prometheus-system-np" created
statefulset "prometheus-system" created
logentry "requestlog" configured
fluentd "requestloghandler" configured
rule "requestlogtofluentd" configured
metric "revisionrequestcount" configured
metric "revisionrequestduration" configured
metric "revisionrequestsize" configured
metric "revisionresponsesize" configured
prometheus "revisionpromhandler" configured
rule "revisionpromhttp" configured
Error from server (NotFound): error when creating "knative.yml": the server could not find the requested resource (post gatewaies.networking.istio.io)

secret gist: https://gist.github.com/maximilien/ba103bcd67f4d8f733d654cf38928454 with contents of istio.yml and knative.yml

[tcnghia]

@maximilien I suspect this is due to older version of kubectl? I hope you can try this again with a newer version of kubectl?

My theory is that older kubectl may have interpreted our knative-shared-gateway object to have the type gatewaies.networking.istio.io because it uses convention instead of reading from the CRD defined in istio.yml.

[tcnghia]

I found this bug https://github.com/kubernetes/apiextensions-apiserver/issues/12 . This PR https://github.com/kubernetes/kubernetes/pull/50012 fixes the bug by reading from CRD instead of using convention. Based on comments in https://github.com/kubernetes/apiextensions-apiserver/issues/12 I think you'll need at least kubectl 1.8 or 1.9.

Please let me know how it goes after upgrading?

[cppforlife]

it does work with new CLI. cc @maximilien

thank you @tcnghia for taking a look.

[Turbots]

I had the same problem so I also upgraded my CLI but it still doesn't work:

Minikube v0.28.2 configured with 8GB RAM and 4 CPUs (OSX)
Kubectl v1.11.2
Riff CLI v0.1.1

following the tutorial at https://projectriff.io/docs/getting-started-with-knative-riff-on-minikube/

edit: I apparently was using Google Cloud SDK version of kubectl and not the brew version... I was still using 1.7.6 which still has the bug. I updated Google Cloud SDK components using gcloud components update and am now using kubectl v1.9.7 and it works!

[davidmehi]

I also had this problem: the server could not find the requested resource (post gatewaies.networking.istio.io)

Even though I installed everything correctly, my path was still pointing to an old version of gcloud sdk, so my K8S client and server versions did not match. If "kubectl version" shows that the client and server versions are different, then upgrade one of them to match.