Kind: Installing Istio fails when trying to match resources

[nimakaviani]

What happened:

I wonder if anyone has tried installing Istio on kind. I get the following errors when trying to install Istio 1.1.11:

deployment.extensions/istio-citadel created
deployment.extensions/istio-sidecar-injector created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
horizontalpodautoscaler.autoscaling/istio-policy created
horizontalpodautoscaler.autoscaling/istio-telemetry created
horizontalpodautoscaler.autoscaling/istio-pilot created
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector created
unable to recognize "STDIN": no matches for kind "attributemanifest" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "attributemanifest" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "metric" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "handler" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "handler" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "rule" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "kubernetes" in version "config.istio.io/v1alpha2"
unable to recognize "STDIN": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"
unable to recognize "STDIN": no matches for kind "DestinationRule" in version "networking.istio.io/v1alpha3"

Looking at this issue https://github.com/knative/serving/issues/2195 it appears to be due to a bug with the the earlier micro versions of K8s 1.11+.

How to reproduce it (as minimally and precisely as possible):

wget -O- https://github.com/istio/istio/releases/download/1.1.11/istio-1.1.11-linux.tar.gz > istio.tgz
tar -zxvf istio.tgz && cd istio-1.1.11
kubectl create namespace istio-system
helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -
helm template install/kubernetes/helm/istio --name istio --namespace istio-system | kubectl apply -f -

Anything else we need to know?:

Environment:

• kind version: (use kind version): 0.4.0
• Docker version: (use docker info): 18.09
• OS (e.g. from /etc/os-release): xenial

[BenTheElder]

kind by default runs Kubernetes v1.15.x at the moment

~What is your kubectl version?~ not relevant: see https://github.com/kubernetes-sigs/kind/issues/762#issuecomment-519792739

Have you looked at https://kind.sigs.k8s.io/docs/user/known-issues/ ?

[BenTheElder]

I don't think this is a bug in kind, I'm pretty sure this is an issue with how istio is being installed.

[BenTheElder]

Did you skip a step installing CRDs before installing the rest? Those errors relate to those resource types not existing yet. Since they are istio specific a normal Kubernetes cluster generally won't ship them out of the box.

[nimakaviani]

No, this snippet installs everything istio needs in any other kubernetes cluster I tried.

wget -O- https://github.com/istio/istio/releases/download/1.1.11/istio-1.1.11-linux.tar.gz > istio.tgz
tar -zxvf istio.tgz && cd istio-1.1.11
kubectl create namespace istio-system
helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -
helm template install/kubernetes/helm/istio --name istio --namespace istio-system | kubectl apply -f -

It is not with Istio only that I see the error, tried Knative too and similar error messages pop up. Only with kind as the cluster.

Here is the command:

kubectl  apply \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving-post-1.14.yaml \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving.yaml \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving-beta-crds.yaml \
   --filename https://github.com/knative/build/releases/download/v0.7.0/build.yaml

similar errors:

unable to recognize "https://github.com/knative/serving/releases/download/v0.7.1/serving-post-1.14.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/serving/releases/download/v0.7.1/serving.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/build/releases/download/v0.7.0/build.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/build/releases/download/v0.7.0/build.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/build/releases/download/v0.7.0/build.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"
unable to recognize "https://github.com/knative/build/releases/download/v0.7.0/build.yaml": no matches for kind "Image" in version "caching.internal.knative.dev/v1alpha1"

[nimakaviani]

aside from the CRD issue, I wonder if anyone has tried installing Istio on Kind to check whether or not it works.

[BenTheElder]

aside from the CRD issue, I wonder if anyone has tried installing Istio on Kind to check whether or not it works.

The istio project has CI running on kind..

[BenTheElder]

No, this snippet installs everything istio needs in any other kubernetes cluster I tried

What other clusters did you try?

Have you checked our known issues guide, is your kubectl too far off from the server version?

[nimakaviani]

what I noticed is that installing the CRDs fails first time around but if I try to redeploy after a little while it will successfully deploy. Basically similar to this comment https://github.com/knative/serving/issues/2195#issuecomment-448648434 from the issue I linked earlier.

I have tried GKE and IKS with the above installation instructions and I get Istio 1.1.11 without any problem.

my kubectl is at v 1.14 so I doubt there is much regression between the version of the server and the client.

[BenTheElder]

This part:

kubectl  apply \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving-post-1.14.yaml \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving.yaml \
   --filename https://github.com/knative/serving/releases/download/v0.7.1/serving-beta-crds.yaml \
   --filename https://github.com/knative/build/releases/download/v0.7.0/build.yaml

Is both installing CRDs and installing CRs based on them, simultaneously. This is a bad idea, and a race.

[BenTheElder]

Installing istio from their latest docs works fine, which involves installing the CRDs seperately first:
https://istio.io/docs/setup/kubernetes/#downloading-the-release
then:
https://istio.io/docs/setup/kubernetes/install/kubernetes/

curl -L https://git.io/getLatestIstio | ISTIO_VERSION=1.2.2 sh -
cd istio-1.2.2
for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
kubectl apply -f install/kubernetes/istio-demo.yaml

[BenTheElder]

I then followed this with https://knative.dev/docs/install/knative-with-any-k8s/ which says:

To install Knative, first install the CRDs by running the kubectl apply command once with the -l knative.dev/crd-install=true flag. This prevents race conditions during the install, which cause intermittent errors:

And this worked fine (on the same kind cluster).

This is a problem with their install process. They should ensure that the CRDs are ready to serve before creating CRs from them.

The workaround in their docs does however seem to work.

[nimakaviani]

~Right so the installation instructions on Knative probably need to split the steps for installing the CRs and the CRDs too~

~https://knative.dev/docs/install/Knative-with-any-k8s~

_Update: Knative docs actually has the instructions to use the crd selector._

But more importantly, the race issue seems to have been addressed before (see https://github.com/kubernetes/kubernetes/issues/62725 and the corresponding PR), so not sure if this is still a problem in the core of kubernetes. Also following the same install instructions, I dont seem to be able to reproduce this kind of race in any cluster other than Kind.

That's why I wondered maybe Kind has an older version of Kube which like you said does not seem to be the case. so yeah I don't know 🤷‍♂

[BenTheElder]

But more importantly, the race issue seems to have been addressed before (see kubernetes/kubernetes#62725 and the corresponding PR), so not sure if this is still a problem in the core of kubernetes.

kubernetes/kubernetes#62725 is about solving the race when you create one shortly after, the problem here is _simultaneously_ creating them.

Think of a CRD (= CustomResourceDefinition) like a table schema in a relational database, and a CR (= CustomResource instance) as an entry in the matching table.

Creating both CRDs and CRs at the same time is then like creating tables while also trying to insert entries into them. Each table needs to be created first, then you can insert.

With the current installation instructions for both Istio and Knative, you do create the CRDs before the CRs, and this works fine.

The old installation instructions were wrong. Trying to create CRs before the CRDs are serving from the API server is a bug in the API usage, it is possible to check that CRDs exist in the API before proceeding (or simply a pause between instructions will generally happen to work).

Also following the same install instructions, I dont seem to be able to reproduce this kind of race in any cluster other than Kind.

On some clusters it may be that you win this race. kind runs on a "noisy shared host" I.E. your development machine with less isolation / dedicated resources.

[rjshrjndrn]

@nimakaviani I created a local rancher, and install istio with that. Works fine for me.

[nimakaviani]

@rjshrjndrn which installation instructions did you use? The one I referenced or @BenTheElder ‘s or something else?

[rjshrjndrn]

@nimakaviani
I didn't tried the helm installation.

1. Create cluster with Kind
2. Create rancher container and imported the cluster
3. From the app catalog, installed istio

[BenTheElder]

Going to close this since the current istio / knative docs do not have this issue. If you have more questions please re-open or file a new issue 😅