Serving: Error installing Istio behind proxy

[palade]

In what area(s)?

/area build

What version of Knative?

HEAD

Expected Behavior

all the required KNATIVE components installed

Actual Behavior

Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/serving.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/serving.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/serving.yaml": Internal error occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/monitoring.yaml": Internal error occurred: failed calling webhook "mixer.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/monitoring.yaml": Internal error occurred: failed calling webhook "mixer.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/monitoring.yaml": Internal error occurred: failed calling webhook "mixer.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/monitoring.yaml": Internal error occurred: failed calling webhook "mixer.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/monitoring.yaml": Internal error occurred: failed calling webhook "mixer.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/monitoring.yaml": Internal error occurred: failed calling webhook "mixer.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/monitoring.yaml": Internal error occurred: failed calling webhook "mixer.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/monitoring.yaml": Internal error occurred: failed calling webhook "mixer.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Service Unavailable
Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/monitoring.yaml": Internal error occurred: failed calling webhook "mixer.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitmixer?timeout=30s: Service Unavailable
ubuntu@ubuntuMaster:~$ Error from server (InternalError): error when creating "https://github.com/knative/serving/releases/download/v0.3.0/serving.yaml": Internal erro r occurred: failed calling webhook "pilot.validation.istio.io": Post https://istio-galley.istio-system.svc:443/admitpilot?timeout=30s: Service Unavailable

Steps to Reproduce the Problem

1. Create a 3 node Kubernetes cluster: one master and two worker nodes.
2. Follow the steps from https://github.com/knative/docs/blob/master/install/Knative-with-any-k8s.md

[markusthoemmes]

Do you mind tuning down the title of this issue to something shorter please?

Your errors look like istio hasn't come up properly. Have you waited for all pods of istio to be started before you applied serving.yml?

[palade]

Do you mind tuning down the title of this issue to something shorter please?

Your errors look like istio hasn't come up properly. Have you waited for all pods of istio to be started before you applied serving.yml?

Yes I followed the instructions from the document

NAME                                       READY   STATUS      RESTARTS   AGE
cluster-local-gateway-547467ccf6-slbft     1/1     Running     0          54m
istio-citadel-7d64db8bcf-gz4rr             1/1     Running     0          54m
istio-cleanup-secrets-mz5qk                0/1     Completed   0          54m
istio-egressgateway-6ddf4c8bd6-t4skd       1/1     Running     0          54m
istio-galley-7dd996474-rszjq               1/1     Running     0          54m
istio-ingressgateway-84b89d647f-ttxmj      1/1     Running     0          54m
istio-pilot-86bb4fcbbd-2jc5l               2/2     Running     0          13m
istio-pilot-86bb4fcbbd-4slxv               2/2     Running     0          13m
istio-pilot-86bb4fcbbd-b7xt7               2/2     Running     0          53m
istio-policy-5c4d9ff96b-598rn              2/2     Running     0          54m
istio-sidecar-injector-6977b5cf5b-dktx4    1/1     Running     0          54m
istio-statsd-prom-bridge-b44b96d7b-qj4p7   1/1     Running     0          54m
istio-telemetry-7676df547f-l72vt           2/2     Running     0          54m
knative-ingressgateway-75644679c7-z6s57    1/1     Running     0          51m
zipkin-6dbbcbf9c8-t2jmc                    1/1     Running     0          50m

[mattmoor]

Can you share more information about the cluster? What K8s version? Where is it running?

[palade]

kubectl version

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

kubectl get nodes -o wide

NAME           STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
ubuntumaster   Ready    master   23h   v1.13.3   10.63.0.19    <none>        Ubuntu 16.04.5 LTS   4.4.0-142-generic   docker://18.6.1
ubuntuslave1   Ready    <none>   22h   v1.13.3   10.63.0.11    <none>        Ubuntu 16.04.5 LTS   4.4.0-142-generic   docker://18.6.1
ubuntuslave2   Ready    <none>   22h   v1.13.3   10.63.0.15    <none>        Ubuntu 16.04.5 LTS   4.4.0-142-generic   docker://18.6.1

The cluster is formed using VMs and is running on a local cloud managed through OpenNebula. Each VM is running Ubuntu 16.04.

[vvraskin]

Could it be that the kube control plane is broken?
It seems that the requests are not being forwarded to the istio webhooks, do you see any failing pods in kube control plane, e.g. kubectl get pods -n kube-system?

[palade]

kubectl get pods -n kube-system

NAME                                   READY   STATUS    RESTARTS   AGE
coredns-86c58d9df4-jwqdc               1/1     Running   0          74m
coredns-86c58d9df4-p78l2               1/1     Running   0          74m
etcd-ubuntumaster                      1/1     Running   0          73m
kube-apiserver-ubuntumaster            1/1     Running   0          73m
kube-controller-manager-ubuntumaster   1/1     Running   0          74m
kube-flannel-ds-amd64-gwzss            1/1     Running   0          72m
kube-flannel-ds-amd64-js44n            1/1     Running   0          74m
kube-flannel-ds-amd64-lmpl4            1/1     Running   0          74m
kube-proxy-5tx4n                       1/1     Running   0          74m
kube-proxy-8q5lv                       1/1     Running   0          72m
kube-proxy-ns6pj                       1/1     Running   0          74m
kube-scheduler-ubuntumaster            1/1     Running   0          74m

The nodes of the cluster are behind a proxy. I managed to setup the Kubernetes cluster behind the proxy. So if I run, kubectl get nodes I can see all the nodes. Also, I can run docker pull commands.

NAME           STATUS   ROLES    AGE    VERSION
ubuntumaster   Ready    master   108m   v1.13.3
ubuntuslave1   Ready    <none>   107m   v1.13.3
ubuntuslave2   Ready    <none>   105m   v1.13.3

I also added the following addresses under the CLUSTER-IP column to the no_proxy field in both /etc/environment and docker.service

kubectl get services --namespace knative-serving -o wide
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE    SELECTOR
activator-service   ClusterIP   10.105.174.189   <none>        80/TCP,81/TCP,9090/TCP   105m   app=activator
autoscaler          ClusterIP   10.109.71.97     <none>        8080/TCP,9090/TCP        105m   app=autoscaler
controller          ClusterIP   10.104.248.105   <none>        9090/TCP                 105m   app=controller
webhook             ClusterIP   10.107.107.169   <none>        443/TCP                  105m   role=webhook

I think the problem is caused by the proxy. Are there are any other ip addresses that should be in the no_proxy field? I will try it on a separate cluster and come back.

[palade]

Installed Istio on a cluster which is not behind a proxy and works as expected. I believe this error/situation is happening when trying to install Istio on a cluster behind a proxy

[skeeey]

I have the same problem, as @palade said, after I removed the proxy env from kube-apiserver configuration, it works

[tcnghia]

@palade @skeeey is this error specific to Istio or the proxy is breaking validation webhooks in general?

[skeeey]

@tcnghia I think this error is from Istio, when I deploy the knative serving, the Istio injects the sidecar for it, in this phase, the Istio use the galley to validate its configuration, because there is http_proxy setting in my environment, the validation webooks are failed

[palade]

@tcnghia Same as @skeeey I believe what the quickest solution is to add all the ip addresses which are used to no_proxy but I haven't checked this solution yet.

[skeeey]

@palade, I tried the no_proxy, but I used the CIDR not list all of the IP addresses, it does not work

[pedroarthur]

@tcnghia Same as @skeeey I believe what the quickest solution is to add all the ip addresses which are used to no_proxy but I haven't checked this solution yet.

It won't solve it. Just tried that.

[piotrekfilip]

Another workaround was recently added to Istio documentation , try adding .svc to no_proxy.

another workaround is to include istio-sidecar-injector.istio-system.svc or .svc in the no_proxy value. Make sure that kube-apiserver is restarted after each workaround.

[tvalasek]

thats correct - we fixed it by adding .istio-system.svc to no_proxy and NO_PROXY api-server's env variable

[tcnghia]

we no longer ship an Istio yaml and leave Istio installation to the users. Our approach is that we will test Knative with various ways of setting up Istio to have good enough coverage (different Istio versions, whether sidecar is injected or now) to enable the users to choose their own Istio installation.

If there are Istio installation challenge please follow up at https://github.com/istio/istio .