Serving: Latest Knative serving master branch doesn't work well with Istio version 1.4.x

[zhangtbj]

Hi all,
Does anyone try to install the latest master branch Knative serving with Istio 1.3.8 or 1.4.x ?

The Istio 1.2.2 works fine with the latest master branch of Knative serving, but when I switched the Istio to 1.3.8 or 1.4.x, all created ksvc became IngressNotConfigured status:

config-map-volume-jtcmrwak               http://config-map-volume-jtcmrwak.serv
ing-tests.example.com               config-map-volume-jtcmrwak-4h7bm                 config-map-volume-jtcmrwak-4h7bm                 Unknown   IngressNotConfigured
projected-config-map-volume-hgviclqr     http://projected-config-map-volume-hgviclqr.serving-tests.example.com     projected-config-map-volume-hgviclqr-hjs2k       projected-config-map-volume-hgviclqr-hjs2k       Unknown   IngressNotConfigured
projected-secret-volume-apgapaie         http://projected-secret-volume-apgapaie.serving-tests.example.com         projected-secret-volume-apgapaie-pr45n           projected-secret-volume-apgapaie-pr45n           Unknown   IngressNotConfigured

But I can still access them well and the pods are running correctly:

✔ $ kubectl get ksvc
NAME   URL                                                                                                                  LATESTCREATED   LATESTREADY    READY     REASON
echo   http://echo-default.coligoserving-193911-f891c22e52f4c0e385707f50c191a2c4-0000.us-south.containers.appdomain.cloud   echo-cgdrx-1    echo-cgdrx-1   Unknown   IngressNotConfigured

✔ $ curl http://echo-default.coligoserving-193911-f891c22e52f4c0e385707f50c191a2c4-0000.us-south.containers.appdomain.cloud
Hello World!

I tried on both kube 1.15 and 1.16, the failures are the same.

I didin't find any useful debug info, this is the ksvc status:

Status:
  Address:
    URL:  http://echo.default.svc.cluster.local
  Conditions:
    Last Transition Time:        2020-02-25T10:12:57Z
    Status:                      True
    Type:                        ConfigurationsReady
    Last Transition Time:        2020-02-25T10:12:58Z
    Message:                     Ingress has not yet been reconciled.
    Reason:                      IngressNotConfigured
    Status:                      Unknown
    Type:                        Ready
    Last Transition Time:        2020-02-25T10:12:58Z
    Message:                     Ingress has not yet been reconciled.
    Reason:                      IngressNotConfigured
    Status:                      Unknown
    Type:                        RoutesReady

If I change back to Istio 1.2.2 or the change back to the official knative serving v0.12.1 release. it works!!!

So I don't know if there is any change in the latest master branch, Does anyone can help me?
Thanks a lot!!

[nak3]

Hi @zhangtbj , Could you please check the networking-istio pod logs in knative-serving namespace? You should find the logs something like these:

"msg": "Processing probe for http://<YOUR_DOMAIN>:80/, ..."
"msg": "Probing of http://<YOUR_DOMAIN>:80/ failed, SOME REASON ..."

That would be the initial step for the troubleshooting.
And if you have any istio's policy configuration like mTLS, JWT auth and so on, please let us know.

[zhangtbj]

Hi @nak3 ,

Thanks for your help!

I got many same errors in the logs of the networking-istio pod:

{"level":"info","ts":"2020-02-25T13:55:47.566Z","logger":"istiocontroller.status-manager","caller":"status/status.go:342","msg":"Processing probe for http://echo.default.svc.cluster.local:80, IP: 172.30.206.185:80 (depth: 0)","commit":"357e042"}
{"level":"error","ts":"2020-02-25T13:55:47.567Z","logger":"istiocontroller.status-manager","caller":"status/status.go:379","msg":"Probing of http://echo.default.svc.cluster.local:80 failed, IP: 172.30.206.185:80, ready: false, error: error roundtripping http://echo.default.svc.cluster.local:80: dial tcp 172.30.206.185:80: connect: connection refused (depth: 0)","commit":"357e042","stacktrace":"knative.dev/serving/pkg/network/status.(*Prober).processWorkItem\n\tknative.dev/serving/pkg/network/status/status.go:379\nknative.dev/serving/pkg/network/status.(*Prober).Start.func1\n\tknative.dev/serving/pkg/network/status/status.go:276"}

I tried to access this echo.default.svc.cluster.local in one of my pod, it fails:

root@helloworld-v1-6757db4ff5-v5wgc:/opt/microservices# curl http://echo.default.svc.cluster.local:80
curl: (6) Could not resolve host: echo.default.svc.cluster.local
root@helloworld-v1-6757db4ff5-v5wgc:/opt/microservices# ping echo.default.svc.cluster.local
ping: unknown host

And my svc status is:

kubectl get svc
NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP                         PORT(S)                             AGE
details                       ClusterIP      172.21.221.53    <none>                              9080/TCP                            35h
echo                          ExternalName   <none>           echo.default.example.com            <none>                              3h59m
echo-wfsnp-1                  ClusterIP      172.21.195.128   <none>                              80/TCP                              3h59m
echo-wfsnp-1-private          ClusterIP      172.21.201.25    <none>                              80/TCP,9090/TCP,9091/TCP,8022/TCP   3h59m
helloworld                    ClusterIP      172.21.152.113   <none>                              5000/TCP                            36h

But it works when I install Knative serving with older Istio version 1.2.2.

Thanks a lot!

[duglin]

Using the latest cluster-local stuff from: https://github.com/knative/serving/blob/master/third_party/istio-1.4.4/istio-knative-extras.yaml seems to work - so you may need to upgrade your install to use this version.

[zhangtbj]

Thanks Doug's help!!!!

After change to the correct cluster-local yaml from:
https://travis.ibm.com/coligo/knative-serving/builds/26774289

It works now!

So next time, if we want to upgrade the Istio version, we need to watch the folder:
https://github.com/knative/serving/tree/master/third_party

To see which Istio version the community support.

And this is the script to generate the cluster-local yaml:
https://github.com/knative/serving/blob/master/third_party/istio-1.4.4/download-istio.sh

[tcnghia]

@zhangtbj in Istio 1.5 cluster local gateways can be installed using istioctl, so things will get a bit less confusing.

[zhangtbj]

Hi @tcnghia ,
Thanks for the info.

I installed the Istio by using helm.

About the Istio 1.5, is that refined for helm way? Or do you prefer to install Istio by using istioctl in future?

[nak3]

Hi @zhangtbj We prefer to install istio by using istioctl in the future, though our branch may still use Helm on Istio 1.5. It depends on https://github.com/knative/serving/issues/6100.
Istio upstream decided to make Helm's installation deprecated so we have to do that anyway :sweat_smile:

https://preliminary.istio.io/docs/setup/install/helm/

The Helm installation approach will be deprecated in the future. We recommend Installing with istioctl, instead.

[zhangtbj]

What a big change for us.....

Thanks for the info @nak3 !

One more question, Do you know when the Istio 1.5 will be published?

And will the Istio upgrade break the Knative serving function during the upgrade?
I am not sure if you have this kind of test, such as upgrade Istio from 1.3.x to 1.4.x :)

Thanks!

[nak3]

We can find the release date of Istio 1.5 here - https://github.com/istio/istio/wiki/Istio-Release-1.5
And we will test serving w/ Istio 1.5 to avoid breaking the function as much as possible once 1.5 is released.

[zhangtbj]

Oh cool! Thanks again! @nak3

[techBeck03]

We can find the release date of Istio 1.5 here - https://github.com/istio/istio/wiki/Istio-Release-1.5
And we will test serving w/ Istio 1.5 to avoid breaking the function as much as possible once 1.5 is released.

Now that 1.5 is released, is there any guidance you can provide on getting Knative to work with it? I'm still not able to get the cluster-local-gateway configured properly using istioctl (or helm). Any help is much appreciated!!