Kfserving: IngressNotConfigured with kubeflow master

[ryandawsonuk]

I installed kubeflow using something close to kfctl_istio_dex.yaml. So I get istio 1.1 rather than 1.3 as I can't currently run 1.3 with DigitalOcean. I've been using this setup successfully before but with the latest manifests coming from kubeflow master I now find the kservice never becomes available and the reason is given as IngressNotConfigured

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| bug | 0.57 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[yuzisun]

@ryandawsonuk that's known issue and we created one upstream https://github.com/knative/serving/issues/6793

[yuzisun]

The issue is that knative serving sends probes to istio ingress gateway but got 302 when redirecting to AuthService for Dex, we are looking into how to whitelist all these probes.

[ryandawsonuk]

Cool, thanks for the quick response

[krishnadurai]

@ryandawsonuk we are looking to provide a way to whitelist as @yuzisun has mentioned.

/assign

[ryandawsonuk]

@krishnadurai have you tried using SKIP_AUTH_URI in the auth service or speaking to @yanniszark ? Or is the part of the problem even knowing what to whitelist?

[krishnadurai]

@ryandawsonuk The requests go to the path '/'. Whitelisting the path '/' isn't the right idea.
The requests are proxied by the istio-ingressgateway using the 'Host' (virtual host) parameter, which contains the KNative service's domain, for e.g. 'xyz-sample.default.example.com'.

I'm looking for the right method to whitelist such requests from KNative's prober service.

/cc @yanniszark

[yanniszark]

Hi @ryandawsonuk @krishnadurai.
Maybe I'm missing something here, but why do the KNative probes need to go through the Istio Gateway?
What is the KNative probe and what does it probe?
Aren't both the probe and the probed deployment in the same cluster?

[krishnadurai]

@yanniszark fair point. I had the same questions initially. The probe does go through the istio-ingressgateway though. Here are the istio-ingressgateway pod logs for helloworld.go sample KNative service running with the Dex based EnvoyFilter:

[2020-02-11T18:16:21.419Z] "GET / HTTP/1.1" 404 NR "-" "-" 0 0 0 - "10.88.0.31" "Go-http-client/1.1" "4a8bd584-2323-4f40-9230-9797d890b9fb" "helloworld-go.default:80" "-" - - 10.88.1.13:80 10.88.0.31:36237 - -
[2020-02-11T18:16:21.419Z] "GET / HTTP/1.1" 404 NR "-" "-" 0 0 0 - "10.88.0.31" "Go-http-client/1.1" "7298dbfc-58bb-430f-92c5-cf39e97f63d7" "helloworld-go.default.svc:80" "-" - - 10.88.1.13:80 10.88.0.31:36239 - -
[2020-02-11T18:16:21.420Z] "GET / HTTP/1.1" 302 UAEX "-" "-" 0 269 21 21 "10.88.0.31" "Go-http-client/1.1" "27aa43fa-ac17-4a71-8ca2-b4d9fb772219" "helloworld-go.default.example.com:80" "-" - - 10.88.1.13:80 10.88.0.31:36249 - -

[krishnadurai]

What is the KNative probe and what does it probe?

The pod networking-istio in Knative serving probes the Istio gateway from outside of the service mesh to see if ingress is available for the Knative service.

Looking at the comments here:
https://github.com/knative/serving/blob/master/config/istio-ingress/controller.yaml

Aren't both the probe and the probed deployment in the same cluster?

Yes they are in the same cluster

/cc @yuzisun

[yanniszark]

@yuzisun @krishnadurai can you please elaborate on why the probe needs to hit the gateway and not the deployment directly, to bypass the need for authentication?
How does this work on GCP?
Is the probe given an IAM serviceaccount?

/cc @jlewi

[yuzisun]

@yanniszark as @krishnadurai said, the probes hit the gateway to test if the ingress(virtual service) is available for the knative services. GCP has similar problem, the probes are failing with 401, we can whitelist with excludePaths on the ingress jwt authentication policy but whitelisting the path '/' isn't good idea.

[yanniszark]

@yuzisun thanks a lot for the explanation. Then, my next questions are:

• Is this prober deployment necessary for KFServing to work? Is KFServing usable without it?
• What is the desired solution? Giving the KFServing probe credentials? Whitelisting the health check paths?

[animeshsingh]

For further tests, lets all test on this configuration
https://github.com/kubeflow/manifests/pull/886

[yanniszark]

My understanding is that because of this issue, KFServing can't function in auth-enabled Kubeflow deployments.
@jlewi do you believe this blocking for Kubeflow v1.0?

[krishnadurai]

@yanniszark when we look at KFServing 0.11.1 and KFServing 0.2.1 combination with either of Istio's configs (1.16 or 1.3.1), you are right.

Whereas v1.0 Kubeflow has KFServing v0.8.0 which shipped with v0.7 Kubeflow.

[JRBANCEL]

Hi, I implemented the probing in Knative.

Is this prober deployment necessary for KFServing to work? Is KFServing usable without it?

No, but without it, it is fire and forget when Knative configure Istio, so a Knative Service CRD being ready might not actually be ready if all the Envoy do not have the latest config. That's what probing solves. Currently, it cannot be disabled.

What is the desired solution? Giving the KFServing probe credentials? Whitelisting the health check paths?

The cleanest solution is probably to allow probing calls without auth (from inside the cluster only would be safer).
Probing calls do not reach the user container, so there is no risk of them actually doing anything.
Filtering for probing calls is implemented here: https://github.com/knative/serving/blob/master/pkg/network/probe_handler.go

[yuzisun]

see https://github.com/knative/serving/issues/6829 also.

• In GCP IAP setup Istio policy can block the probes and there is no way to filter probes on header content.
• For Istio-Dex setup, not sure if there is a way to bypass auth for probes in envoy filter.

[krishnadurai]

@JRBANCEL @yuzisun

* For Istio-Dex setup, not sure if there is a way to bypass auth for probes in envoy filter.

Any method to bypass will require the authservice envoy-filter for Dex to read the origin of the request. It is not secure to whitelist requests based on the origin.

We might want to consider using another gateway for KFServing for the time being until knative/serving#6829 is resolved.

This comes with the downsides of having another gateway and as well as not supporting authenticated requests for serving models.

[krishnadurai]

@JRBANCEL as an alternative is it a good idea to have the prober supplied with programmatic credentials for these requests? This approach will not need whitelisting and keep the gateway access secure.

What is your take on this?

[JRBANCEL]

How would you provide the credentials?

[krishnadurai]

By mounting secrets and allowing to specify them in headers to the probing requests in the istio-networking pod.

[Jude188]

Any news or progress on this one for Istio 1.3 with Kubeflow 1.0 out?

[krishnadurai]

@Jude188
I'm looking to solve this issue for kfctl_istio_dex.yaml which uses 1.3.1 istio.

Though it seems downgrading istio to 1.1.6 in kfctl_istio_dex.yaml and using this PR will help:
https://github.com/kubeflow/manifests/pull/949

You could easily change Istio manifests to 1.1.6 by changing the entries in kfdef applications: istio-crds and istio-install.
Change:

• istio-crds-1-3-1 in path to istio-crds
• istio-install-1-3-1 in path to istio-install

[krishnadurai]

@ryandawsonuk you could use the PR mentioned kubeflow/manifests#949 to enable your setup in Digital Ocean.

[ryandawsonuk]

Hi @krishnadurai sorry it has taken me so long to test. I've just taken the latest master, which is at https://github.com/kubeflow/manifests/commit/9ec6bb39e9bb76d7a9bdef290628a6c6f36952ce and includes https://github.com/kubeflow/manifests/pull/949. My InferenceServices do now start. But I'm not clear on how I now make calls to them as they're no longer reachable on the external IP given by kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'. Just get 404s for that now. Is there a different way I should be making the calls?

[krishnadurai]

@ryandawsonuk
The gateway you should be looking for is kfserving-ingressgateway instead of istio-ingressgateway. I'll update the docs as soon as the patch is released.

The IP should be retrieved by:

kubectl -n istio-system get service kfserving-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

[ryandawsonuk]

Hmm, for me that gives Error from server (NotFound): services "kfserving-ingressgateway" not found. I am using a kfdef based on istio_dex. Perhaps it is missing something. I'll have to compare against the manifests repo as that deployment isn't coming through for me.

[krishnadurai]

@ryandawsonuk thanks for sharing your manifests and kfdef.

From what I see, the component istio/kfserving-gateway is missing in your kfdef:

Please refer to the following diff where I've patched it to some other kfdefs.
https://github.com/kubeflow/manifests/pull/949/files#diff-3246bb7e71d05445ff82aeff687b3d34R31-R38

[ryandawsonuk]

@krishnadurai I don't see istio/kfserving-gateway in https://github.com/kubeflow/manifests/blob/6abf161dc352018bd5be22d5394120013d7e54e3/kfdef/kfctl_istio_dex.yaml. Looks like it is missing for the 1.3.1 configurations. I've tried adding in istio/kfserving-gateway to my kfdef but that doesn't get deployed, presumably because I'm using 1.3.1.

[ryandawsonuk]

I've now switched to a 1.1.6 version of istio with the istio_dex variant kfdef.

Now it deploys the kfserving-gateway if I do a kfctl apply -f on that file in github. My usual workflow though is to first do a kfctl build and then an apply on a path to a local config. That workflow doesn't create the kfserving-gateway resources, even though the local manifests contain the files.

Provided I use the straight kfctl apply -f flow, I am able to get predictions in this setup. Though I notice that my prediction requests go through even if I don't add any auth header. It seems it's effectively skipping the isto-dex auth. Not that this change introduced that (it was skipping auth previously), just not sure if it's intentional.

[Jeffwan]

@ryandawsonuk If you switch to kfdef with kfserving-gateway, it is supposed to skip auth since it uses LoadBalancer without auth directly?

[ryandawsonuk]

Yes that’s what I see.

[yuzisun]

@ryandawsonuk @cliveseldon Can you help test KFServing in Istio/Dex setup with kubeflow manifest master now?

[sachua]

@yuzisun I tried KFServing's sklearn example with kubeflow manifest master in Istio-Dex, I am getting IngressNotConfigured when deploying the inferenceservice:

NAMESPACE   NAME                             URL                                                       LATESTCREATED                          LATESTREADY                            READY     REASON
admin       sklearn-iris-predictor-default   http://sklearn-iris-predictor-default.admin.example.com   sklearn-iris-predictor-default-6j628   sklearn-iris-predictor-default-6j628   Unknown   IngressNotConfigured

Checking the logs of networking-istio, I am getting 404s and 302s.

[yuzisun]

@sachua Can you help get the output of kubectl get deployment networking-istio -n knative-serving -oyaml | grep image: ?

[sachua]

@yuzisun

$ kubectl get deployment networking-istio -n knative-serving -oyaml | grep image:
        image: gcr.io/knative-releases/knative.dev/serving/cmd/networking/istio@sha256:727a623ccb17676fae8058cb1691207a9658a8d71bc7603d701e23b1a6037e6c

hmm weird, i checked my deployment.yaml and the image should have been

gcr.io/knative-releases/knative.dev/serving/cmd/networking/istio@sha256:61461fa789e19895d7d1e5ab96d8bb52a63788e0607e1bd2948b9570efeb6a8f

Thanks!

[yuzisun]

@sachua Let me know if that solves the issue.

[sachua]

@yuzisun yes, the inferenceservice can be deployed now:

NAMESPACE   NAME                             URL                                                       LATESTCREATED                          LATESTREADY                            READY   REASON
admin       sklearn-iris-predictor-default   http://sklearn-iris-predictor-default.admin.example.com   sklearn-iris-predictor-default-6j628   sklearn-iris-predictor-default-6j628   True    

Also, I had to generate the auth token when requesting for predictions, I followed https://github.com/kubeflow/kfctl/issues/140#issuecomment-578837304

$ curl -v -H "Host: sklearn-iris.admin.example.com" -H 'Cookie: authservice_session=MTU4OTE3Njk5MHxOd3dBTkVKTE1sWllURmsxV2tkSlRsTTBSalpKV1ZWQ1NqVk5VVGRCV0VKYVJUTlhRbGhaV0RaWE5FbFNWa0pMUTBjM1RGYzNRa0U9fLJ0uQXwPz6iS5j6MSD9L' http://localhost:8080/v1/models/sklearn-iris:predict -d @./docs/samples/sklearn/iris-input.json
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.admin.example.com
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: authservice_session=MTU4OTE3Njk5MHxOd3dBTkVKTE1sWllURmsxV2tkSlRsTTBSalpKV1ZWQ1NqVk5VVGRCV0VKYVJUTlhRbGhaV0RaWE5FbFNWa0pMUTBjM1RGYzNRa0U9fLJ0uQXwPz6iSPj_JDhgBlcie2JiVmup39-45j6MSD9L
> Content-Length: 76
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 76 out of 76 bytes
< HTTP/1.1 200 OK
< content-length: 23
< content-type: text/html; charset=UTF-8
< date: Mon, 11 May 2020 06:20:34 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 5731
< 
* Connection #0 to host localhost left intact

$ curl -H "Host: sklearn-iris.admin.example.com" -H 'Cookie: authservice_session=MTU4OTE3Njk5MHxOd3dBTkVKTE1sWllURmsxV2tkSlRsTTBSalpKV1ZWQ1NqVk5VVGRCV0VKYVJUTlhRbGhaV0RaWE5FbFNWa0pMUTBjM1RGYzNRa0U9fLJ0uQXwPz6iSPj_JDhgBlcie2JiVmup39-45j6MSD9L' http://localhost:8080/v1/models/sklearn-iris:predict -d @./docs/samples/sklearn/iris-input.json
{"predictions": [1, 1]}

Thanks!

[yuzisun]

Nice! @sachua Would you be interested in contributing the docs for how KFServing works with Istio/Dex ?

[sachua]

@yuzisun sure, how can I contribute?

[yuzisun]

@sachua Add Istio/Dex example here and send a PR following contribution guideline?

[yuzisun]

Thanks @sachua for the awesome write up !
https://github.com/kubeflow/kfserving/blob/master/docs/samples/istio-dex/README.md

/close

[k8s-ci-robot]

@yuzisun: Closing this issue.

In response to this:

Thanks @sachua for the awesome write up !
https://github.com/kubeflow/kfserving/blob/master/docs/samples/istio-dex/README.md

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.