kfserving tensorflow flowers-sampleprediction through curl not working in kubeflow-v1.0.2 in GKE

[prem0912]

/kind bug

What steps did you take and what happened:
Kfserving tensorflow flowers-sample prediction through curl command not working in kubeflow-v1.0 in GKE
Followed steps in https://github.com/kubeflow/kfserving/tree/master/docs/samples/tensorflow
flowers-sample in kubeflow v1.0

But error got through curl command to predict data
Error :

• Connected to X.X.X.X port 31380 (#0)
  > POST /v1/models/flowers-sample:predict HTTP/1.1
  > Host: flowers-sample.anonymous.example.com
  > User-Agent: curl/7.58.0
  > Accept: /
  > Content-Length: 16139
  > Content-Type: application/x-www-form-urlencoded
  > Expect: 100-continue
  >
  < HTTP/1.1 100 Continue
• We are completely uploaded and fine
  < HTTP/1.1 404 Not Found
  < x-powered-by: Express
  < content-security-policy: default-src 'none'
  < x-content-type-options: nosniff
  < content-type: text/html; charset=utf-8
  < content-length: 172
  < date: Sun, 26 Apr 2020 17:04:37 GMT
  < x-envoy-upstream-service-time: 0
  < server: istio-envoy
  <
  
  
  
  
  
  
  

Cannot POST /v1/models/flowers-sample:predict

• Connection #0 to host X.X.X.X1 left intact

What did you expect to happen:
flowers-sample curl command to predict the result

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Istio Version: default version comes with kubeflow 1.0
• Knative Version: default version comes with kubeflow 1.0
• KFServing Version: default version comes with kubeflow 1.0
• Kubeflow version: v1.0
• Kubernetes version: (use kubectl version): 1.14.10
• OS (e.g. from /etc/os-release): ubuntu

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| bug | 0.99 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[yuzisun]

@prem0912 Can you get the full output of inference service status?

[prem0912]

@yuzisun You mean logs of flowers-sample inference service or kfserving-control-manager or logs of pods in knative-serving namespace...
One more update kfserving is working in the installation of kubeflow V1.0, which I have done before a month.
But kfserving not working where kubeflow is installed past a week.

[apryiomka]

I can reproduce the issue with sklearn example as well. I think the issue is with kfnative gateway using the wrong selector.

curl -v -H "Host: sklearn-iris.aliaksandrp.example.com" http://localhost:8080/v1/models/sklearn-iris:predict -d https://raw.githubusercontent.com/kubeflow/kfserving/master/docs/samples/sklearn/iris-input.json
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.aliaksandrp.example.com
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 96
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 96 out of 96 bytes
< HTTP/1.1 404 Not Found
< date: Wed, 29 Apr 2020 18:19:44 GMT
< server: istio-envoy
< connection: close
< content-length: 0
<
* Closing connection 0

this is the virtual service:

kubectl get vs sklearn-iris -n aliaksandrp
NAME           GATEWAYS                                    HOSTS                                    AGE
sklearn-iris   [knative-ingress-gateway.knative-serving]   [sklearn-iris.aliaksandrp.example.com]   4d23h

kubectl get vs sklearn-iris -n aliaksandrp -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"serving.kubeflow.org/v1alpha2","kind":"InferenceService","metadata":{"annotations":{},"name":"sklearn-iris","namespace":"aliaksandrp"},"spec":{"default":{"predictor":{"sklearn":{"storageUri":"gs://kfserving-samples/models/sklearn/iris"}}}}}
  creationTimestamp: "2020-04-24T18:31:05Z"
  generation: 1
  name: sklearn-iris
  namespace: aliaksandrp
  ownerReferences:
  - apiVersion: serving.kubeflow.org/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: InferenceService
    name: sklearn-iris
    uid: 24b68254-0771-4ebb-8f9f-00f39ab779fb
  resourceVersion: "32440204"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/aliaksandrp/virtualservices/sklearn-iris
  uid: 07cf1bc7-a8e7-404c-8623-47597575813b
spec:
  gateways:
  - knative-ingress-gateway.knative-serving
  hosts:
  - sklearn-iris.aliaksandrp.example.com
  http:
  - match:
    - uri:
        prefix: /v1/models/sklearn-iris:predict
    route:
    - destination:
        host: kfserving-ingressgateway.istio-system.svc.cluster.local
        port: {}
      headers:
        request:
          set:
            Host: sklearn-iris-predictor-default.aliaksandrp.example.com
      weight: 100

[lsuanet]

I am running into the same problem. I'm on KFServing v0.3.0 and Knative v0.11.2, using Knative with Istio v1.3.1.

After some digging, I found that the url returned by my inference service routes to the cluster-local-gateway and not to the kubernetes service as created by knative:

Virtual Services:

$ kubectl get virtualservices.networking.istio.io -n my-namespace                                                                                                         
NAME                          GATEWAYS                                                                          HOSTS                                                                                                                                                                                                                                                                                                                                     AGE
test                          [knative-ingress-gateway.knative-serving knative-serving/cluster-local-gateway]   [test.my-namespace.example.com test.my-namespace.svc.cluster.local]                                                                                                                                                                                     46m
test-predictor-default        [knative-serving/cluster-local-gateway knative-serving/knative-ingress-gateway]   [test-predictor-default.my-namespace test-predictor-default.my-namespace.example.com test-predictor-default.my-namespace.svc test-predictor-default.my-namespace.svc.cluster.local]   46m
test-predictor-default-mesh   [mesh]                                                                            [test-predictor-default.my-namespace test-predictor-default.my-namespace.svc test-predictor-default.my-namespace.svc.cluster.local]                                                                                                    46m

Virtual service:

$ k get virtualservices.networking.istio.io -n my-namespace test -o yaml
...
spec:
  gateways:
  - knative-ingress-gateway.knative-serving
  - knative-serving/cluster-local-gateway
  hosts:
  - test.my-namespace.example.com
  - test.my-namespace.svc.cluster.local
...
    route:
    - destination:
        host: cluster-local-gateway.istio-system.svc.cluster.local
        port:
          number: 80
      headers:
        request:
          set:
            Host: test-predictor-default.my-namespace.svc.cluster.local
      weight: 100

Since I don't have the cluster-local-gateway enabled, my predictions don't return anything.
However, predictions do work on test-predictor-default.my-namespace.example.com.

Can anyone verify whether this is expected behaviour?

[yuzisun]

@larssuanet We currently depend on local gateway, can you get that installed ?

[bjmrevilla]

I encountered the same problem as well. I followed the instructions in setting up Kubeflow in Minikube running on a Linux VM. This is the KFDef that I used.

$ MODEL_NAME=flowers-sample
$ export INPUT_PATH=@./input.json
$ CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')
10.103.137.5
$ SERVICE_HOSTNAME=$(sudo -E kubectl get inferenceservice ${MODEL_NAME} -n bjmrevilla -o jsonpath='{.status.url}' | cut -d "/" -f 3)
flowers-sample.bjmrevilla.example.com

$ curl -v -H "Host: ${SERVICE_HOSTNAME}" http://$CLUSTER_IP/v1/models/$MODEL_NAME:predict -d $INPUT_PATH

*   Trying 10.103.137.5...
* TCP_NODELAY set
* Connected to 10.103.137.5 (10.103.137.5) port 80 (#0)
> POST /v1/models/flowers-sample:predict HTTP/1.1
> Host: flowers-sample.bjmrevilla.example.com
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 16201
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 404 Not Found
< x-powered-by: Express
< content-security-policy: default-src 'none'
< x-content-type-options: nosniff
< content-type: text/html; charset=utf-8
< content-length: 172
< date: Thu, 14 May 2020 14:57:01 GMT
< x-envoy-upstream-service-time: 0
< server: istio-envoy
< 
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot POST /v1/models/flowers-sample:predict</pre>
</body>
</html>
* Connection #0 to host 10.103.137.5 left intact

Virtual Services:

$ kubectl get vs -n bjmrevilla
NAME                                    GATEWAYS                                                                          HOSTS                                                                                                                                                                                                                 AGE
flowers-sample                          [knative-ingress-gateway.knative-serving]                                         [flowers-sample.bjmrevilla.example.com]                                                                                                                                                                               67m
flowers-sample-predictor-default        [knative-serving/cluster-local-gateway knative-serving/knative-ingress-gateway]   [flowers-sample-predictor-default.bjmrevilla flowers-sample-predictor-default.bjmrevilla.example.com flowers-sample-predictor-default.bjmrevilla.svc flowers-sample-predictor-default.bjmrevilla.svc.cluster.local]   68m
flowers-sample-predictor-default-mesh   [mesh]                                                                            [flowers-sample-predictor-default.bjmrevilla flowers-sample-predictor-default.bjmrevilla.svc flowers-sample-predictor-default.bjmrevilla.svc.cluster.local]                                                           68m

$ kubectl get vs flowers-sample -n bjmrevilla -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"serving.kubeflow.org/v1alpha2","kind":"InferenceService","metadata":{"annotations":{},"name":"flowers-sample","namespace":"bjmrevilla"},"spec":{"default":{"predictor":{"tensorflow":{"storageUri":"gs://kfserving-samples/models/tensorflow/flowers"}}}}}
  creationTimestamp: "2020-05-14T13:54:37Z"
  generation: 1
  name: flowers-sample
  namespace: bjmrevilla
  ownerReferences:
  - apiVersion: serving.kubeflow.org/v1alpha2
    blockOwnerDeletion: true
    controller: true
    kind: InferenceService
    name: flowers-sample
    uid: ec37509c-d032-4935-bef8-6b0145971ed2
  resourceVersion: "5998"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/bjmrevilla/virtualservices/flowers-sample
  uid: fd9c1d91-fd02-4251-be41-9614712a0ced
spec:
  gateways:
  - knative-ingress-gateway.knative-serving
  hosts:
  - flowers-sample.bjmrevilla.example.com
  http:
  - match:
    - uri:
        prefix: /v1/models/flowers-sample:predict
    route:
    - destination:
        host: kfserving-ingressgateway.istio-system.svc.cluster.local
        port: {}
      headers:
        request:
          set:
            Host: flowers-sample-predictor-default.bjmrevilla.example.com
      weight: 100

$ kubectl describe inferenceservice flowers-sample -nbjmrevilla
Name:         flowers-sample
Namespace:    bjmrevilla
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"serving.kubeflow.org/v1alpha2","kind":"InferenceService","metadata":{"annotations":{},"name":"flowers-sample","namespace":"...
API Version:  serving.kubeflow.org/v1alpha2
Kind:         InferenceService
Metadata:
  Creation Timestamp:  2020-05-14T13:54:26Z
  Generation:          5
  Resource Version:    5999
  Self Link:           /apis/serving.kubeflow.org/v1alpha2/namespaces/bjmrevilla/inferenceservices/flowers-sample
  UID:                 ec37509c-d032-4935-bef8-6b0145971ed2
Spec:
  Default:
    Predictor:
      Tensorflow:
        Resources:
          Limits:
            Cpu:     1
            Memory:  2Gi
          Requests:
            Cpu:          1
            Memory:       2Gi
        Runtime Version:  1.14.0
        Storage Uri:      gs://kfserving-samples/models/tensorflow/flowers
Status:
  Canary:
  Conditions:
    Last Transition Time:  2020-05-14T13:54:37Z
    Status:                True
    Type:                  DefaultPredictorReady
    Last Transition Time:  2020-05-14T13:54:37Z
    Status:                True
    Type:                  Ready
    Last Transition Time:  2020-05-14T13:54:37Z
    Status:                True
    Type:                  RoutesReady
  Default:
    Predictor:
      Host:  flowers-sample-predictor-default.bjmrevilla.example.com
      Name:  flowers-sample-predictor-default-4wd4q
  Traffic:   100
  URL:       http://flowers-sample.bjmrevilla.example.com/v1/models/flowers-sample
Events:      <none>

[yuzisun]

@bjmrevilla there is an issue unfortunately in kubeflow manifests v1.0.2 https://github.com/kubeflow/manifests/issues/1142, either you can fall back to v1.0.0 or use manifest from v1.0 branch/master.

[yuzisun]

This is fixed in upcoming kubeflow 1.1, however you can still configure to use kubeflow gateway to fix the issue if you are using 1.0.2. There are two places which need to configure:
1) Configure knative config-istio configmap in knative-serving namespace to use the kubeflow gateway

• kubectl edit cm config-istio -n knative-serving
• replace gateway.knative-serving.knative-ingress-gateway: kfserving-ingressgateway.istio-system.svc.cluster.local with
  gateway.kubeflow.kubeflow-gateway: istio-ingressgateway.istio-system.svc.cluster.local
  2) Configure kfserving inferenceservice-config configmap to use the kubeflow gateway
  
  
  
  • kubectl edit cm inferenceservice-config -n kubeflow
  
  
  • replace
  
  

        "ingressGateway" : "knative-ingress-gateway.knative-serving",
        "ingressService" : "kfserving-ingressgateway.istio-system.svc.cluster.local"

with

        "ingressGateway" : "kubeflow-gateway.kubeflow",
        "ingressService" : "istio-ingressgateway.istio-system.svc.cluster.local"

[SachinVarghese]

Hi, @yuzisun I tried the above configuration as you have mentioned on kubeflow v1.0.2, but now the inference service fails to deploy. I did some debugging and found that the underlying knative route is not ready due to IngressNotConfigured. I am posting the status of a kubectl describe on the route below. But I saw that the revisions are ready, any idea what else could be missing?

Status:
  Address:
    URL:  http://sk-iris-predictor-default.seldon.svc.cluster.local
  Conditions:
    Last Transition Time:  2020-07-10T11:17:20Z
    Status:                True
    Type:                  AllTrafficAssigned
    Last Transition Time:  2020-07-10T11:17:20Z
    Message:               Ingress has not yet been reconciled.
    Reason:                IngressNotConfigured
    Status:                Unknown
    Type:                  IngressReady
    Last Transition Time:  2020-07-10T11:17:20Z
    Message:               Ingress has not yet been reconciled.
    Reason:                IngressNotConfigured
    Status:                Unknown
    Type:                  Ready
  Observed Generation:     1
  Traffic:
    Latest Revision:  true
    Percent:          100
    Revision Name:    sk-iris-predictor-default-v6fhc
  URL:                http://sk-iris-predictor-default.seldon.example.com
Events:
  Type     Reason         Age   From              Message
  ----     ------         ----  ----              -------
  Normal   Created        11s   route-controller  Created placeholder service "sk-iris-predictor-default"
  Normal   Created        11s   route-controller  Created Ingress "sk-iris-predictor-default"
  Warning  InternalError  11s   route-controller  failed to update Ingress: Operation cannot be fulfilled on ingresses.networking.internal.knative.dev "sk-iris-predictor-default": the object has been modified; please apply your changes to the latest version and try again

[yuzisun]

@SachinVarghese you want to give a shot with kubeflow 1.1 RC0?

[yuzisun]

this has been fixed in kubeflow 1.1
/close

[k8s-ci-robot]

@yuzisun: Closing this issue.

In response to this:

this has been fixed in kubeflow 1.1
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.98 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.