Kfserving: Unable to retrieve SERVICE_HOSTNAME for flowers-sample example

[jal06]

/kind bug

What steps did you take and what happened:

I installed Kubeflow on GCP using https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_gcp_iap.yaml (which works with manifests/master branch), as suggested in 991
I used this kfdef file because I already had the same issue with version 1.0.0 and there is an error 500 when installing with 1.0.1

The installation with this kfdef file doesn't generate any error.

When trying to deploy the flowers-sample inferenceService, it looks OK :

kubectl apply -f tensorflow.yaml
inferenceservice.serving.kubeflow.org/flowers-sample created

(according to the documentation, the message should be inferenceservice.serving.kubeflow.org/flowers-sample configured instead of created)

when trying to run a prediction I am unable to get the SERVICE_IP if I follow the instructions of the documentation. The command echo $CLUSTER_IP returns null

MODEL_NAME=flowers-sample
INPUT_PATH=@./input.json
CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $CLUSTER_IP

If I try with another jsonpath , I get :
CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')
echo $CLUSTER_IP
10.27.243.240

Is {.spec.clusterIP}') the right jsonpath for retrieving the CLUSTER_IP on GCP ?

Then I try to retrieve the SERVICE_HOSTNAME, the command echo $SERVICE_HOSTNAME returns null:
SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
echo $SERVICE_HOSTNAME

When describing the inferenceservice flowers-sample, I see some errors :
Message: Ingress has not yet been reconciled.
Reason: IngressNotConfigured
Status: Unknown
...
Message: Failed to reconcile predictor
Reason: PredictorHostnameUnknown

Did I miss something ?

More details about the inferenceservice flowers-sample below. Please let me know if you need more details:

kubectl describe inferenceservice flowers-sample
Name: flowers-sample
Namespace: kubeflow-jeanarmel-luce
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"serving.kubeflow.org/v1alpha2","kind":"InferenceService","metadata":{"annotations":{},"name":"flowers-sample","namespace":"...
API Version: serving.kubeflow.org/v1alpha2
Kind: InferenceService
Metadata:
Creation Timestamp: 2020-03-13T06:22:10Z
Generation: 4
Resource Version: 22777
Self Link: /apis/serving.kubeflow.org/v1alpha2/namespaces/kubeflow-jeanarmel-luce/inferenceservices/flowers-sample
UID: f8f39e4d-64f2-11ea-bde0-42010a840165
Spec:
Default:
Predictor:
Tensorflow:
Resources:
Limits:
Cpu: 1
Memory: 2Gi
Requests:
Cpu: 1
Memory: 2Gi
Runtime Version: 1.14.0
Storage Uri: gs://kfserving-samples/models/tensorflow/flowers
Status:
Canary:
Conditions:
Last Transition Time: 2020-03-13T06:22:53Z
Message: Ingress has not yet been reconciled.
Reason: IngressNotConfigured
Status: Unknown
Type: DefaultPredictorReady
Last Transition Time: 2020-03-13T06:22:11Z
Message: Failed to reconcile predictor
Reason: PredictorHostnameUnknown
Status: False
Type: Ready
Last Transition Time: 2020-03-13T06:22:11Z
Message: Failed to reconcile predictor
Reason: PredictorHostnameUnknown
Status: False
Type: RoutesReady
Default:
Predictor:
Name: flowers-sample-predictor-default-8mpwh
Events:

What did you expect to happen:

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Istio Version: 1.1.6 (installed by kfctl)
• Knative Version: (installed by kfctl)
• KFServing Version:
• Kubeflow version: 1.0 and master branch
• Minikube version:
• Kubernetes version: (use kubectl version): v1.14.10-gke.27
• OS (e.g. from /etc/os-release):

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| bug | 0.99 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[yuzisun]

/cc @krishnadurai

[krishnadurai]

@jal06 we are working on releasing a patch for this change. You can track the status with this PR here:

https://github.com/kubeflow/manifests/pull/1017

[yuzisun]

@krishnadurai is this patched in kubeflow 1.0.1 yet ?

[yuzisun]

@jal06 Can you try now the fix the patched to kubeflow v1.0.1

[yuzisun]

@jal06 note that you will need to change to use kfserving ingress gateway instead of default istio ingress gateway, we are going to update our docs to reflect his change.

kubectl -n istio-system get service kfserving-ingressgateway

[jal06]

I created a new instance with the patched kubeflow v1.01.
The good news is that I can now retrieve the CLUSTER_IP
But it looks that I am not able to retrieve the SERVICE_HOSTNAME

~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ MODEL_NAME=flowers-sample
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ INPUT_PATH=@./input.json
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ CLUSTER_IP=$(kubectl -n istio-system get service kfserving-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ echo $CLUSTER_IP
35.205.8.134
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ echo $SERVICE_HOSTNAME

[jal06]

The command
SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
returns null

Did I miss something ?

[krishnadurai]

Can you paste the output of kubectl get ksvc,configuration,revision,route ?

[jal06]

@yuzisun @krishnadurai

When describing the inferenceservice , I see that Ingress has not yet been reconciled

kubectl get ksvc,configuration,revision,route
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/flowers-sample-predictor-default http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.example.com flowers-sample-predictor-default-7nrvt flowers-sample-predictor-default-7nrvt Unknown IngressNotConfigured

NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/flowers-sample-predictor-default flowers-sample-predictor-default-7nrvt flowers-sample-predictor-default-7nrvt True

NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
revision.serving.knative.dev/flowers-sample-predictor-default-7nrvt flowers-sample-predictor-default flowers-sample-predictor-default-7nrvt 1 True

NAME URL READY REASON
route.serving.knative.dev/flowers-sample-predictor-default http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.example.com Unknown IngressNotConfigured

[yuzisun]

@jal06 any luck ?

[jal06]

@yuzisun
thanks for upgrading the doc in 741

It sounds that there is a typo :
the name of the service is kfserving-ingressgateway instead of kfserving-ingressgatway (the letter 'e' is missing in gateway)

With this new command, I can retrieve the CLUSTER_IP
MODEL_NAME=flowers-sample
INPUT_PATH=@./input.json
INGRESS_GATEWAY=kfserving-ingressgateway
CLUSTER_IP=$(kubectl -n istio-system get service $INGRESS_GATEWAY -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

However, I think there is still something wrong when retrieving the SERVICE_HOSTNAME. The command for getting the SERVICE_HOSTNAME still returns no value.

SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
echo $SERVICE_HOSTNAME

I still have themessage IngressNotConfigured when I execute kubectl get route or kubectl get ksvc

kubectl get route
NAME URL READY REASON
flowers-sample-predictor-default http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.example.com Unknown IngressNotConfigured

kubectl get ksvc
NAME URL LATESTCREATED LATESTREADY READY REASON
flowers-sample-predictor-default http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.example.com flowers-sample-predictor-default-mq4bs flowers-sample-predictor-default-mq4bs Unknown IngressNotConfigured

Do you know how I could configure Ingress ?

[yuzisun]

@jal06 Can you check if there are errors with the instruction I added in above PR?

[jal06]

@yuzisun
Thanks for fixing typo. Looks good

About IngressNotConfigured, my understanding is that it will be solved by 949 . Do you confirm ?
FYI, I installed the version 1.0.1 a few days ago, and the kfserving-ingressgateway was installed with this version. And I still encounter this issue IngressNotConfigured.
If I look at the description of 1.0.1, I don't see 949 . Did I miss something ?

[yuzisun]

@jal06 can you check this log see if you find any error there
kubectl logs -l app=networking-istio -n knative-serving?

[jal06]

@yuzisun please find below the logs:
kubectl logs -l app=networking-istio -n knative-serving
{"level":"error","ts":"2020-03-26T19:12:00.739Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.svc:80/ failed, IP: 10.0.1.17:80, ready: false, error: unexpected status code: want [200], got 404 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"error","ts":"2020-03-26T19:12:00.739Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce:80/ failed, IP: 10.0.1.17:80, ready: false, error: unexpected status code: want [200], got 404 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"info","ts":"2020-03-26T19:12:00.747Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.example.com:80/, IP: 10.0.0.29:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"error","ts":"2020-03-26T19:12:00.749Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.example.com:80/ failed, IP: 10.0.0.29:80, ready: false, error: unexpected status code: want [200], got 403 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"info","ts":"2020-03-26T19:12:00.774Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.svc.cluster.local:80/, IP: 10.0.0.29:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}

Let me know if you would like more nfo

[yuzisun]

@jal06 I see HTTP 403, so I guess you may have istio RBAC turned on?

[krishnadurai]

@jal06 Can you show us the output of both?

kubectl get cm -n kubeflow inferenceservice-config
kubectl get cm -n knative-serving config-istio

If I look at the description of 1.0.1, I don't see 949 . Did I miss something ?

https://github.com/kubeflow/manifests/tree/v1.0.1

If you take a look at this tagged state of the repo, you'll find PR 949 there.

The tag was updated once post the release. Could you ensure if the folder, within which you start the install with kfctl, is new and doesn't contain older manifests?

[jal06]

@yuzisun : yes RBAC is on. I installed KF for GCP usinf OAuth. With this kindd of install, RBAC is on

@krishnadurai
date; kubectl get cm -n kubeflow inferenceservice-config
Fri Mar 27 01:20:15 CET 2020
NAME DATA AGE
inferenceservice-config 7 2d16h

date; kubectl get cm -n knative-serving config-istio
Fri Mar 27 01:20:27 CET 2020
NAME DATA AGE
config-istio 4 2d16h

I used kfctl_gcp_iap.v1.0.1.yaml from the v1.0-branch : https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_gcp_iap.v1.0.1.yaml

The content of the folder sounds good.
cd ~/kubeflow/install/installdfykubeflowjal2/kfdeployment/kfjal2/kustomize/kfserving-gateway/base
ls -lh
-rw-r--r-- 1 jaluce users 4.9K Mar 24 08:42 deployment.yaml
-rw-r--r-- 1 jaluce users 412 Mar 24 08:42 horizontal-pod-autoscaler.yaml
-rw-r--r-- 1 jaluce users 165 Mar 24 08:42 kustomization.yaml
-rw-r--r-- 1 jaluce users 1.1K Mar 24 08:42 service.yaml

I reinstalled a few minutes ago a new instance of Kubeflow and still have same problem ; echo $SERVICE_HOSTNAME returns null
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ kubectl apply -f tensorflow.yaml
inferenceservice.serving.kubeflow.org/flowers-sample created
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ MODEL_NAME=flowers-sample
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ INPUT_PATH=@./input.json
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ INGRESS_GATEWAY=kfserving-ingressgateway
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ CLUSTER_IP=$(kubectl -n istio-system get service $INGRESS_GATEWAY -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ echo $CLUSTER_IP
34.76.206.6
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$ echo $SERVICE_HOSTNAME

~/kubeflow/kfserving/kfserving/docs/samples/tensorflow$

When I look at the json definition of the inferenceservice flowers-sample, the status looks wrong:
kubectl get inferenceservice ${MODEL_NAME} -o json
...
"status": {
"canary": {},
"conditions": [
{
"lastTransitionTime": "2020-03-27T01:09:06Z",
"message": "Ingress has not yet been reconciled.",
"reason": "IngressNotConfigured",
"status": "Unknown",
"type": "DefaultPredictorReady"
},
{
"lastTransitionTime": "2020-03-27T01:08:22Z",
"message": "Failed to reconcile predictor",
"reason": "PredictorHostnameUnknown",
"status": "False",
"type": "Ready"
},
{
"lastTransitionTime": "2020-03-27T01:08:22Z",
"message": "Failed to reconcile predictor",
"reason": "PredictorHostnameUnknown",
"status": "False",
"type": "RoutesReady"
}
],
"default": {
"predictor": {
"name": "flowers-sample-predictor-default-xvpkc"
}
}
}
}

[krishnadurai]

@jal06 I'm sorry I did not write the complete command. I wanted to see the values (yaml content) within this configuration:

kubectl get cm -n kubeflow inferenceservice-config -o yaml
kubectl get cm -n knative-serving config-istio -o yaml

A possibility is that these config maps are misconfigured to point to istio-ingressgateway.

[jal06]

kubectl get cm -n kubeflow inferenceservice-config -o yaml
apiVersion: v1
data:
credentials: |-
{
"gcs": {
"gcsCredentialFileName": "gcloud-application-credentials.json"
},
"s3": {
"s3AccessKeyIDName": "awsAccessKeyID",
"s3SecretAccessKeyName": "awsSecretAccessKey"
}
}
explainers: |-
{
"alibi": {
"image" : "gcr.io/kfserving/alibi-explainer",
"defaultImageVersion": "0.2.2",
"allowedImageVersions": [
"0.2.2"
]
}
}
ingress: |-
{
"ingressGateway" : "knative-ingress-gateway.knative-serving",
"ingressService" : "kfserving-ingressgateway.istio-system.svc.cluster.local"
}
logger: |-
{
"image" : "gcr.io/kfserving/logger:0.2.2",
"memoryRequest": "100Mi",
"memoryLimit": "1Gi",
"cpuRequest": "100m",
"cpuLimit": "1"
}
predictors: |-
{
"tensorflow": {
"image": "tensorflow/serving",
"defaultImageVersion": "1.14.0",
"defaultGpuImageVersion": "1.14.0-gpu",
"allowedImageVersions": [
"1.11.0",
"1.11.0-gpu",
"1.12.0",
"1.12.0-gpu",
"1.13.0",
"1.13.0-gpu",
"1.14.0",
"1.14.0-gpu"
]
},
"onnx": {
"image": "mcr.microsoft.com/onnxruntime/server",
"defaultImageVersion": "v0.5.1",
"allowedImageVersions": [
"v0.5.1"
]
},
"sklearn": {
"image": "gcr.io/kfserving/sklearnserver",
"defaultImageVersion": "0.2.2",
"allowedImageVersions": [
"0.2.2"
]
},
"xgboost": {
"image": "gcr.io/kfserving/xgbserver",
"defaultImageVersion": "0.2.2",
"allowedImageVersions": [
"0.2.2"
]
},
"pytorch": {
"image": "gcr.io/kfserving/pytorchserver",
"defaultImageVersion": "0.2.2",
"allowedImageVersions": [
"0.2.2"
]
},
"tensorrt": {
"image": "nvcr.io/nvidia/tensorrtserver",
"defaultImageVersion": "19.05-py3",
"allowedImageVersions": [
"19.05-py3"
]
}
}
storageInitializer: |-
{
"image" : "gcr.io/kfserving/storage-initializer:0.2.2",
"memoryRequest": "100Mi",
"memoryLimit": "1Gi",
"cpuRequest": "100m",
"cpuLimit": "1"
}
transformers: |-
{
}
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"credentials":"{\n \"gcs\": {\n \"gcsCredentialFileName\": \"gcloud-application-credentials.json\"\n },\n \"s3\": {\n \"s3AccessKeyIDName\": \"awsAccessKeyID\",\n \"s3SecretAccessKeyName\": \"awsSecretAccessKey\"\n }\n}","explainers":"{\n \"alibi\": {\n \"image\" : \"gcr.io/kfserving/alibi-explainer\",\n \"defaultImageVersion\": \"0.2.2\",\n \"allowedImageVersions\": [\n \"0.2.2\"\n ]\n }\n}","ingress":"{\n \"ingressGateway\" : \"knative-ingress-gateway.knative-serving\",\n \"ingressService\" : \"kfserving-ingressgateway.istio-system.svc.cluster.local\"\n}","logger":"{\n \"image\" : \"gcr.io/kfserving/logger:0.2.2\",\n \"memoryRequest\": \"100Mi\",\n \"memoryLimit\": \"1Gi\",\n \"cpuRequest\": \"100m\",\n \"cpuLimit\": \"1\"\n}","predictors":"{\n \"tensorflow\": {\n \"image\": \"tensorflow/serving\",\n \"defaultImageVersion\": \"1.14.0\",\n \"defaultGpuImageVersion\": \"1.14.0-gpu\",\n \"allowedImageVersions\": [\n \"1.11.0\",\n \"1.11.0-gpu\",\n \"1.12.0\",\n \"1.12.0-gpu\",\n \"1.13.0\",\n \"1.13.0-gpu\",\n \"1.14.0\",\n \"1.14.0-gpu\"\n ]\n },\n \"onnx\": {\n \"image\": \"mcr.microsoft.com/onnxruntime/server\",\n \"defaultImageVersion\": \"v0.5.1\",\n \"allowedImageVersions\": [\n \"v0.5.1\"\n ]\n },\n \"sklearn\": {\n \"image\": \"gcr.io/kfserving/sklearnserver\",\n \"defaultImageVersion\": \"0.2.2\",\n \"allowedImageVersions\": [\n \"0.2.2\"\n ]\n },\n \"xgboost\": {\n \"image\": \"gcr.io/kfserving/xgbserver\",\n \"defaultImageVersion\": \"0.2.2\",\n \"allowedImageVersions\": [\n \"0.2.2\"\n ]\n },\n \"pytorch\": {\n \"image\": \"gcr.io/kfserving/pytorchserver\",\n \"defaultImageVersion\": \"0.2.2\",\n \"allowedImageVersions\": [\n \"0.2.2\"\n ]\n },\n \"tensorrt\": {\n \"image\": \"nvcr.io/nvidia/tensorrtserver\",\n \"defaultImageVersion\": \"19.05-py3\",\n \"allowedImageVersions\": [\n \"19.05-py3\"\n ]\n }\n}","storageInitializer":"{\n \"image\" : \"gcr.io/kfserving/storage-initializer:0.2.2\",\n \"memoryRequest\": \"100Mi\",\n \"memoryLimit\": \"1Gi\",\n \"cpuRequest\": \"100m\",\n \"cpuLimit\": \"1\"\n}","transformers":"{\n}"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"kfserving-install","app.kubernetes.io/instance":"kfserving-install-0.2.2","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"kfserving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"0.2.2","kustomize.component":"kfserving"},"name":"inferenceservice-config","namespace":"kubeflow"}}
creationTimestamp: "2020-03-27T00:53:36Z"
labels:
app.kubernetes.io/component: kfserving-install
app.kubernetes.io/instance: kfserving-install-0.2.2
app.kubernetes.io/managed-by: kfctl
app.kubernetes.io/name: kfserving-install
app.kubernetes.io/part-of: kubeflow
app.kubernetes.io/version: 0.2.2
kustomize.component: kfserving
name: inferenceservice-config
namespace: kubeflow
resourceVersion: "3681"

[jal06]

kubectl get cm -n knative-serving config-istio -o yaml
apiVersion: v1
data:
gateway.knative-serving.knative-ingress-gateway: kfserving-ingressgateway.istio-system.svc.cluster.local
local-gateway.knative-serving.cluster-local-gateway: cluster-local-gateway.istio-system.svc.cluster.local
local-gateway.mesh: mesh
reconcileExternalGateway: "false"
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"gateway.knative-serving.knative-ingress-gateway":"kfserving-ingressgateway.istio-system.svc.cluster.local","local-gateway.knative-serving.cluster-local-gateway":"cluster-local-gateway.istio-system.svc.cluster.local","local-gateway.mesh":"mesh","reconcileExternalGateway":"false"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative","networking.knative.dev/ingress-provider":"istio","serving.knative.dev/release":"v0.11.1"},"name":"config-istio","namespace":"knative-serving"}}
creationTimestamp: "2020-03-27T00:53:29Z"
labels:
app.kubernetes.io/component: knative-serving-install
app.kubernetes.io/instance: knative-serving-install-v0.11.1
app.kubernetes.io/managed-by: kfctl
app.kubernetes.io/name: knative-serving-install
app.kubernetes.io/part-of: kubeflow
app.kubernetes.io/version: v0.11.1
kustomize.component: knative
networking.knative.dev/ingress-provider: istio
serving.knative.dev/release: v0.11.1
name: config-istio
namespace: knative-serving
resourceVersion: "3487"
selfLink: /api/v1/namespaces/knative-serving/configmaps/config-istio
uid: 6015e672-6fc5-11ea-8d52-42010a840118

[yuzisun]

@jal06 check my instruction here https://github.com/kubeflow/kfserving/pull/749, looks you need to create Istio RBAC rule to unblock the request since by default it is deny all if you have RBAC on.

[jal06]

@yuzisun thanks for your help.

I think I'll need some guidance.

I tried to create Istio RBAC rule but I continue to see some 403 errors in logs.

What looks strange to me is that the serviceaccount controller from the knative-serving namespace is able to get (or watch) the pods and service even if I do not create RBAC rules

kubectl auth can-i get services -n <user-namespace> --as system:serviceaccount:knative-serving:controller
yes
kubectl auth can-i get pods -n <user-namespace> --as system:serviceaccount:knative-serving:controller
yes

Probably I missed something.
Could you please confirm that the serviceaccount I must allow is controller from the knative-serving namespace ?
And which rules (resources/verbs) must be set ?

[yuzisun]

@jal06 Istio RBAC is different from kubernetes RBAC, Istio RBAC defines a permission to access a list of services and in this case you might need to create a Istio ServiceRoleBinding to walk around for now, you should already have a ServiceRole named ns-access-istio in your user namespace. We will raise an issue to kubeflow manifests to see how we can handle this case when Istio RBAC is on.

apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
  name: knative-network-istio
  namespace: <user-namespace>
spec:
  roleRef:
    kind: ServiceRole
    name: ns-access-istio
  subjects:
  - properties:
      user: "*"

[jal06]

@yuzisun Thanks for the info about Istio RBAC. Your help is highly appreciated.

I applied the servicerolebinding and I still have the same problem : the SERVICE_HOSTNAME is not retrieved.
In the logs, I still see 403 errors

kubectl logs -l app=networking-istio -n knative-serving
{"level":"error","ts":"2020-03-29T07:31:03.058Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.example.com:80/ failed, IP: 10.24.0.18:80, ready: false, error: unexpected status code: want [200], got 403 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}

[jal06]

According to the RBAC documentation, we need to add some methods an paths in the servicerole.

I see no methods and paths in the servicerole ns-access-istio.
Maybe we need some ? WDYT ?
Any other idea ?

k describe servicerole ns-access-istio

Name:         ns-access-istio
Namespace:    kubeflow-jeanarmel-luce
Labels:       <none>
Annotations:  role: admin
              user: [email protected]
API Version:  rbac.istio.io/v1alpha1
Kind:         ServiceRole
Metadata:
  Creation Timestamp:  2020-03-29T06:38:05Z
  Generation:          1
  Owner References:
    API Version:           kubeflow.org/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Profile
    Name:                  kubeflow-jeanarmel-luce
    UID:                   a762f36b-7187-11ea-9264-42010a840098
  Resource Version:        5282
  Self Link:               /apis/rbac.istio.io/v1alpha1/namespaces/kubeflow-jeanarmel-luce/serviceroles/ns-access-istio
  UID:                     d8662a74-7187-11ea-9264-42010a840098
Spec:
  Rules:
    Services:
      *
Status:
Events:  <none>

[yuzisun]

path and methods are optional and defaults to *, I think I gave you the wrong spec, can you try following instead?

apiVersion: rbac.istio.io/v1alpha1
kind: ServiceRoleBinding
metadata:
  name: knative-network-istio
  namespace: <user-namespace>
spec:
  roleRef:
    kind: ServiceRole
    name: ns-access-istio
  subjects:
     - user: "*"

[jal06]

@yuzisun Good news . It works

Thanks a lot !!!

[jal06]

As a follow up :

• when I apply tensorflow.yaml, it works as expected. The requests are now executed successfullly
• I did a canary rollout, and it works too

However, I just notice that , after stopping to send requests to the application during 3 hours, I don't see the application scales itself back down to zero . The pod is still running. Maybe we need to open another issue for that ?

[yuzisun]

As a follow up :

• when I apply tensorflow.yaml, it works as expected. The requests are now executed successfullly
• I did a canary rollout, and it works too

However, I just notice that , after stopping to send requests to the application during 3 hours, I don't see the application scales itself back down to zero . The pod is still running. Maybe we need to open another issue for that ?

This should definitely work, please check the knative autoscaler pod log see if any clues if it is not scale down to 0.

[jal06]

@yuzisun Thanks for your help

In knative-serving, it looks that all pods are working
kubectl get pod

NAME                                READY   STATUS    RESTARTS   AGE
activator-cfc66dc7-98x69            2/2     Running   3          25h
autoscaler-6cc8bc459b-vhjq8         2/2     Running   2          25h
autoscaler-hpa-766c77c5f4-qpnpm     1/1     Running   0          25h
controller-7769b4c845-z2zpr         1/1     Running   0          25h
networking-istio-7fd5c6bd64-qgk5l   1/1     Running   0          25h
webhook-775db85f78-9hc6q            1/1     Running   0          25h

I checked the logs of all containers in all pods, and I see a lot of errors 'unsuccessful scrape' in the container autoscaler in pod autoscaler
kubectl logs autoscaler-6cc8bc459b-vhjq8 -c autoscaler

{"level":"error","ts":"2020-04-05T11:08:46.750Z","logger":"autoscaler.collector","caller":"autoscaler/collector.go:276","msg":"Failed to scrape metrics","commit":"6b0e5c6","knative.dev/key":"kubeflow-jeanarmel-luce/moviesentiment-explainer-default-znclv","error":"unsuccessful scrape, sampleSize=1: Get http://moviesentiment-explainer-default-znclv-private.kubeflow-jeanarmel-luce:9090/metrics: EOF","stacktrace":"knative.dev/serving/pkg/autoscaler.newCollection.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/autoscaler/collector.go:276"}
{"level":"error","ts":"2020-04-05T11:08:46.975Z","logger":"autoscaler.collector","caller":"autoscaler/collector.go:276","msg":"Failed to scrape metrics","commit":"6b0e5c6","knative.dev/key":"kubeflow-jeanarmel-luce/flowers-sample-predictor-default-p8wk4","error":"unsuccessful scrape, sampleSize=1: Get http://flowers-sample-predictor-default-p8wk4-private.kubeflow-jeanarmel-luce:9090/metrics: EOF","stacktrace":"knative.dev/serving/pkg/autoscaler.newCollection.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/autoscaler/collector.go:276"}

Another point which looks strange IMHO ; the RT looks high when launching the test with hey. I executed this test several times; the number of requests on 5 threads is always near 60 requests. And the number of pods is still 1 (no scaling up and no scaling down). Due probably to the high RT ?

hey -z 30s -c 5 -m POST -host ${HOST} -D $INPUT_PATH http://$CLUSTER_IP/v1/models/$MODEL_NAME:predict

Summary:
  Total:        31.2252 secs
  Slowest:      3.3287 secs
  Fastest:      2.0268 secs
  Average:      2.4993 secs
  Requests/sec: 1.9856

  Total data:   13640 bytes
  Size/request: 220 bytes

Response time histogram:
  2.027 [1]     |■■
  2.157 [3]     |■■■■■■
  2.287 [3]     |■■■■■■
  2.417 [20]    |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  2.548 [12]    |■■■■■■■■■■■■■■■■■■■■■■■■
  2.678 [10]    |■■■■■■■■■■■■■■■■■■■■
  2.808 [11]    |■■■■■■■■■■■■■■■■■■■■■■
  2.938 [1]     |■■
  3.068 [0]     |
  3.198 [0]     |
  3.329 [1]     |■■


Latency distribution:
  10% in 2.2906 secs
  25% in 2.3949 secs
  50% in 2.4995 secs
  75% in 2.6350 secs
  90% in 2.7950 secs
  95% in 2.8077 secs
  0% in 0.0000 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0018 secs, 2.0268 secs, 3.3287 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0000 secs
  req write:    0.0001 secs, 0.0001 secs, 0.0004 secs
  resp wait:    2.4971 secs, 2.0266 secs, 3.3055 secs
  resp read:    0.0001 secs, 0.0000 secs, 0.0002 secs

Status code distribution:
  [200] 62 responses

[jal06]

@yuzisun :
1- I think I could now close this issue.
The issue about autoscaling is different than the initial issue about ingress-gateway.
Probably I'll need to do more investigations in logs and metrics for this problem about autoscaling .
And open a new issue if necessary.

2- Maybe we'll also need to add the Istio RBAC RoleBinding in the manifests file so that users who want to install KF in GCP don't encounter this issue. Or would you prefer to wait that 760 is solved/ merged ?

WDYT ?

Thanks !!!

[yuzisun]

@jal06 Thanks for exercising all these in the kubeflow environment.
for 1) Looks like kubeflow namespace has istio sidecar injected for AuthZ, this is the upstream Knative issue I reported sometime back https://github.com/knative/serving/issues/6081, it is still not resolved yet. For now I would suggest disable the sidecar by adding the annotation sidecar.istio.io/inject: false on KFServing inference service yaml.
for 2) I think the better option for now is to add sidecar.istio.io/inject: false annotation until we figure out the plan for AuthN & AuthZ story for KFServing in https://github.com/kubeflow/kfserving/issues/760. With that annotation you do not need the Istio service rolebinding anymore.

[jal06]

@yuzisun

I confirm that the autoscaling sample works after disabling the sidecar.

For the record, I added the annotation in autoscaling.yaml like below :

more autoscale.yaml

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "flowers-sample"
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  default:
    predictor:
      tensorflow:
        storageUri: "gs://kfserving-samples/models/tensorflow/flowers"

I close this issue now.

Thanks !!!

[jal06]

Solved by adding annotation

[jal06]

@yuzisun

I saw that the PR 1137 is merged in master (btw, thanks for this PR) so I installed Kubeflow from master branch using kfctl_gcp_iap.yaml

And then I tried again the tensorflow sample but I still have the same issue than a few weeks ago : the variable CLUSTER_IP is not set. Did I miss something ?

You can find below some investigations I did after applying the tensorflow CRD.
Please let me know if you need more infos.
Let me know if you would like I open another issue

Thanks !!!

kubectl apply -f tensorflow.yaml
inferenceservice.serving.kubeflow.org/flowers-sample created

MODEL_NAME=flowers-sample
INPUT_PATH=@./input.json
INGRESS_GATEWAY=istio-ingressgateway
CLUSTER_IP=$(kubectl -n istio-system get service $INGRESS_GATEWAY -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $CLUSTER_IP

If I look at the yaml description of the service istio-ingressgateway, I see that .status.loadBalancer is empty (no ip).

kubectl get svc istio-ingressgateway -n istio-system -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    backendlock: ""
...
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
    release: istio
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

And I see that the toute and ksvc objects are not ready (REASON : IngressNotConfigured )

kubectl get route
NAME                               URL                                                                           READY     REASON
flowers-sample-predictor-default   http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.example.com   Unknown   IngressNotConfigured

kubectl get ksvc
NAME                               URL                                                                           LATESTCREATED                            LATESTREADY                              READY     REASON
flowers-sample-predictor-default   http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.example.com   flowers-sample-predictor-default-gppgz   flowers-sample-predictor-default-gppgz   Unknown   IngressNotConfigured

[yuzisun]

@jal06 I do not think gcp iap setup is using load balancer. Istio ingress is protected behind IAP so you would need to go through IAP ingress to access the inference service, previously we had issue which exposes the istio ingress publicly. Also can you help output the knative network istio pod image version it should be using 0.11.2.

[jal06]

Yes, it is version v0.11.2 as you can see below

Please, could you elaborate more what you mean by

you would need to go through IAP ingress to access the inference service

?

Thanks

kubectl get po networking-istio-67c9674597-69467 -n knative-serving -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    sidecar.istio.io/inject: "false"
  creationTimestamp: "2020-05-14T06:35:24Z"
  generateName: networking-istio-67c9674597-
  labels:
    app: networking-istio
    app.kubernetes.io/component: knative-serving-install
    app.kubernetes.io/name: knative-serving-install
    kustomize.component: knative
    pod-template-hash: 67c9674597
    serving.knative.dev/release: v0.11.2
...

[yuzisun]

Yes, it is version v0.11.2 as you can see below

Please, could you elaborate more what you mean by

you would need to go through IAP ingress to access the inference service

?

Thanks

kubectl get po networking-istio-67c9674597-69467 -n knative-serving -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    sidecar.istio.io/inject: "false"
  creationTimestamp: "2020-05-14T06:35:24Z"
  generateName: networking-istio-67c9674597-
  labels:
    app: networking-istio
    app.kubernetes.io/component: knative-serving-install
    app.kubernetes.io/name: knative-serving-install
    kustomize.component: knative
    pod-template-hash: 67c9674597
    serving.knative.dev/release: v0.11.2
...

please check https://github.com/kubeflow/kfserving/issues/760, I hope we will have a guide soon for GCP/IAP. Also I guess you have istio sidecar injected on inference service pod, that's probably why you still get IngressNotConfigured error?

[jal06]

Thanks for your answer.
I will take a look at 760 and 824

Yes, Istio sidecar is injected :

• label istio-injection=enabled in my namespace
• the pod flowers-sample-predictor contains 3 containers : kfserving-container, queue-proxy and istio-proxy

So, I don't understand why I still get IngressNotConfigured error

[yuzisun]

Thanks for your answer.
I will take a look at 760 and 824

Yes, Istio sidecar is injected :

• label istio-injection=enabled in my namespace
• the pod flowers-sample-predictor contains 3 containers : kfserving-container, queue-proxy and istio-proxy

So, I don't understand why I still get IngressNotConfigured error

Sorry actually you should not get that error, Can you check the knative istio network pod logs?

[jal06]

k logs networking-istio-67c9674597-69467 -n knative-serving

{"level":"info","ts":"2020-05-14T12:55:44.909Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce:80/, IP: 10.56.1.34:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"error","ts":"2020-05-14T12:55:44.910Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce:80/ failed, IP: 10.56.1.34:80, ready: false, error: unexpected status code: want [200], got 401 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"info","ts":"2020-05-14T12:55:44.945Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.svc.cluster.local:80/, IP: 10.56.0.49:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"error","ts":"2020-05-14T12:55:44.946Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://flowers-sample-predictor-default.kubeflow-jeanarmel-luce.svc.cluster.local:80/ failed, IP: 10.56.0.49:80, ready: false, error: unexpected status code: want [200], got 401 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
...

[james426759]

Hi
I have same question!!
I try ${CLUSTER_IP} return null.

try
kubectl -n istio-system get service $INGRESS_GATEWAY -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
return
...
"status": {
"loadBalancer": {}
}

[yuzisun]

@jal06 Can you check the networking istio pod image sha, it should be gcr.io/knative-releases/knative.dev/serving/cmd/networking/istio@sha256:61461fa789e19895d7d1e5ab96d8bb52a63788e0607e1bd2948b9570efeb6a8f

[jal06]

Good catch !!!
the sha looks different.

image: gcr.io/knative-releases/knative.dev/serving/cmd/networking/istio@sha256:727a623ccb17676fae8058cb1691207a9658a8d71bc7603d701e23b1a6037e6c

[jal06]

My understanding is that 6962 is merged in the master branch of knative, but not pushed in version 0.11.2 of Knative.

So hopefully, 6962 will be pushed in Knative 0.11.3+ and we'll need to link to this new release of KNative. Is it correct ?

And don't hesitate to tell me when you will have a guide for GCP/IAP. I should be more than happy to give it a try :-)

Thanks.

[pvgbabu]

Hi All, I need your help on fixing this Kfserving issue
I am using KFserving installed using PKS(1.15)/ Kubflow 1.0
When i try > kubectl apply -f tensorflow.yaml -n kubeflow
inferenceservice.serving.kubeflow.org/flowers-sample created
but

kubectl get inferenceservices -n kubeflow
NAME URL READY DEFAULT TRAFFIC CANARY TRAFFIC AGE
flowers-sample False 2m43s
kubectl describe inferenceservices -n kubeflow
Message: Configuration "flowers-sample-predictor-default" is waiting for a Revision to become ready
kubectl get pods -n knative-serving
NAME READY STATUS RESTARTS AGE
activator-6dc4884-g5h82 2/2 Running 1 23d
autoscaler-69bcc99c79-xnlvp 2/2 Running 2 23d
autoscaler-hpa-68cc87bfb9-c5cht 1/1 Running 0 23d
controller-95dc7f8bd-ptqq2 1/1 Running 0 23d
networking-istio-5b8c5c6cff-x4746 1/1 Running 0 23d
webhook-67847fb4b5-mgwsb 1/1 Running 0 23d

kubectl logs -l app=networking-istio -n knative-serving
W0514 20:27:32.108954 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 18995786 (18999379)
W0514 20:35:11.115562 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 18999993 (19003682)

Appreciate all your help in this.

[yuzisun]

My understanding is that 6962 is merged in the master branch of knative, but not pushed in version 0.11.2 of Knative.

So hopefully, 6962 will be pushed in Knative 0.11.3+ and we'll need to link to this new release of KNative. Is it correct ?

And don't hesitate to tell me when you will have a guide for GCP/IAP. I should be more than happy to give it a try :-)

Thanks.

The knative fix 6962 is back ported to knative 0.11.2, so If you upgrade knative to 0.11.2 that should work, the same fix has been merged in kubeflow manifest master as well.

[pvgbabu]

Thanksa @yuzisun, , could you please give me the url and upgrade instructions for knative 0.11.2

appreciate all your help

[yuzisun]

@pvgbabu

export KNATIVE_VERSION=v0.11.2
kubectl apply --selector knative.dev/crd-install=true \
--filename https://github.com/knative/serving/releases/download/${KNATIVE_VERSION}/serving.yaml \

kubectl apply --filename https://github.com/knative/serving/releases/download/${KNATIVE_VERSION}/serving.yaml \

[james426759]

@pvgbabu
I have a similar issue.
Have you solved the problem？

[pvgbabu]

No luck @yuzisun , still same issue after upgrade to 0.11.2 using below commands
export KNATIVE_VERSION=v0.11.2
kubectl apply --selector knative.dev/crd-install=true \
--filename https://github.com/knative/serving/releases/download/${KNATIVE_VERSION}/serving.yaml \

kubectl apply --filename https://github.com/knative/serving/releases/download/${KNATIVE_VERSION}/serving.yaml \

kubectl get namespace knative-serving -o 'go-template={{index .metadata.labels "serving.knative.dev/release"}}'
v0.11.2

kubectl get inferenceservices -n kubeflow
NAME URL READY DEFAULT TRAFFIC CANARY TRAFFIC AGE
flowers-sample False 12s
Deleted old inference and recreated but no change Please let us know for further suggestions
kubectl describe inferenceservices -n kubeflow
Message: Configuration "flowers-sample-predictor-default" is waiting for a Revision to become ready
kubectl logs -l app=networking-istio -n knative-serving
W0515 12:00:59.084476 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of v1.ConfigMap ended with: too old resource version: 34913228 (34916105)
W0515 12:08:29.092188 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 34916700 (34919671)
W0515 12:17:28.099368 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 34920244 (34924194)
W0515 12:26:04.108236 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 34924597 (34928627)
W0515 12:33:00.116843 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 34929174 (34932297)
W0515 12:42:21.128824 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 34932880 (34937259)
{"level":"info","ts":"2020-05-15T12:45:11.765Z","logger":"istiocontroller.ingress-controller.config-store","caller":"configmap/store.go:157","msg":"ingress config \"config-network\" config was added or updated: &network.Config{IstioOutboundIPRanges:\"\", DefaultIngressClass:\"istio.ingress.networking.knative.dev\", DomainTemplate:\"{{.Name}}.{{.Namespace}}.{{.Domain}}\", TagTemplate:\"{{.Tag}}-{{.Name}}\", AutoTLS:false, HTTPProtocol:\"enabled\", DefaultCertificateClass:\"cert-manager.certificate.networking.internal.knative.dev\"}","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-05-15T12:45:11.943Z","logger":"istiocontroller.ingress-controller.config-store","caller":"configmap/store.go:157","msg":"ingress config \"config-istio\" config was added or updated: &config.Istio{IngressGateways:[]config.Gateway{config.Gateway{Namespace:\"knative-serving\", Name:\"knative-ingress-gateway\", ServiceURL:\"istio-ingressgateway.istio-system.svc.cluster.local\"}}, LocalGateways:[]config.Gateway{config.Gateway{Namespace:\"knative-serving\", Name:\"cluster-local-gateway\", ServiceURL:\"cluster-local-gateway.istio-system.svc.cluster.local\"}}, ReconcileExternalGateway:false}","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
W0515 12:48:13.136040 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 34939310 (34940384)
W0515 12:53:31.144058 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 34940954 (34943097)

[yuzisun]

@pvgbabu it might be helpful if you can describe the inference service pods, looks different issue

[pvgbabu]

@yuzisun
kubectl describe inferenceservices -n kubeflow
Name: flowers-sample
Namespace: kubeflow
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"serving.kubeflow.org/v1alpha2","kind":"InferenceService","metadata":{"annotations":{},"name":"flowers-sample","namespace":"...
API Version: serving.kubeflow.org/v1alpha2
Kind: InferenceService
Metadata:
Creation Timestamp: 2020-05-15T12:59:35Z
Generation: 4
Resource Version: 34946667
Self Link: /apis/serving.kubeflow.org/v1alpha2/namespaces/kubeflow/inferenceservices/flowers-sample
UID: 9761d3d7-36b8-4849-8205-6b23c3ff1e34
Spec:
Default:
Predictor:
Tensorflow:
Resources:
Storage Uri: gs://kfserving-samples/models/tensorflow/flowers
Status:
Canary:
Conditions:
Last Transition Time: 2020-05-15T12:59:36Z
Message: Configuration "flowers-sample-predictor-default" is waiting for a Revision to become ready.
Reason: RevisionMissing
Status: Unknown
Type: DefaultPredictorReady
Last Transition Time: 2020-05-15T12:59:35Z
Message: Failed to reconcile predictor
Reason: PredictorHostnameUnknown
Status: False
Type: Ready
Last Transition Time: 2020-05-15T12:59:35Z
Message: Failed to reconcile predictor
Reason: PredictorHostnameUnknown
Status: False
Type: RoutesReady
Default:
Predictor:
Name: flowers-sample-predictor-default-52ssg
Events:

[pvgbabu]

my other pks cluster after upgrading v0.11.2 to error log as below with pns executor

kubectl logs -l app=networking-istio -n knative-serving
W0515 16:25:17.317417 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of v1.ConfigMap ended with: too old resource version: 19639913 (19643789)
W0515 16:33:59.324636 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 19644277 (19648593)
W0515 16:42:19.332241 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 19649234 (19652946)
W0515 16:51:52.339761 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 19653594 (19658210)
W0515 16:57:56.347286 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 19658850 (19661447)
W0515 17:03:26.354677 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 19662089 (19664476)
W0515 17:10:47.361688 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 19665091 (19668427)
W0515 17:16:10.369329 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 19668892 (19671416)
{"level":"info","ts":"2020-05-15T17:23:54.180Z","logger":"istiocontroller.ingress-controller.config-store","caller":"configmap/store.go:157","msg":"ingress config \"config-network\" config was added or updated: &network.Config{IstioOutboundIPRanges:\"\", DefaultIngressClass:\"istio.ingress.networking.knative.dev\", DomainTemplate:\"{{.Name}}.{{.Namespace}}.{{.Domain}}\", TagTemplate:\"{{.Tag}}-{{.Name}}\", AutoTLS:false, HTTPProtocol:\"enabled\", DefaultCertificateClass:\"cert-manager.certificate.networking.internal.knative.dev\"}","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-05-15T17:23:54.306Z","logger":"istiocontroller.ingress-controller.config-store","caller":"configmap/store.go:157","msg":"ingress config \"config-istio\" config was added or updated: &config.Istio{IngressGateways:[]config.Gateway{config.Gateway{Namespace:\"knative-serving\", Name:\"knative-ingress-gateway\", ServiceURL:\"istio-ingressgateway.istio-system.svc.cluster.local\"}}, LocalGateways:[]config.Gateway{config.Gateway{Namespace:\"knative-serving\", Name:\"cluster-local-gateway\", ServiceURL:\"cluster-local-gateway.istio-system.svc.cluster.local\"}}, ReconcileExternalGateway:false}","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}

[pvgbabu]

@yuzisun, any update on above issue, or do you need more information for me. please let us know.

[bjmrevilla]

@jal06 Can you check the networking istio pod image sha, it should be gcr.io/knative-releases/knative.dev/serving/cmd/networking/istio@sha256:61461fa789e19895d7d1e5ab96d8bb52a63788e0607e1bd2948b9570efeb6a8f

I can confirm that this is working. I used this KFDef from my fork of the manifests repo to create a new Kubeflow deployment in GKE (1.14). I keep getting errors on @yuzisun's commands for upgrading to v0.11.2 so I just manually changed every version of knative to v0.11.2.

Also don't use the namespace kubeflow for kfserving. Use a different namespace but make sure it has a label of serving.kubeflow.org/inferenceservice: enabled. You can use this config for reference.

For example:
kubectl apply -f tensorflow.yaml -nbjmrevilla

[pvgbabu]

@yuzsin/@krishnadurai any update or suggestion on my knative/kfserving issue on PKS cluster

[jal06]

@bjmrevilla Thanks for your help (and sorry for the delay)

I followed your instructions, and it looks better now.
I confirm that there is no more error "IngressNotConfigured"
The status of objects ksvc, route and inferenceservice is "Ready" 👍

I just need to find now how to query the model

When I follow the instructions of the documentation, the variable IP is not set.

CLUSTER_IP=$(kubectl -n istio-system get service $INGRESS_GATEWAY -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

That's because there is no ip in the description of service istio-ingressgateway ( .status.loadBalancer.ingress[0]).
kubectl -n istio-system get service istio-ingressgateway -o json
...

    },
    "status": {
        "loadBalancer": {}
    }
}

I didn't have time yet to investigate anymore how to query the model; if you are able to query the model, please let me know

Thanks !!!

[pvgbabu]

@yuzisun / @krishnadurai , after upgrading to knative v0.11.2, tried troubleshoot using sklearn.yaml as mentioned in below link. my cluster PKS with nfs_storage and pns executors.

https://github.com/kubeflow/kfserving/blob/master/docs/KFSERVING_DEBUG_GUIDE.md

kubectl apply -f sklearn.yaml -n kubeflow
inferenceservice.serving.kubeflow.org/sklearn-iris created

kubectl get inferenceservices sklearn-iris -n kubeflow
NAME URL READY DEFAULT TRAFFIC CANARY TRAFFIC AGE
sklearn-iris False 55s

kubectl get configuration sklearn-iris-predictor-default -n kubeflow
NAME LATESTCREATED LATESTREADY READY REASON
sklearn-iris-predictor-default sklearn-iris-predictor-default-q8pbn False RevisionFailed

kubectl get revision $(kubectl get configuration sklearn-iris-predictor-default -n kubeflow --output jsonpath="{.status.latestCreatedRevisionName}") -
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
sklearn-iris-predictor-default-q8pbn sklearn-iris-predictor-default 1 False ExitCode1

kubectl logs sklearn-iris-predictor-default-q8pbn-deployment-5dc9d76c56f68nz -n kubeflow kfserving-container
[I 200520 14:11:43 storage:35] Copying contents of /mnt/models to local
Traceback (most recent call last):
File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/sklearnserver/sklearnserver/__main__.py", line 33, in
model.load()
File "/sklearnserver/sklearnserver/model.py", line 33, in load
model_path = kfserving.Storage.download(self.model_dir)
File "/kfserving/kfserving/storage.py", line 58, in download
(_GCS_PREFIX, _S3_PREFIX, _LOCAL_PREFIX))
Exception: Cannot recognize storage type for /mnt/models
'gs://', 's3://', and 'file://' are the current available storage type.

Please let us know for any suggestions, appreciate all your help

more info:
kubectl get ksvc,configuration,revision,route -n kubeflow
NAME URL LATESTCREATED LATESTREADY READY REASON
service.serving.knative.dev/sklearn-iris-predictor-default http://sklearn-iris-predictor-default.kubeflow.example.com sklearn-iris-predictor-default-q8pbn False RevisionMissing

NAME LATESTCREATED LATESTREADY READY REASON
configuration.serving.knative.dev/sklearn-iris-predictor-default sklearn-iris-predictor-default-q8pbn False RevisionFailed

NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
revision.serving.knative.dev/sklearn-iris-predictor-default-q8pbn sklearn-iris-predictor-default 1 False ExitCode1

NAME URL READY REASON
route.serving.knative.dev/sklearn-iris-predictor-default http://sklearn-iris-predictor-default.kubeflow.example.com False RevisionMissing
kubectl get cm -n kubeflow inferenceservice-config
NAME DATA AGE
inferenceservice-config 7 28d
kubectl get cm -n knative-serving config-istio
NAME DATA AGE