Kfserving: Issues with KFServing as part of a Kubeflow install
/kind bug
I'm seeing some issues with the KFServing install that's part of the 'out-of-the-box' Kubeflow install (0.7.1). As documented, this 'should' work without the need to install additional stuff: the KF 0.7.1 install includes istio, knative-serving, and installs the kfserving-controller-manager statefulset.
Not clear if the following probs are all related, so this bug might need to be factored out into several.
It's possible that some of these issues relate to knative-serving vs kubeflow gateways conflicting, as apparently there can be issues if the transport like http or https could go via either gateway. e.g.: https://github.com/istio/istio/issues/11509.
First, it looks like an inferenceservice can only be deployed into the automatically-created kubeflow-<user> namespace. Is this intended?
Otherwise, there's this error:
Error from server: error when creating "kfserving-tf-flowers.yaml": admission webhook "inferenceservice.kfserving-webhook-server.validator" denied the request: Cannot create the Inferenceservice "flowers-sample" in namespace "kubeflow": the namespace lacks label "serving.kubeflow.org/inferenceservice: enabled"
Once it is deployed, the inferenceservice is showing Ready==False, and giving 'Failed to reconcile predictor' errors:
% kubectl describe inferenceservice flowers-sample -n kubeflow-amyu
Name: flowers-sample
Namespace: kubeflow-amyu
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"serving.kubeflow.org/v1alpha2","kind":"InferenceService","metadata":{"annotations":{},"name":"flowers-sample","namespace":"...
API Version: serving.kubeflow.org/v1alpha2
Kind: InferenceService
Metadata:
Creation Timestamp: 2020-01-09T02:55:25Z
Generation: 6
Resource Version: 7257
Self Link: /apis/serving.kubeflow.org/v1alpha2/namespaces/kubeflow-amyu/inferenceservices/flowers-sample
UID: 7c11596e-328b-11ea-a589-42010a800278
Spec:
Default:
Predictor:
Tensorflow:
Resources:
Limits:
Cpu: 1
Memory: 2Gi
Requests:
Cpu: 1
Memory: 2Gi
Runtime Version: 1.14.0
Storage Uri: gs://kfserving-samples/models/tensorflow/flowers
Status:
Canary:
Conditions:
Last Transition Time: 2020-01-09T02:56:01Z
Message: Waiting for VirtualService to be ready
Reason: Uninitialized
Status: Unknown
Type: DefaultPredictorReady
Last Transition Time: 2020-01-09T02:55:25Z
Message: Failed to reconcile predictor
Reason: PredictorHostnameUnknown
Status: False
Type: Ready
Last Transition Time: 2020-01-09T02:55:25Z
Message: Failed to reconcile predictor
Reason: PredictorHostnameUnknown
Status: False
Type: RoutesReady
Default:
Predictor:
Name: flowers-sample-predictor-default-fjjkn
Events: <none>
Then, in this section of the instructions: https://github.com/kubeflow/kfserving/tree/master/docs/samples/tensorflow#run-a-prediction
..this command does not return a value: SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
It’s not finding the status.url. Here’s what the json looks like. What should it be returning? I'm guessing the SERVICE_HOSTNAME should be set to flowers-sample-predictor-default.kubeflow-amyu.svc.cluster.local or similar, right? But perhaps due to the above issue I don't see that string in the json below.
kubectl get inferenceservice ${MODEL_NAME} -o json
{
"apiVersion": "serving.kubeflow.org/v1alpha2",
"kind": "InferenceService",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"serving.kubeflow.org/v1alpha2\",\"kind\":\"InferenceService\",\"metadata\":{\"annotations\":{},\"name\":\"flowers-sample\",\"namespace\":\"kubeflow\"},\"spec\":{\"default\":{\"predictor\":{\"tensorflow\":{\"storageUri\":\"gs://kfserving-samples/models/tensorflow/flowers\"}}}}}\n"
},
"creationTimestamp": "2020-01-08T21:17:24Z",
"generation": 5,
"name": "flowers-sample",
"namespace": "kubeflow",
"resourceVersion": "19958",
"selfLink": "/apis/serving.kubeflow.org/v1alpha2/namespaces/kubeflow/inferenceservices/flowers-sample",
"uid": "43c7f8c0-325c-11ea-bed8-42010a80015f"
},
"spec": {
"default": {
"predictor": {
"tensorflow": {
"resources": {
"limits": {
"cpu": "1",
"memory": "2Gi"
},
"requests": {
"cpu": "1",
"memory": "2Gi"
}
},
"runtimeVersion": "1.14.0",
"storageUri": "gs://kfserving-samples/models/tensorflow/flowers"
}
}
}
},
"status": {
"canary": {},
"conditions": [
{
"lastTransitionTime": "2020-01-08T21:17:58Z",
"message": "Waiting for VirtualService to be ready",
"reason": "Uninitialized",
"status": "Unknown",
"type": "DefaultPredictorReady"
},
{
"lastTransitionTime": "2020-01-08T21:17:24Z",
"message": "Failed to reconcile predictor",
"reason": "PredictorHostnameUnknown",
"status": "False",
"type": "Ready"
},
{
"lastTransitionTime": "2020-01-08T21:17:24Z",
"message": "Failed to reconcile predictor",
"reason": "PredictorHostnameUnknown",
"status": "False",
"type": "RoutesReady"
}
],
"default": {
"predictor": {
"name": "flowers-sample-predictor-default-jnp9v"
}
}
}
}
Finally, even if I deploy the KF install so that the istio-ingressgateway is set up with an exernal IP, I can't successfully make an inference request by following the instructions. I get an origin auth failure.
% kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
35.192.166.45
% curl -v -H "Host: ${SERVICE_HOSTNAME}" http://35.192.166.45/v1/models/$MODEL_NAME:predict -d $INPUT_PATH
* Trying 35.192.166.45...
* TCP_NODELAY set
* Connected to 35.192.166.45 (35.192.166.45) port 80 (#0)
> POST /v1/models/flowers-sample:predict HTTP/1.1
> Host: flowers-sample-predictor-default.kubeflow-amyu.svc.cluster.local
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Length: 16201
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 401 Unauthorized
< content-length: 29
< content-type: text/plain
< date: Thu, 09 Jan 2020 17:30:36 GMT
< server: istio-envoy
< connection: close
<
* we are done reading and this is set to close, stop send
* Closing connection 0
Origin authentication failed.
(cc @jlewi as fyi)
amygdala
All 22 comments
I've also had issues getting the Tensorflow KFServing example up on a vanilla GCP Kubeflow deployment (v0.7.1 and v0.7.0). As @amygdala also mentioned above, I can't get the SERVICE_HOSTNAME (or CLUSTER_IP for that matter). Both are blank because the specified part of the json doesn't appear in the kubectl get responses.
I try getting KFServing into the cluster as suggested here. Relevant lines from the kfserving TF guide:
CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
SERVICE_HOSTNAME=$(kubectl -n kubeflow get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
wronk
on 10 Jan 2020
@wronk did you get the same error amy is getting “waiting for virtual service to be ready”?
yuzisun
on 10 Jan 2020
Here's my output after re-running these lines on the kubeflow deployment. I don't see that error about virtual service now, but I'm not sure it didn't appear before... I'm also not sure which namespace is the correct one to use (in case it's supposed to differ from the docs).
# After Kubeflow deployment:
TAG=0.2.2
CONFIG_URI=https://raw.githubusercontent.com/kubeflow/kfserving/master/install/$TAG/kfserving.yaml
kubectl apply -f ${CONFIG_URI}
kubectl apply -f kfserving_flowers.yaml
$ MODEL_NAME=flowers-sample
$ kubectl get inferenceservice ${MODEL_NAME} -o json
{
"apiVersion": "serving.kubeflow.org/v1alpha2",
"kind": "InferenceService",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"serving.kubeflow.org/v1alpha2\",\"kind\":\"InferenceService\",\"metadata\":{\"annotations\":{},\"name\":\"flowers-sample\",\"namespace\":\"default\"},\"spec\":{\"default\":{\"predictor\":{\"minReplicas\":1,\"tensorflow\":{\"storageUri\":\"gs://kfserving-samples/models/tensorflow/flowers\"}}},\"tolerations\":[{\"effect\":\"NoSchedule\",\"key\":\"mlUseOnly\",\"operator\":\"Equal\",\"value\":\"true\"}]}}\n"
},
"creationTimestamp": "2020-01-10T00:44:41Z",
"generation": 2,
"name": "flowers-sample",
"namespace": "default",
"resourceVersion": "855937",
"selfLink": "/apis/serving.kubeflow.org/v1alpha2/namespaces/default/inferenceservices/flowers-sample",
"uid": "638d82cf-3342-11ea-803c-42010a8e000e"
},
"spec": {
"default": {
"predictor": {
"minReplicas": 1,
"tensorflow": {
"resources": {
"limits": {
"cpu": "1",
"memory": "2Gi"
},
"requests": {
"cpu": "1",
"memory": "2Gi"
}
},
"runtimeVersion": "1.14.0",
"storageUri": "gs://kfserving-samples/models/tensorflow/flowers"
}
}
}
},
"status": {
"canary": {},
"conditions": [
{
"lastTransitionTime": "2020-01-10T00:44:42Z",
"message": "Failed to reconcile predictor",
"reason": "PredictorHostnameUnknown",
"status": "False",
"type": "Ready"
},
{
"lastTransitionTime": "2020-01-10T00:44:42Z",
"message": "Failed to reconcile predictor",
"reason": "PredictorHostnameUnknown",
"status": "False",
"type": "RoutesReady"
}
],
"default": {
"predictor": {}
}
}
}
So you already installed kubeflow with kfserving and apply the kfserving 0.2.2 release yaml again ? With kubeflow installation kfserving gets deployed to kubeflow namespace while the standalone installation deploys to kfserving-system namespace.
yuzisun
on 10 Jan 2020
Edited
I'm facing a similar issue while I'm trying to setup Kubeflow ~0.7.0~ master branch KFServing and trying the flowers example.
The revisions 'flowers-sample-predictor-default' is stuck in status 'Deploying':
kubectl describe revision flowers-sample-predictor-default
Outputs
Name: flowers-sample-predictor-default-w6qr4
Namespace: kubeflow
Labels: serving.knative.dev/configuration=flowers-sample-predictor-default
serving.knative.dev/configurationGeneration=1
serving.knative.dev/service=flowers-sample-predictor-default
serving.kubeflow.org/inferenceservice=flowers-sample
Annotations: autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/target: 1
internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/tensorflow/flowers
queue.sidecar.serving.knative.dev/resourcePercentage: 0.2
serving.knative.dev/creator: system:serviceaccount:kubeflow:default
API Version: serving.knative.dev/v1alpha1
Kind: Revision
Metadata:
Creation Timestamp: 2020-01-10T00:13:26Z
Generate Name: flowers-sample-predictor-default-
Generation: 1
Owner References:
API Version: serving.knative.dev/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Configuration
Name: flowers-sample-predictor-default
UID: 059f78ec-333e-11ea-93a5-42010a8e0070
Resource Version: 674261
Self Link: /apis/serving.knative.dev/v1alpha1/namespaces/kubeflow/revisions/flowers-sample-predictor-default-w6qr4
UID: 05a62ebe-333e-11ea-93a5-42010a8e0070
Spec:
Containers:
Args:
--port=9000
--rest_api_port=8080
--model_name=flowers-sample
--model_base_path=/mnt/models
Command:
/usr/bin/tensorflow_model_server
Image: tensorflow/serving:
Name: kfserving-container
Readiness Probe:
Success Threshold: 1
Tcp Socket:
Port: 0
Resources:
Timeout Seconds: 60
Status:
Conditions:
Last Transition Time: 2020-01-10T00:15:37Z
Message: The target could not be activated.
Reason: TimedOut
Severity: Info
Status: False
Type: Active
Last Transition Time: 2020-01-10T00:13:26Z
Reason: Deploying
Status: Unknown
Type: ContainerHealthy
Last Transition Time: 2020-01-10T00:13:26Z
Reason: Deploying
Status: Unknown
Type: Ready
Last Transition Time: 2020-01-10T00:13:26Z
Reason: Deploying
Status: Unknown
Type: ResourcesAvailable
Image Digest: index.docker.io/tensorflow/serving@sha256:091c1d0440815e250114a6d0232ad3cb1d320c64b1ebb75ed8a80184fc25482d
Log URL: http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'05a62ebe-333e-11ea-93a5-42010a8e0070',type:phrase))))
Observed Generation: 1
Service Name: flowers-sample-predictor-default-w6qr4
Events: <none>
Deployment seems to have scaled down to 0 replica-sets from 1 initially.
kubectl describe deployment flowers-sample-predictor-default-w6qr4-deployment
Name: flowers-sample-predictor-default-w6qr4-deployment
Namespace: kubeflow
CreationTimestamp: Thu, 09 Jan 2020 16:13:26 -0800
Labels: app=flowers-sample-predictor-default-w6qr4
serving.knative.dev/configuration=flowers-sample-predictor-default
serving.knative.dev/configurationGeneration=1
serving.knative.dev/revision=flowers-sample-predictor-default-w6qr4
serving.knative.dev/revisionUID=05a62ebe-333e-11ea-93a5-42010a8e0070
serving.knative.dev/service=flowers-sample-predictor-default
serving.kubeflow.org/inferenceservice=flowers-sample
Annotations: autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/target: 1
deployment.kubernetes.io/revision: 1
internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/tensorflow/flowers
queue.sidecar.serving.knative.dev/resourcePercentage: 0.2
serving.knative.dev/creator: system:serviceaccount:kubeflow:default
Selector: serving.knative.dev/revisionUID=05a62ebe-333e-11ea-93a5-42010a8e0070
Replicas: 0 desired | 0 updated | 0 total | 0 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=flowers-sample-predictor-default-w6qr4
serving.knative.dev/configuration=flowers-sample-predictor-default
serving.knative.dev/configurationGeneration=1
serving.knative.dev/revision=flowers-sample-predictor-default-w6qr4
serving.knative.dev/revisionUID=05a62ebe-333e-11ea-93a5-42010a8e0070
serving.knative.dev/service=flowers-sample-predictor-default
serving.kubeflow.org/inferenceservice=flowers-sample
Annotations: autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/target: 1
internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/tensorflow/flowers
queue.sidecar.serving.knative.dev/resourcePercentage: 0.2
serving.knative.dev/creator: system:serviceaccount:kubeflow:default
sidecar.istio.io/inject: true
traffic.sidecar.istio.io/includeOutboundIPRanges: *
Containers:
kfserving-container:
Image: index.docker.io/tensorflow/serving@sha256:091c1d0440815e250114a6d0232ad3cb1d320c64b1ebb75ed8a80184fc25482d
Port: 8080/TCP
Host Port: 0/TCP
Command:
/usr/bin/tensorflow_model_server
Args:
--port=9000
--rest_api_port=8080
--model_name=flowers-sample
--model_base_path=/mnt/models
Environment:
PORT: 8080
K_REVISION: flowers-sample-predictor-default-w6qr4
K_CONFIGURATION: flowers-sample-predictor-default
K_SERVICE: flowers-sample-predictor-default
Mounts:
/var/log from knative-var-log (rw)
queue-proxy:
Image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:e0654305370cf3bbbd0f56f97789c92cf5215f752b70902eba5d5fc0e88c5aca
Ports: 8022/TCP, 9090/TCP, 9091/TCP, 8012/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Requests:
cpu: 25m
Readiness: exec [/ko-app/queue -probe-period 0] delay=0s timeout=10s period=1s #success=1 #failure=3
Environment:
SERVING_NAMESPACE: kubeflow
SERVING_SERVICE: flowers-sample-predictor-default
SERVING_CONFIGURATION: flowers-sample-predictor-default
SERVING_REVISION: flowers-sample-predictor-default-w6qr4
QUEUE_SERVING_PORT: 8012
CONTAINER_CONCURRENCY: 0
REVISION_TIMEOUT_SECONDS: 60
SERVING_POD: (v1:metadata.name)
SERVING_POD_IP: (v1:status.podIP)
SERVING_LOGGING_CONFIG: {
"level": "info",
"development": false,
"outputPaths": ["stdout"],
"errorOutputPaths": ["stderr"],
"encoding": "json",
"encoderConfig": {
"timeKey": "ts",
"levelKey": "level",
"nameKey": "logger",
"callerKey": "caller",
"messageKey": "msg",
"stacktraceKey": "stacktrace",
"lineEnding": "",
"levelEncoder": "",
"timeEncoder": "iso8601",
"durationEncoder": "",
"callerEncoder": ""
}
}
SERVING_LOGGING_LEVEL:
SERVING_REQUEST_LOG_TEMPLATE:
SERVING_REQUEST_METRICS_BACKEND:
TRACING_CONFIG_ENABLE: false
TRACING_CONFIG_ZIPKIN_ENDPOINT:
TRACING_CONFIG_DEBUG: false
TRACING_CONFIG_SAMPLE_RATE: 0.100000
USER_PORT: 8080
SYSTEM_NAMESPACE: knative-serving
METRICS_DOMAIN: knative.dev/serving
USER_CONTAINER_NAME: kfserving-container
ENABLE_VAR_LOG_COLLECTION: false
VAR_LOG_VOLUME_NAME: knative-var-log
INTERNAL_VOLUME_PATH: /var/knative-internal
SERVING_READINESS_PROBE: {"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}
Mounts: <none>
Volumes:
knative-var-log:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: flowers-sample-predictor-default-w6qr4-deployment-556fdb96b7 (0/0 replicas created)
Events: <none>
krishnadurai
on 10 Jan 2020
@amygdala and for others who faced the last issue you mentioned which related to istio authorization
I had discussion with the team who built the istio authorization solution ( @yanniszark )
So the temporary solution will be either
- disable the auth by using kfctl_k8s_istio deployment
- copy the token from the cookie when you login to the dashboard from browser to the X-Auth-Token in the curl command
and as a reminder, before adding X-Auth-Token to the curl command you have to edit envoyfilter to allow X-Auth-Token in the header (reference to this PR)
ChanYiLin
on 10 Jan 2020
One option here is to configure knative to use kubeflow gateway https://github.com/knative/serving/blob/master/config/istio-ingress/config.yaml#L49, based on my testing this only works with knative 0.11 and previous versions do not support configuring the gateway in a different namespace other than knative-serving. We also need to change kfserving to point to the kubeflow gateway https://github.com/kubeflow/kfserving/blob/master/config/default/configmap/inferenceservice.yaml#L93. After that all the virtual services created by knative or kfserving will start to use kubeflow’s gateway to avoid conflicts.
yuzisun
on 11 Jan 2020
Just a note that to do the above, seems that Kubeflow's version of knative needs to be upgraded (from 0.8 to 0.11)
amygdala
on 14 Jan 2020
@yuzisun What values should those configuration variables be set to?
alexwennerberg
on 14 Jan 2020
In kubeflow installation we have two istio gateway installed, they both serve on http port 80 and route to the same istio ingress gateway, so one way to solve the issue is to configure knative to use the kubeflow gateway instead.
kubectl get gateways --all-namespaces
NAMESPACE NAME AGE
knative-serving knative-ingress-gateway 4d4h
kubeflow kubeflow-gateway 5d5h
1) Currently kubeflow installs knative 0.8 and we need to get onto 0.11 due to an issue fixed in 0.11 which allows configuring the gateway in custom namespace (https://github.com/knative/serving/pull/5859)
2) Edit knative config-istio configmap
kubectl edit cm config-istio -n knative-serving
change
gateway.knative-serving.knative-ingress-gateway: "istio-ingressgateway.istio-system.svc.cluster.local" to
gateway.kubeflow.kubeflow-gateway: "istio-ingressgateway.istio-system.svc.cluster.local
3) Edit kfserving configmap
kubectl edit cm inferenceservice-config -n kfserving-system
change
"ingressGateway" : "knative-ingress-gateway.knative-serving" to
"ingressGateway" : "kubeflow-gateway.kubeflow"
I'd love to see if someone can try out this and confirm this can fix the issue.
yuzisun
on 15 Jan 2020
@yuzisun there's a correction here:
3. Edit kfserving configmap `kubectl edit cm inferenceservice-config -n kfserving-system`
The namespace should be 'kubeflow'.
I'd love to see if someone can try out this and confirm this can fix the issue.
Unfortunately, this does not yet solve this issue.
kubectl describe service.serving.knative.dev flowers-sample-predictor-default
Status:
Address:
URL: http://flowers-sample-predictor-default.kubeflow.svc.cluster.local
Conditions:
Last Transition Time: 2020-01-17T06:53:33Z
Status: True
Type: ConfigurationsReady
Last Transition Time: 2020-01-17T06:53:33Z
Message: Waiting for VirtualService to be ready
Reason: Uninitialized
Status: Unknown
Type: Ready
Last Transition Time: 2020-01-17T06:53:33Z
Message: Waiting for VirtualService to be ready
Reason: Uninitialized
Status: Unknown
Type: RoutesReady
I have a similar issue, the output for the following is BLANK
CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
I believe this is because my environment don't have an external load balancer (using default Kubeflow installation)
kubectl -n istio-system get service istio-ingressgateway -o json
...
},
"status": {
"loadBalancer": {}
}
}
kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway NodePort 10.98.65.35 <none> 15020:31369/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:32341/TCP,15030:31286/TCP,15031:31518/TCP,15032:30609/TCP,15443:32557/TCP 126d
My istio-ingressgateway's type is NodePort. If I use CLUSTER-IP(10.98.65.35), the following work fine for me:
curl -v -H "Host: ${SERVICE_HOSTNAME}" http://$CLUSTER_IP/v1/models/$MODEL_NAME:predict -d $INPUT_PATH
Question: is an external load balancer required for KFServing?
janeman98
on 25 Jan 2020
@janeman98 load balancer is not required, there is a PR to fix the doc https://github.com/kubeflow/kfserving/pull/618
yuzisun
on 25 Jan 2020
@janeman98, I'm still having problems getting the TF flowers example to work from a KF deployment. Did you have to do anything special to make the example work after manually setting the CLUSTER_IP?
Perhaps I'm not setting SERVICE_HOSTNAME correctly? Or maybe there's a special cluster setting I need to define on GCP to enable incoming prediction requests?
cc @yuzisun
Steps to reproduce
Set up KF with GCP IAP
Used the instructions here. Modified to use the kfctl version v1.0-rc.1-0-g963c787 and the master version of CONFIG_URI that addresses #596 ☝️
Applied the example
kubectl -n kubeflow apply -f <my_path_to_kfserving>/kfserving_flowers.yaml
> inferenceservice.serving.kubeflow.org/flowers-sample created
Set the necessary env variables
MODEL_NAME=flowers-sample
INPUT_PATH=<my_path_to_kfserving>/input.json # Same results with or without '@' symbol proceeding path
# Switched this to grab the cluster IP since the load balancer isn't setup (the specified code [here](https://github.com/kubeflow/kfserving/tree/master/docs/samples/tensorflow#run-a-prediction) returns empty string for CLUSTER_IP).
CLUSTER_IP=$(kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.spec.clusterIP}')
echo $CLUSTER_IP
10.39.249.51
# The specified SERVICE_HOSTNAME script in the example also returns empty string, so setting it manually. Is this right? I don't know the exact proper format or how to check if it's valid.
SERVICE_HOSTNAME=flowers-sample-predictor-default.kubeflow.example.com
Curl times out
curl -v -H "Host: ${SERVICE_HOSTNAME}" http://$CLUSTER_IP/v1/models/$MODEL_NAME:predict -d $INPUT_PATH
* Trying 10.39.249.51...
* TCP_NODELAY set
* Connection failed
* connect to 10.39.249.51 port 80 failed: Network is unreachable
* Failed to connect to 10.39.249.51 port 80: Network is unreachable
* Closing connection 0
I'm experiencing issues with this as well. Used the https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_istio_dex.v1.0.0.yaml config first and tried this and it failed as it seems no configuration was made that sets up the ingress for it so instead I tried this one: https://github.com/kubeflow/manifests/blob/master/kfdef/kfctl_aws.v1.0.1.yaml and it doesn't even create an inferenceservice.
[ec2-user@ip-10-0-0-170 tfserving-test]$ kubectl apply -f tensorflow.yaml
error: unable to recognize "tensorflow.yaml": no matches for kind "InferenceService" in version "serving.kubeflow.org/v1alpha2"
stomplee
on 12 Mar 2020
@wronk
Sorry for my late response. I don't see your question until stomplee comment on this issue.
- I install Kubeflow in minikube using
export CONFIG_URI="https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.0.yaml"
Since I don't have load balancer in my env, I use
CLUSTER_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.clusterIP}')I don't do anything special on
SERVICE_HOSTNAME
SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
--> this will set SERVICE_HOSTNAME=flowers-sample.default.example.com
janeman98
on 13 Mar 2020
I'm facing this issue as well (even after upgrading to 1.0.1). On GCP with IAP. Used pip install in TF2-CPU notebook to upgrade to latest Kfserving package.
Seems this is the culprit:
Error detected in taxi-sample-predictor-default version taxi-sample-predictor-default-s5kvs
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/kf-poc-edi/o?projection=noAcl&prefix=tfx_pipeline_output%2Fmy_tfx_on_kf_pipeline%2Fserving_model%2F1584479473%2F: Primary: /namespaces/saas-ml-dev.svc.id.goog with additional claims does not have storage.objects.list access to kf-poc-edi.
Tried following to no avail
KFServing = KFServingClient()
KFServing.set_credentials(storage_type='GCS',
#namespace='kubeflow-edi-bice',
#credentials_file='/secret/gcp/user-gcp-sa.json')#,
service_account='default-editor')
KFServing.create(isvc)
yantriks-edi-bice
on 19 Mar 2020
3. SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)
Thanks @janeman98 - those last two bullets helped out as the sample instructions did not work as is
yantriks-edi-bice
on 20 Mar 2020
Hi All, I need your help on fixing this Kfserving issue
I am using KFserving installed using PKS(1.15) Kubflow 1.0
When i try > kubectl apply -f tensorflow.yaml -n kubeflow
inferenceservice.serving.kubeflow.org/flowers-sample created
but
kubectl get inferenceservices -n kubeflow
NAME URL READY DEFAULT TRAFFIC CANARY TRAFFIC AGE
flowers-sample False 2m43s
kubectl describe inferenceservices -n kubeflow
Message: Configuration "flowers-sample-predictor-default" is waiting for a Revision to become ready
kubectl get pods -n knative-serving
NAME READY STATUS RESTARTS AGE
activator-6dc4884-g5h82 2/2 Running 1 23d
autoscaler-69bcc99c79-xnlvp 2/2 Running 2 23d
autoscaler-hpa-68cc87bfb9-c5cht 1/1 Running 0 23d
controller-95dc7f8bd-ptqq2 1/1 Running 0 23d
networking-istio-5b8c5c6cff-x4746 1/1 Running 0 23d
webhook-67847fb4b5-mgwsb 1/1 Running 0 23d
kubectl logs -l app=networking-istio -n knative-serving
W0514 20:27:32.108954 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 18995786 (18999379)
W0514 20:35:11.115562 1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 18999993 (19003682)
Appreciate all your help in this.
pvgbabu
on 14 May 2020
Issue-Label Bot is automatically applying the labels:
| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.85 |
Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.
![issue-label-bot[bot] picture](https://avatars2.githubusercontent.com/in/27079?v=4&s=40)
issue-label-bot[bot]
on 14 May 2020
@janeman98, I am facing exactly the same issue, as mentioned by @wronk. I tried this solution by you, but it again says connection timed out. Do I need to change the kubeflow version? (Currently I am using the configuration kfctl_k8s_istio.v1.0.2)
Also, the SERVICE_HOSTNAME is blank.
@janeman98, I'm still having problems getting the TF flowers example to work from a KF deployment. Did you have to do anything special to make the example work after manually setting the
CLUSTER_IP?Perhaps I'm not setting
SERVICE_HOSTNAMEcorrectly? Or maybe there's a special cluster setting I need to define on GCP to enable incoming prediction requests?cc @yuzisun
Steps to reproduce
Set up KF with GCP IAP
Used the instructions here. Modified to use the kfctl versionv1.0-rc.1-0-g963c787and the master version of CONFIG_URI that addresses #596Applied the example
kubectl -n kubeflow apply -f <my_path_to_kfserving>/kfserving_flowers.yaml > inferenceservice.serving.kubeflow.org/flowers-sample createdSet the necessary env variables
MODEL_NAME=flowers-sample INPUT_PATH=<my_path_to_kfserving>/input.json # Same results with or without '@' symbol proceeding path # Switched this to grab the cluster IP since the load balancer isn't setup (the specified code [here](https://github.com/kubeflow/kfserving/tree/master/docs/samples/tensorflow#run-a-prediction) returns empty string for CLUSTER_IP). CLUSTER_IP=$(kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.spec.clusterIP}') echo $CLUSTER_IP 10.39.249.51 # The specified SERVICE_HOSTNAME script in the example also returns empty string, so setting it manually. Is this right? I don't know the exact proper format or how to check if it's valid. SERVICE_HOSTNAME=flowers-sample-predictor-default.kubeflow.example.comCurl times out
curl -v -H "Host: ${SERVICE_HOSTNAME}" http://$CLUSTER_IP/v1/models/$MODEL_NAME:predict -d $INPUT_PATH * Trying 10.39.249.51... * TCP_NODELAY set * Connection failed * connect to 10.39.249.51 port 80 failed: Network is unreachable * Failed to connect to 10.39.249.51 port 80: Network is unreachable * Closing connection 0
Manvi07
on 26 Jun 2020
Hi Folks,
This issue was closed on January 29. If you are still having trouble I would suggest opening new issues. It also seems like multiple issues and platforms might be being discussed; I would suggest trying to create one issue for each specific problem.
jlewi
on 26 Jun 2020
Related issues
AGKhalil
·
3Comments
navneet1075
·
3Comments
mokpolar
·
6Comments
zyxue
·
4Comments
valentin-stefan-popa
·
3Comments