Kfserving: [Failed] Cannot execute sklearn-iris-predictor-default
/kind bug
What steps did you take and what happened:
Cannot execute sklearn-iris-predictor-default
After booting kfserving from istio, I do following commands
kubectl label namespace/default serving.kubeflow.org/inferenceservice=enabled
kubectl apply -f docs/samples/sklearn/sklearn.yaml
I cannot solve this problem even checking
KFSERVING_DEBUG_GUIDE.md
What did you expect to happen:
sklearn works fine.
Anything else you would like to add:
$ kubectl get ksvc
NAME URL LATESTCREATED LATESTREADY READY REASON
sklearn-iris-predictor-default http://sklearn-iris-predictor-default.default.example.com sklearn-iris-predictor-default-tmrq6 Unknown RevisionMissing
$ kubectl get revision $(kubectl get configuration sklearn-iris-predictor-default --output jsonpath="{.status.latestCreatedRevisionName}")
NAME CONFIG NAME K8S SERVICE NAME GENERATION READY REASON
sklearn-iris-predictor-default-tmrq6 sklearn-iris-predictor-default sklearn-iris-predictor-default-tmrq6 1 Unknown
$ kubectl get pods -A | grep kfserving
istio-system kfserving-ingressgateway-64c7bd9b76-4v9m8 1/1 Running 0 122m
kubeflow kfserving-controller-manager-0 2/2 Running 1 119m
Environment:
- Istio Version:1.0.2
- Knative Version:on Istio
- KFServing Version:on Istio
- Kubeflow version:on Istio
- Kfdef:k8s_istio
- Microk8s:1.15.11
- Kubernetes version: (use
kubectl version):1.15.11 - OS (e.g. from
/etc/os-release):Ubuntu 19.10
sakaia
All 14 comments
Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.
![issue-label-bot[bot] picture](https://avatars2.githubusercontent.com/in/27079?v=4&s=40)
issue-label-bot[bot]
on 22 May 2020
Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.
issue-label-bot[bot]
on 22 May 2020
show more detail by kubectl describe pods {your-sklearn-iris-pod-name}
Iamlovingit
on 22 May 2020
Thank you
After executing kubectl apply, some pods are running but dissapearing.
So I reexecute it and get following message.
$ kubectl describe pods sklearn-iris-predictor-default-6xb7c-deployment-5cc9784f94nglgx
Name: sklearn-iris-predictor-default-6xb7c-deployment-5cc9784f94nglgx
Namespace: default
Priority: 0
Node: ip-10-10-174-153/10.10.174.153
Start Time: Fri, 22 May 2020 02:43:53 +0000
Labels: app=sklearn-iris-predictor-default-6xb7c
pod-template-hash=5cc9784f94
serving.knative.dev/configuration=sklearn-iris-predictor-default
serving.knative.dev/configurationGeneration=1
serving.knative.dev/revision=sklearn-iris-predictor-default-6xb7c
serving.knative.dev/revisionUID=30786994-3e54-4232-9513-a2394f61478e
serving.knative.dev/service=sklearn-iris-predictor-default
serving.kubeflow.org/inferenceservice=sklearn-iris
Annotations: autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/target: 1
internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/sklearn/iris
queue.sidecar.serving.knative.dev/resourcePercentage: 0.2
serving.knative.dev/creator: system:serviceaccount:kubeflow:default
traffic.sidecar.istio.io/includeOutboundIPRanges: *
Status: Terminating (lasts <invalid>)
Termination Grace Period: 60s
IP: 10.1.1.237
Controlled By: ReplicaSet/sklearn-iris-predictor-default-6xb7c-deployment-5cc9784f94
Init Containers:
storage-initializer:
Container ID: containerd://97b9aead0a9183099a70b9da7574756811537d9cbbb60bc48c8a89dedddc7f44
Image: gcr.io/kfserving/storage-initializer:0.2.2
Image ID: gcr.io/kfserving/storage-initializer@sha256:7a7d3cf4c5121a3e6bad0acc9e88bbdfa9c7f774d80bd64d8e35a84dcfef8890
Port: <none>
Host Port: <none>
Args:
gs://kfserving-samples/models/sklearn/iris
/mnt/models
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 22 May 2020 02:44:07 +0000
Finished: Fri, 22 May 2020 02:44:13 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 1Gi
Requests:
cpu: 100m
memory: 100Mi
Environment: <none>
Mounts:
/mnt/models from kfserving-provision-location (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dqmzs (ro)
Containers:
kfserving-container:
Container ID: containerd://c8a28e515247419e9aaf90ef551d2c4bde36baa6c2e1edae13605f8ce5b2ccd1
Image: gcr.io/kfserving/sklearnserver@sha256:6702dddff99de5b8945f35d913a423e406b6cd039a14cefdbd1886078d393ce5
Image ID: gcr.io/kfserving/sklearnserver@sha256:6702dddff99de5b8945f35d913a423e406b6cd039a14cefdbd1886078d393ce5
Port: 8080/TCP
Host Port: 0/TCP
Args:
--model_name=sklearn-iris
--model_dir=/mnt/models
--http_port=8080
State: Running
Started: Fri, 22 May 2020 02:44:27 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 2Gi
Requests:
cpu: 1
memory: 2Gi
Environment:
PORT: 8080
K_REVISION: sklearn-iris-predictor-default-6xb7c
K_CONFIGURATION: sklearn-iris-predictor-default
K_SERVICE: sklearn-iris-predictor-default
Mounts:
/mnt/models from kfserving-provision-location (ro)
/var/log from knative-var-log (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dqmzs (ro)
queue-proxy:
Container ID: containerd://998873b17ee5374e7074a3af8c3440ffc3f2787c39a6778e473dd8793c52a6c4
Image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:792f6945c7bc73a49a470a5b955c39c8bd174705743abf5fb71aa0f4c04128eb
Image ID: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:792f6945c7bc73a49a470a5b955c39c8bd174705743abf5fb71aa0f4c04128eb
Ports: 8022/TCP, 9090/TCP, 9091/TCP, 8012/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
State: Running
Started: Fri, 22 May 2020 02:44:37 +0000
Ready: False
Restart Count: 0
Limits:
cpu: 40m
memory: 200Mi
Requests:
cpu: 25m
memory: 50Mi
Readiness: exec [/ko-app/queue -probe-period 0] delay=0s timeout=10s period=1s #success=1 #failure=3
Environment:
SERVING_NAMESPACE: default
SERVING_SERVICE: sklearn-iris-predictor-default
SERVING_CONFIGURATION: sklearn-iris-predictor-default
SERVING_REVISION: sklearn-iris-predictor-default-6xb7c
QUEUE_SERVING_PORT: 8012
CONTAINER_CONCURRENCY: 0
REVISION_TIMEOUT_SECONDS: 60
SERVING_POD: sklearn-iris-predictor-default-6xb7c-deployment-5cc9784f94nglgx (v1:metadata.name)
SERVING_POD_IP: (v1:status.podIP)
SERVING_LOGGING_CONFIG: {
"level": "info",
"development": false,
"outputPaths": ["stdout"],
"errorOutputPaths": ["stderr"],
"encoding": "json",
"encoderConfig": {
"timeKey": "ts",
"levelKey": "level",
"nameKey": "logger",
"callerKey": "caller",
"messageKey": "msg",
"stacktraceKey": "stacktrace",
"lineEnding": "",
"levelEncoder": "",
"timeEncoder": "iso8601",
"durationEncoder": "",
"callerEncoder": ""
}
}
SERVING_LOGGING_LEVEL:
SERVING_REQUEST_LOG_TEMPLATE:
SERVING_REQUEST_METRICS_BACKEND: prometheus
TRACING_CONFIG_BACKEND: none
TRACING_CONFIG_ZIPKIN_ENDPOINT:
TRACING_CONFIG_STACKDRIVER_PROJECT_ID:
TRACING_CONFIG_DEBUG: false
TRACING_CONFIG_SAMPLE_RATE: 0.100000
USER_PORT: 8080
SYSTEM_NAMESPACE: knative-serving
METRICS_DOMAIN: knative.dev/internal/serving
USER_CONTAINER_NAME: kfserving-container
ENABLE_VAR_LOG_COLLECTION: false
VAR_LOG_VOLUME_NAME: knative-var-log
INTERNAL_VOLUME_PATH: /var/knative-internal
SERVING_READINESS_PROBE: {"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}
ENABLE_PROFILING: false
SERVING_ENABLE_PROBE_REQUEST_LOG: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-dqmzs (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
knative-var-log:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-dqmzs:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-dqmzs
Optional: false
kfserving-provision-location:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m57s default-scheduler Successfully assigned default/sklearn-iris-predictor-default-6xb7c-deployment-5cc9784f94nglgx to ip-10-10-174-153
Normal Pulled 2m49s kubelet, ip-10-10-174-153 Container image "gcr.io/kfserving/storage-initializer:0.2.2" already present on machine
Normal Created 2m47s kubelet, ip-10-10-174-153 Created container storage-initializer
Normal Started 2m43s kubelet, ip-10-10-174-153 Started container storage-initializer
Normal Pulled 2m29s kubelet, ip-10-10-174-153 Container image "gcr.io/kfserving/sklearnserver@sha256:6702dddff99de5b8945f35d913a423e406b6cd039a14cefdbd1886078d393ce5" already present on machine
Normal Created 2m28s kubelet, ip-10-10-174-153 Created container kfserving-container
Normal Started 2m22s kubelet, ip-10-10-174-153 Started container kfserving-container
Normal Pulled 2m22s kubelet, ip-10-10-174-153 Container image "gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:792f6945c7bc73a49a470a5b955c39c8bd174705743abf5fb71aa0f4c04128eb" already present on machine
Normal Created 2m20s kubelet, ip-10-10-174-153 Created container queue-proxy
Normal Started 2m13s kubelet, ip-10-10-174-153 Started container queue-proxy
Warning Unhealthy 113s kubelet, ip-10-10-174-153 Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "92717e5305f051496ac40f9b89334bb2e4af82058a2b66a37cae6f9d0445977f": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Warning Unhealthy 104s kubelet, ip-10-10-174-153 Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "fc3ec8341e53f59c25e0130f16fccbb147fef04366b2944b9089a1967e9d3e29": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Warning Unhealthy 103s kubelet, ip-10-10-174-153 Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "bd5da390825cd40a47b159f5f96c73af91c2d7014df0e736d5e534bd10696621": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Warning Unhealthy 103s kubelet, ip-10-10-174-153 Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "d45a3634f359c1837b24edec9ba4e3d410ce24c45ed4e97d7bb5cd64404b934c": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Warning Unhealthy 102s kubelet, ip-10-10-174-153 Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "f0377a7172085b12507352c611c3bc11b6b2d89961a9e34dee37a16cf8b51ab1": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Warning Unhealthy 101s kubelet, ip-10-10-174-153 Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "e2887e58a5bf1982c52651a179ea340c37a9a525363c786bfe3627544de7a847": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Warning Unhealthy 100s kubelet, ip-10-10-174-153 Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "d09c1f55e2cd9e61114899973b86463bfb548f00b8cecf6d182644b156367feb": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Warning Unhealthy 99s kubelet, ip-10-10-174-153 Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "e97e622700b488f16386859e5f51b949117cf7a14d71ae490fe5a31e3b727efe": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Warning Unhealthy 98s kubelet, ip-10-10-174-153 Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "8a8e1ea13a9f54f6ea720391fd9ae5c1f57f7156430bb80ea4ad5ff9fdf080af": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Warning Unhealthy 91s (x7 over 97s) kubelet, ip-10-10-174-153 (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "539a04d150cd1f1563cdd79e3c890d188de2943336e2c76eb6b5c77d54d904c2": OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "apparmor failed to apply profile: write /proc/self/attr/exec: operation not permitted": unknown
Issue-Label Bot is automatically applying the labels:
| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.57 |
Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.
issue-label-bot[bot]
on 22 May 2020
👎1
/kind example
sakaia
on 22 May 2020
@sakaia: The label(s) kind/example cannot be applied, because the repository doesn't have them
In response to this:
/kind example
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
k8s-ci-robot
on 22 May 2020
👍1
@sakaia looks similar to this issue https://github.com/ubuntu/microk8s/issues/784 ?
yuzisun
on 22 May 2020
👍1
@yuzisun Thank you for your suggestion.
I passed to following but still have a problem for curl command.
Additional Command from ubuntu/microk8s#784
Workaround command.
sudo apparmor_parser -R /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-containerd
sudo apparmor_parser -a /var/lib/snapd/apparmor/profiles/snap.microk8s.daemon-containerd
Current status.
$ kubectl get inferenceservices -n kfserving-test
NAME URL READY DEFAULT TRAFFIC CANARY TRAFFIC AGE
sklearn-iris http://sklearn-iris.kfserving-test.example.com/v1/models/sklearn-iris True 100 8m57s
From seeing error message, kfserving controller is not registered.
Any suggestion?
Ref.
Test KFServing Installation
https://github.com/kubeflow/kfserving
$ curl -v -H "Host: ${SERVICE_HOSTNAME}" http://localhost:8080/v1/models/sklearn-iris:predict -d @./docs/samples/sklearn/iris-input.json
* Trying 127.0.0.1:8080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kfserving-test.example.com
> User-Agent: curl/7.65.3
> Accept: */*
> Content-Length: 76
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 76 out of 76 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: application/json
< Date: Mon, 25 May 2020 07:31:46 GMT
< Content-Length: 487
<
{
"paths": [
"/apis",
"/apis/",
"/apis/apiextensions.k8s.io",
"/apis/apiextensions.k8s.io/v1beta1",
"/healthz",
"/healthz/etcd",
"/healthz/log",
"/healthz/ping",
"/healthz/poststarthook/crd-informer-synced",
"/healthz/poststarthook/generic-apiserver-start-informers",
"/healthz/poststarthook/start-apiextensions-controllers",
"/healthz/poststarthook/start-apiextensions-informers",
"/metrics",
"/openapi/v2",
"/version"
]
* Connection #0 to host localhost left intact
@sakaia Can you try the following config
kubectl edit cm inferenceservice-config -n kubeflowand set"ingressService" : "istio-ingressgateway.istio-system.svc.cluster.local"kubectl edit cm config-istio -n knative-servingand change
gateway.knative-serving.knative-ingress-gatewaytogateway.kubeflow.kubeflow-gateway: "istio-ingressgateway.istio-system.svc.cluster.local"
yuzisun
on 26 May 2020
consolidate to https://github.com/kubeflow/kfserving/issues/798
/close
yuzisun
on 22 Jun 2020
@yuzisun: Closing this issue.
In response to this:
consolidate to https://github.com/kubeflow/kfserving/issues/798
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
k8s-ci-robot
on 22 Jun 2020
@yuzisun Same issue, with default kubeflow installation on Google Cloud. This solution did not work for me.
sml0820
on 10 Jul 2020
@yuzisun Same issue, with default kubeflow installation on Google Cloud. This solution did not work for me.
@sml0820 check my instruction in last comment here https://github.com/kubeflow/kfserving/issues/798, it is going to be fixed in kubeflow 1.1
yuzisun
on 10 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
mokpolar
路
6Comments
mokpolar
路
5Comments
decewei
路
3Comments
yuzliu
路
6Comments
ace-gabriel
路
3Comments