Kfserving: Unable to use KFServing on Kubeflow

Created on 1 Jul 2020  路  46Comments  路  Source: kubeflow/kfserving

/kind bug

What steps did you take and what happened:
I was trying to use KFServing for Model Prediction as described here.

MODEL_NAME=flowers-sample
INPUT_PATH=@./input.json
INGRESS_GATEWAY=istio-ingressgateway
CLUSTER_IP=$(kubectl -n istio-system get service $INGRESS_GATEWAY -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
SERVICE_HOSTNAME=$(kubectl get inferenceservice ${MODEL_NAME} -o jsonpath='{.status.url}' | cut -d "/" -f 3)

I realize that CLUSTER_IP & SERVICE_HOSTNAME are blank. On checking the JSON file, it can be figured out that the jsonpath specified for CLUSTER_IP has been modified to spec.clusterIP, whereas the SERVICE_HOSTNAME continues to remain blank leading to failure of the steps that follow.

What did you expect to happen:
The commands to execute successfully & output a model prediction.

Anything else you would like to add:
I tried to hardcode the SERVICE_HOSTNAME to flowers-sample.default.example.com but that led to the connection time out error.

Environment:

  • Istio Version:
  • Knative Version:
  • KFServing Version:
  • Kubeflow version: v1.0.2
  • Kfdef: k8s_istio
  • Minikube version:
  • Kubernetes version: (use kubectl version): Client Version: 1.18 Server Version: 1.15
  • OS (e.g. from /etc/os-release): Ubuntu 18.04
areinference kinbug
Source
picture Manvi07

All 46 comments

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.74 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] picture issue-label-bot[bot] on 1 Jul 2020

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/inference | 0.74 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

issue-label-bot[bot] picture issue-label-bot[bot] on 1 Jul 2020

@Manvi07 Can you follow the debug guide to see if that can help you figure out the problem ?

yuzisun picture yuzisun on 1 Jul 2020
👍1

Hi @yuzisun
Thanks for sharing the debug guide. I get the RevisionMissing error.
But on running the command kubectl get pod -l model=sklearn-iris, it says no resource found, whereas the pod exists and is running, as I can see here:

kubectl get pods
NAME                                                              READY   STATUS                  RESTARTS   AGE
flowers-sample-predictor-default-grlzq-deployment-6b788b85xnfxg   2/2     Running                 0          5d5h
hello-1593600420-5vfhf                                            0/1     Completed               0          2m49s
hello-1593600480-t5w9c                                            0/1     Completed               0          109s
hello-1593600540-gpb9w                                            0/1     Completed               0          48s
nuclio-controller-768f665b5f-s4zjz                                1/1     Running                 0          6d2h
nuclio-dashboard-586564dcdb-92gsc                                 1/1     Running                 0          6d2h
pytorch-cifar10-predictor-default-gt8v5-deployment-5b85f75qpngj   2/2     Running                 0          6d
sklearn-iris-predictor-default-gqt6h-deployment-578d88798-gh4vt   2/2     Running                 0          8d

How is that the label model=sklearn-iris is missing? :confused:

Manvi07 picture Manvi07 on 1 Jul 2020

@Manvi07 I think you might be on older version of KFServing which does not have this label, but your pod looks good though.
can you paste the output of kubectl get revision $(kubectl get configuration sklearn-iris-predictor-default --output jsonpath="{.status.latestCreatedRevisionName}") -oyaml ?

yuzisun picture yuzisun on 1 Jul 2020

@yuzisun here it is:

kubectl get revision $(kubectl get configuration sklearn-iris-predictor-default --output jsonpath="{.status.latestCreatedRevisionName}") -oyaml
apiVersion: serving.knative.dev/v1
kind: Revision
metadata:
  annotations:
    autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
    autoscaling.knative.dev/target: "1"
    internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/sklearn/iris
    queue.sidecar.serving.knative.dev/resourcePercentage: "0.2"
    serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
  creationTimestamp: "2020-06-23T05:06:32Z"
  generateName: sklearn-iris-predictor-default-
  generation: 1
  labels:
    serving.knative.dev/configuration: sklearn-iris-predictor-default
    serving.knative.dev/configurationGeneration: "1"
    serving.knative.dev/service: sklearn-iris-predictor-default
    serving.kubeflow.org/inferenceservice: sklearn-iris
  name: sklearn-iris-predictor-default-gqt6h
  namespace: default
  ownerReferences:
  - apiVersion: serving.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Configuration
    name: sklearn-iris-predictor-default
    uid: 388c1d42-c9d4-44dc-bc35-374cefdbe0ce
  resourceVersion: "25490270"
  selfLink: /apis/serving.knative.dev/v1/namespaces/default/revisions/sklearn-iris-predictor-default-gqt6h
  uid: 1a541a44-5cff-4ac2-8eaa-d2c37ac6d49b
spec:
  containerConcurrency: 0
  containers:
  - args:
    - --model_name=sklearn-iris
    - --model_dir=/mnt/models
    - --http_port=8080
    image: gcr.io/kfserving/sklearnserver:0.2.2
    name: kfserving-container
    readinessProbe:
      successThreshold: 1
      tcpSocket:
        port: 0
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
  timeoutSeconds: 60
status:
  conditions:
  - lastTransitionTime: "2020-06-23T05:06:32Z"
    reason: Deploying
    severity: Info
    status: Unknown
    type: Active
  - lastTransitionTime: "2020-06-23T05:06:32Z"
    reason: Deploying
    status: Unknown
    type: ContainerHealthy
  - lastTransitionTime: "2020-06-23T05:06:32Z"
    reason: Deploying
    status: Unknown
    type: Ready
  - lastTransitionTime: "2020-06-23T05:06:32Z"
    reason: Deploying
    status: Unknown
    type: ResourcesAvailable
  imageDigest: gcr.io/kfserving/sklearnserver@sha256:6702dddff99de5b8945f35d913a423e406b6cd039a14cefdbd1886078d393ce5
  logUrl: http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana#/discover?_a=(query:(match:(kubernetes.labels.knative-dev%2FrevisionUID:(query:'1a541a44-5cff-4ac2-8eaa-d2c37ac6d49b',type:phrase))))
  observedGeneration: 1
Manvi07 picture Manvi07 on 1 Jul 2020

@Manvi07 seems like hitting this issue https://github.com/knative/serving/issues/6265. Can you try redeploying the inference service?

yuzisun picture yuzisun on 1 Jul 2020

@yuzisun I deleted the existing inference service and deployed it again, but it still says revision missing:

kubectl apply -f sklearn.yaml
inferenceservice.serving.kubeflow.org/sklearn-iris created
manvi@manvi-pc:~/K8_cluster/kfserving/docs/samples/sklearn$ kubectl get ksvc
NAME                                URL                                                            LATESTCREATED                             LATESTREADY   READY     REASON
flowers-sample-predictor-default    http://flowers-sample-predictor-default.default.example.com    flowers-sample-predictor-default-grlzq                  Unknown   RevisionMissing
pytorch-cifar10-predictor-default   http://pytorch-cifar10-predictor-default.default.example.com   pytorch-cifar10-predictor-default-gt8v5                 Unknown   RevisionMissing
sklearn-iris-predictor-default      http://sklearn-iris-predictor-default.default.example.com      sklearn-iris-predictor-default-9fd7c                    Unknown   RevisionMissing
style-sample-predictor-default      http://style-sample-predictor-default.default.example.com      style-sample-predictor-default-qrh8c                    False     RevisionMissing
Manvi07 picture Manvi07 on 1 Jul 2020

@Manvi07 can you share the output of kubectl get knative-internal according to the instruction of above knative issue?

yuzisun picture yuzisun on 1 Jul 2020

@yuzisun I am new to Knative, and many terms discussed in the above issue are not known to me.
Here's the output, as asked:

kubectl get knative-internal
NAME                                                                                     DESIREDSCALE   ACTUALSCALE   READY   REASON
podautoscaler.autoscaling.internal.knative.dev/flowers-sample-predictor-default-grlzq                                         
podautoscaler.autoscaling.internal.knative.dev/pytorch-cifar10-predictor-default-gt8v5                                        
podautoscaler.autoscaling.internal.knative.dev/sklearn-iris-predictor-default-9fd7c                                           
podautoscaler.autoscaling.internal.knative.dev/style-sample-predictor-default-qrh8c                                           

NAME                                                                               AGE
image.caching.internal.knative.dev/flowers-sample-predictor-default-grlzq-cache    5d6h
image.caching.internal.knative.dev/pytorch-cifar10-predictor-default-gt8v5-cache   6d1h
image.caching.internal.knative.dev/sklearn-iris-predictor-default-9fd7c-cache      18m
image.caching.internal.knative.dev/style-sample-predictor-default-qrh8c-cache      5d6h
Manvi07 picture Manvi07 on 1 Jul 2020

The latest output looks like the PodAutoscalers are never picked up.

Can you make sure that Knative's control plane is running correctly? The output of kubectl get pods -n knative-serving should do the trick there.

Also the full YAML output of the podautoscaler resources would be handy. kubectl get podautoscaler.autoscaling.internal.knative.dev -oyaml should give us what we need.

markusthoemmes picture markusthoemmes on 1 Jul 2020

Hi @markusthoemmes
Yeah, I see the control plane running correctly

kubectl get pods -n knative-serving
NAME                                READY   STATUS    RESTARTS   AGE
autoscaler-hpa-68cc87bfb9-rndz7     1/1     Running   0          21d
controller-95dc7f8bd-6xbqr          1/1     Running   0          21d
networking-istio-5b8c5c6cff-x64ch   1/1     Running   0          21d
webhook-67847fb4b5-z6rf2            1/1     Running   0          21d

Here's the output of podautoscaler:

kubectl get podautoscaler.autoscaling.internal.knative.dev -oyaml
apiVersion: v1
items:
- apiVersion: autoscaling.internal.knative.dev/v1alpha1
  kind: PodAutoscaler
  metadata:
    annotations:
      autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
      autoscaling.knative.dev/metric: concurrency
      autoscaling.knative.dev/target: "1"
      internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/tensorflow/flowers
      queue.sidecar.serving.knative.dev/resourcePercentage: "0.2"
      serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
    creationTimestamp: "2020-06-26T05:49:22Z"
    generation: 2
    labels:
      app: flowers-sample-predictor-default-grlzq
      serving.knative.dev/configuration: flowers-sample-predictor-default
      serving.knative.dev/configurationGeneration: "1"
      serving.knative.dev/revision: flowers-sample-predictor-default-grlzq
      serving.knative.dev/revisionUID: 320acf72-5f85-45ed-a358-438ef2a65f8d
      serving.knative.dev/service: flowers-sample-predictor-default
      serving.kubeflow.org/inferenceservice: flowers-sample
    name: flowers-sample-predictor-default-grlzq
    namespace: default
    ownerReferences:
    - apiVersion: serving.knative.dev/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Revision
      name: flowers-sample-predictor-default-grlzq
      uid: 320acf72-5f85-45ed-a358-438ef2a65f8d
    resourceVersion: "28781961"
    selfLink: /apis/autoscaling.internal.knative.dev/v1alpha1/namespaces/default/podautoscalers/flowers-sample-predictor-default-grlzq
    uid: 7b592c5d-214a-496f-891d-79e69cb8d735
  spec:
    protocolType: http1
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: flowers-sample-predictor-default-grlzq-deployment
- apiVersion: autoscaling.internal.knative.dev/v1alpha1
  kind: PodAutoscaler
  metadata:
    annotations:
      autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
      autoscaling.knative.dev/metric: concurrency
      autoscaling.knative.dev/target: "1"
      internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/pytorch/cifar10/
      queue.sidecar.serving.knative.dev/resourcePercentage: "0.2"
      serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
    creationTimestamp: "2020-06-25T10:16:14Z"
    generation: 2
    labels:
      app: pytorch-cifar10-predictor-default-gt8v5
      serving.knative.dev/configuration: pytorch-cifar10-predictor-default
      serving.knative.dev/configurationGeneration: "1"
      serving.knative.dev/revision: pytorch-cifar10-predictor-default-gt8v5
      serving.knative.dev/revisionUID: 61a654c3-da28-4996-bda7-b17d25f4db39
      serving.knative.dev/service: pytorch-cifar10-predictor-default
      serving.kubeflow.org/inferenceservice: pytorch-cifar10
    name: pytorch-cifar10-predictor-default-gt8v5
    namespace: default
    ownerReferences:
    - apiVersion: serving.knative.dev/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Revision
      name: pytorch-cifar10-predictor-default-gt8v5
      uid: 61a654c3-da28-4996-bda7-b17d25f4db39
    resourceVersion: "27889148"
    selfLink: /apis/autoscaling.internal.knative.dev/v1alpha1/namespaces/default/podautoscalers/pytorch-cifar10-predictor-default-gt8v5
    uid: 762c2db7-aa20-47f3-9db2-0aa81d3de1ff
  spec:
    protocolType: http1
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: pytorch-cifar10-predictor-default-gt8v5-deployment
- apiVersion: autoscaling.internal.knative.dev/v1alpha1
  kind: PodAutoscaler
  metadata:
    annotations:
      autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
      autoscaling.knative.dev/metric: concurrency
      autoscaling.knative.dev/target: "1"
      internal.serving.kubeflow.org/storage-initializer-sourceuri: gs://kfserving-samples/models/sklearn/iris
      queue.sidecar.serving.knative.dev/resourcePercentage: "0.2"
      serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
    creationTimestamp: "2020-07-01T11:53:20Z"
    generation: 2
    labels:
      app: sklearn-iris-predictor-default-9fd7c
      serving.knative.dev/configuration: sklearn-iris-predictor-default
      serving.knative.dev/configurationGeneration: "1"
      serving.knative.dev/revision: sklearn-iris-predictor-default-9fd7c
      serving.knative.dev/revisionUID: 5d14ce6d-740e-406b-a4fd-d4e908e1f0a0
      serving.knative.dev/service: sklearn-iris-predictor-default
      serving.kubeflow.org/inferenceservice: sklearn-iris
    name: sklearn-iris-predictor-default-9fd7c
    namespace: default
    ownerReferences:
    - apiVersion: serving.knative.dev/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Revision
      name: sklearn-iris-predictor-default-9fd7c
      uid: 5d14ce6d-740e-406b-a4fd-d4e908e1f0a0
    resourceVersion: "34236585"
    selfLink: /apis/autoscaling.internal.knative.dev/v1alpha1/namespaces/default/podautoscalers/sklearn-iris-predictor-default-9fd7c
    uid: e87a1d53-fbfd-44b8-bbca-69fb6d80fce0
  spec:
    protocolType: http1
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: sklearn-iris-predictor-default-9fd7c-deployment
- apiVersion: autoscaling.internal.knative.dev/v1alpha1
  kind: PodAutoscaler
  metadata:
    annotations:
      autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
      autoscaling.knative.dev/metric: concurrency
      autoscaling.knative.dev/target: "1"
      internal.serving.kubeflow.org/storage-initializer-sourceuri: https://kfserving.blob.core.windows.net/examples/onnx/style/model.onnx
      queue.sidecar.serving.knative.dev/resourcePercentage: "0.2"
      serving.knative.dev/creator: system:serviceaccount:kfserving-system:default
    creationTimestamp: "2020-06-26T05:45:28Z"
    generation: 3
    labels:
      app: style-sample-predictor-default-qrh8c
      serving.knative.dev/configuration: style-sample-predictor-default
      serving.knative.dev/configurationGeneration: "1"
      serving.knative.dev/revision: style-sample-predictor-default-qrh8c
      serving.knative.dev/revisionUID: 102e6c35-8797-4f2f-bb07-e11724542790
      serving.knative.dev/service: style-sample-predictor-default
      serving.kubeflow.org/inferenceservice: style-sample
    name: style-sample-predictor-default-qrh8c
    namespace: default
    ownerReferences:
    - apiVersion: serving.knative.dev/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Revision
      name: style-sample-predictor-default-qrh8c
      uid: 102e6c35-8797-4f2f-bb07-e11724542790
    resourceVersion: "28780613"
    selfLink: /apis/autoscaling.internal.knative.dev/v1alpha1/namespaces/default/podautoscalers/style-sample-predictor-default-qrh8c
    uid: 6ca3c633-a25d-4994-a206-b9100040e357
  spec:
    protocolType: http1
    reachability: Unreachable
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: style-sample-predictor-default-qrh8c-deployment
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
Manvi07 picture Manvi07 on 1 Jul 2020

The control-plane is definitely lacking an autoscaler. You only have autoscaler-hpa which is only an extension. The autoscaler will be responsible to pick these resources up.

markusthoemmes picture markusthoemmes on 1 Jul 2020

@Manvi07 Can you help get the output of kubectl get deployment autoscaler -n knative-serving -oyaml ?

yuzisun picture yuzisun on 1 Jul 2020

@markusthoemmes, I think it has timed out, as the below message states.
@yuzisun Here it is:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative","serving.knative.dev/release":"v0.11.1"},"name":"autoscaler","namespace":"knative-serving"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"autoscaler","app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative"}},"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict":"false","sidecar.istio.io/inject":"true","traffic.sidecar.istio.io/includeInboundPorts":"8080,9090"},"labels":{"app":"autoscaler","app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative","serving.knative.dev/release":"v0.11.1"}},"spec":{"containers":[{"args":["--secure-port=8443","--cert-dir=/tmp"],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/serving"}],"image":"gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler@sha256:ef1f01b5fb3886d4c488a219687aac72d28e72f808691132f658259e4e02bb27","livenessProbe":{"httpGet":{"httpHeaders":[{"name":"k-kubelet-probe","value":"autoscaler"}],"path":"/healthz","port":8080}},"name":"autoscaler","ports":[{"containerPort":8080,"name":"websocket"},{"containerPort":9090,"name":"metrics"},{"containerPort":8443,"name":"custom-metrics"},{"containerPort":8008,"name":"profiling"}],"readinessProbe":{"httpGet":{"httpHeaders":[{"name":"k-kubelet-probe","value":"autoscaler"}],"path":"/healthz","port":8080}},"resources":{"limits":{"cpu":"300m","memory":"400Mi"},"requests":{"cpu":"30m","memory":"40Mi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"serviceAccountName":"controller"}}}}
  creationTimestamp: "2020-05-28T10:30:50Z"
  generation: 1
  labels:
    app.kubernetes.io/component: knative-serving-install
    app.kubernetes.io/instance: knative-serving-install-v0.11.1
    app.kubernetes.io/managed-by: kfctl
    app.kubernetes.io/name: knative-serving-install
    app.kubernetes.io/part-of: kubeflow
    app.kubernetes.io/version: v0.11.1
    kustomize.component: knative
    serving.knative.dev/release: v0.11.1
  name: autoscaler
  namespace: knative-serving
  ownerReferences:
  - apiVersion: app.k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: false
    kind: Application
    name: knative-serving-install
    uid: 9114a3e9-6d31-4413-8830-f45a0b8da0cd
  resourceVersion: "76848"
  selfLink: /apis/extensions/v1beta1/namespaces/knative-serving/deployments/autoscaler
  uid: ee76848b-b5a6-4f66-9bff-3e216726a7b0
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: autoscaler
      app.kubernetes.io/component: knative-serving-install
      app.kubernetes.io/instance: knative-serving-install-v0.11.1
      app.kubernetes.io/managed-by: kfctl
      app.kubernetes.io/name: knative-serving-install
      app.kubernetes.io/part-of: kubeflow
      app.kubernetes.io/version: v0.11.1
      kustomize.component: knative
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
        sidecar.istio.io/inject: "true"
        traffic.sidecar.istio.io/includeInboundPorts: 8080,9090
      creationTimestamp: null
      labels:
        app: autoscaler
        app.kubernetes.io/component: knative-serving-install
        app.kubernetes.io/instance: knative-serving-install-v0.11.1
        app.kubernetes.io/managed-by: kfctl
        app.kubernetes.io/name: knative-serving-install
        app.kubernetes.io/part-of: kubeflow
        app.kubernetes.io/version: v0.11.1
        kustomize.component: knative
        serving.knative.dev/release: v0.11.1
    spec:
      containers:
      - args:
        - --secure-port=8443
        - --cert-dir=/tmp
        env:
        - name: SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CONFIG_LOGGING_NAME
          value: config-logging
        - name: CONFIG_OBSERVABILITY_NAME
          value: config-observability
        - name: METRICS_DOMAIN
          value: knative.dev/serving
        image: gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler@sha256:ef1f01b5fb3886d4c488a219687aac72d28e72f808691132f658259e4e02bb27
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: k-kubelet-probe
              value: autoscaler
            path: /healthz
            port: 8080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: autoscaler
        ports:
        - containerPort: 8080
          name: websocket
          protocol: TCP
        - containerPort: 9090
          name: metrics
          protocol: TCP
        - containerPort: 8443
          name: custom-metrics
          protocol: TCP
        - containerPort: 8008
          name: profiling
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: k-kubelet-probe
              value: autoscaler
            path: /healthz
            port: 8080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 300m
            memory: 400Mi
          requests:
            cpu: 30m
            memory: 40Mi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: controller
      serviceAccountName: controller
      terminationGracePeriodSeconds: 30
status:
  conditions:
  - lastTransitionTime: "2020-05-28T10:30:50Z"
    lastUpdateTime: "2020-05-28T10:30:50Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2020-05-28T10:30:50Z"
    lastUpdateTime: "2020-05-28T10:30:50Z"
    message: 'pods "autoscaler-69bcc99c79-" is forbidden: unable to validate against
      any pod security policy: [spec.initContainers[0].securityContext.capabilities.add:
      Invalid value: "NET_ADMIN": capability may not be added]'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure
  - lastTransitionTime: "2020-05-28T10:40:51Z"
    lastUpdateTime: "2020-05-28T10:40:51Z"
    message: ReplicaSet "autoscaler-69bcc99c79" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 1
  unavailableReplicas: 1
Manvi07 picture Manvi07 on 1 Jul 2020

Ah there you go. I'm assuming you're running with injected istio sidecars? The security settings of your cluster are not allowing the istio sidecars to be created for you. The activator is also not there for the same reason.

markusthoemmes picture markusthoemmes on 1 Jul 2020
😕1

@markusthoemmes I am not aware of istio-sidecars injection. Is there a way I can set the activator?

Manvi07 picture Manvi07 on 1 Jul 2020

Actually I am using an external cluster, that is a part of an enterprise.

Manvi07 picture Manvi07 on 1 Jul 2020

@Manvi07 if you remove the annotation "sidecar.istio.io/inject":"true" on both autoscaler and activator deployment, the pod should then come up.

yuzisun picture yuzisun on 1 Jul 2020
👍1

@yuzisun I modified the autoscaler and activator deployments, and redeployed the inference service, but ksvc output still shows revision missing.

Manvi07 picture Manvi07 on 1 Jul 2020

@Manvi07 did you see the autoscaler and activator pod coming up now in knative-serving namespace?

yuzisun picture yuzisun on 1 Jul 2020

No @yuzisun, still not coming up :(

Manvi07 picture Manvi07 on 1 Jul 2020

No @yuzisun, still not coming up :(

Can you confirm if the annotation is removed from knative activator and autoscaler deployment?

yuzisun picture yuzisun on 1 Jul 2020

@yuzisun 

kubectl get deployment autoscaler -n knative-serving -oyaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative","serving.knative.dev/release":"v0.11.1"},"name":"autoscaler","namespace":"knative-serving"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"autoscaler","app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative"}},"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict":"false","traffic.sidecar.istio.io/includeInboundPorts":"8080,9090"},"labels":{"app":"autoscaler","app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative","serving.knative.dev/release":"v0.11.1"}},"spec":{"containers":[{"args":["--secure-port=8443","--cert-dir=/tmp"],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/serving"}],"image":"gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler@sha256:ef1f01b5fb3886d4c488a219687aac72d28e72f808691132f658259e4e02bb27","livenessProbe":{"httpGet":{"httpHeaders":[{"name":"k-kubelet-probe","value":"autoscaler"}],"path":"/healthz","port":8080}},"name":"autoscaler","ports":[{"containerPort":8080,"name":"websocket"},{"containerPort":9090,"name":"metrics"},{"containerPort":8443,"name":"custom-metrics"},{"containerPort":8008,"name":"profiling"}],"readinessProbe":{"httpGet":{"httpHeaders":[{"name":"k-kubelet-probe","value":"autoscaler"}],"path":"/healthz","port":8080}},"resources":{"limits":{"cpu":"300m","memory":"400Mi"},"requests":{"cpu":"30m","memory":"40Mi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"serviceAccountName":"controller"}}}}
  creationTimestamp: "2020-05-28T10:30:50Z"
  generation: 3
  labels:
    app.kubernetes.io/component: knative-serving-install
    app.kubernetes.io/instance: knative-serving-install-v0.11.1
    app.kubernetes.io/managed-by: kfctl
    app.kubernetes.io/name: knative-serving-install
    app.kubernetes.io/part-of: kubeflow
    app.kubernetes.io/version: v0.11.1
    kustomize.component: knative
    serving.knative.dev/release: v0.11.1
  name: autoscaler
  namespace: knative-serving
  ownerReferences:
  - apiVersion: app.k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: false
    kind: Application
    name: knative-serving-install
    uid: 9114a3e9-6d31-4413-8830-f45a0b8da0cd
  resourceVersion: "34333425"
  selfLink: /apis/extensions/v1beta1/namespaces/knative-serving/deployments/autoscaler
  uid: ee76848b-b5a6-4f66-9bff-3e216726a7b0
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: autoscaler
      app.kubernetes.io/component: knative-serving-install
      app.kubernetes.io/instance: knative-serving-install-v0.11.1
      app.kubernetes.io/managed-by: kfctl
      app.kubernetes.io/name: knative-serving-install
      app.kubernetes.io/part-of: kubeflow
      app.kubernetes.io/version: v0.11.1
      kustomize.component: knative
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
        traffic.sidecar.istio.io/includeInboundPorts: 8080,9090
      creationTimestamp: null
      labels:
        app: autoscaler
        app.kubernetes.io/component: knative-serving-install
        app.kubernetes.io/instance: knative-serving-install-v0.11.1
        app.kubernetes.io/managed-by: kfctl
        app.kubernetes.io/name: knative-serving-install
        app.kubernetes.io/part-of: kubeflow
        app.kubernetes.io/version: v0.11.1
        kustomize.component: knative
        serving.knative.dev/release: v0.11.1
    spec:
      containers:
      - args:
        - --secure-port=8443
        - --cert-dir=/tmp
        env:
        - name: SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CONFIG_LOGGING_NAME
          value: config-logging
        - name: CONFIG_OBSERVABILITY_NAME
          value: config-observability
        - name: METRICS_DOMAIN
          value: knative.dev/serving
        image: gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler@sha256:ef1f01b5fb3886d4c488a219687aac72d28e72f808691132f658259e4e02bb27
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: k-kubelet-probe
              value: autoscaler
            path: /healthz
            port: 8080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: autoscaler
        ports:
        - containerPort: 8080
          name: websocket
          protocol: TCP
        - containerPort: 9090
          name: metrics
          protocol: TCP
        - containerPort: 8443
          name: custom-metrics
          protocol: TCP
        - containerPort: 8008
          name: profiling
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: k-kubelet-probe
              value: autoscaler
            path: /healthz
            port: 8080
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 300m
            memory: 400Mi
          requests:
            cpu: 30m
            memory: 40Mi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: controller
      serviceAccountName: controller
      terminationGracePeriodSeconds: 30
status:
  conditions:
  - lastTransitionTime: "2020-05-28T10:30:50Z"
    lastUpdateTime: "2020-05-28T10:30:50Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2020-05-28T10:30:50Z"
    lastUpdateTime: "2020-05-28T10:30:50Z"
    message: 'pods "autoscaler-69bcc99c79-" is forbidden: unable to validate against
      any pod security policy: [spec.initContainers[0].securityContext.capabilities.add:
      Invalid value: "NET_ADMIN": capability may not be added]'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure
  - lastTransitionTime: "2020-07-01T14:04:47Z"
    lastUpdateTime: "2020-07-01T14:04:47Z"
    message: ReplicaSet "autoscaler-77d44b8884" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 3
  unavailableReplicas: 2
Manvi07 picture Manvi07 on 1 Jul 2020 
kubectl get deployment activator -n knative-serving -oyaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative","serving.knative.dev/release":"v0.11.1"},"name":"activator","namespace":"knative-serving"},"spec":{"selector":{"matchLabels":{"app":"activator","app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative","role":"activator"}},"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict":"false"},"labels":{"app":"activator","app.kubernetes.io/component":"knative-serving-install","app.kubernetes.io/instance":"knative-serving-install-v0.11.1","app.kubernetes.io/managed-by":"kfctl","app.kubernetes.io/name":"knative-serving-install","app.kubernetes.io/part-of":"kubeflow","app.kubernetes.io/version":"v0.11.1","kustomize.component":"knative","role":"activator","serving.knative.dev/release":"v0.11.1"}},"spec":{"containers":[{"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/internal/serving"}],"image":"gcr.io/knative-releases/knative.dev/serving/cmd/activator@sha256:8e606671215cc029683e8cd633ec5de9eabeaa6e9a4392ff289883304be1f418","livenessProbe":{"httpGet":{"httpHeaders":[{"name":"k-kubelet-probe","value":"activator"}],"path":"/healthz","port":8012}},"name":"activator","ports":[{"containerPort":8012,"name":"http1"},{"containerPort":8013,"name":"h2c"},{"containerPort":9090,"name":"metrics"},{"containerPort":8008,"name":"profiling"}],"readinessProbe":{"httpGet":{"httpHeaders":[{"name":"k-kubelet-probe","value":"activator"}],"path":"/healthz","port":8012}},"resources":{"limits":{"cpu":"1000m","memory":"600Mi"},"requests":{"cpu":"300m","memory":"60Mi"}},"securityContext":{"allowPrivilegeEscalation":false}}],"serviceAccountName":"controller","terminationGracePeriodSeconds":300}}}}
  creationTimestamp: "2020-05-28T10:30:49Z"
  generation: 2
  labels:
    app.kubernetes.io/component: knative-serving-install
    app.kubernetes.io/instance: knative-serving-install-v0.11.1
    app.kubernetes.io/managed-by: kfctl
    app.kubernetes.io/name: knative-serving-install
    app.kubernetes.io/part-of: kubeflow
    app.kubernetes.io/version: v0.11.1
    kustomize.component: knative
    serving.knative.dev/release: v0.11.1
  name: activator
  namespace: knative-serving
  ownerReferences:
  - apiVersion: app.k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: false
    kind: Application
    name: knative-serving-install
    uid: 9114a3e9-6d31-4413-8830-f45a0b8da0cd
  resourceVersion: "34336932"
  selfLink: /apis/extensions/v1beta1/namespaces/knative-serving/deployments/activator
  uid: d785b4e3-5060-410c-90b5-025d24669350
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: activator
      app.kubernetes.io/component: knative-serving-install
      app.kubernetes.io/instance: knative-serving-install-v0.11.1
      app.kubernetes.io/managed-by: kfctl
      app.kubernetes.io/name: knative-serving-install
      app.kubernetes.io/part-of: kubeflow
      app.kubernetes.io/version: v0.11.1
      kustomize.component: knative
      role: activator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
      creationTimestamp: null
      labels:
        app: activator
        app.kubernetes.io/component: knative-serving-install
        app.kubernetes.io/instance: knative-serving-install-v0.11.1
        app.kubernetes.io/managed-by: kfctl
        app.kubernetes.io/name: knative-serving-install
        app.kubernetes.io/part-of: kubeflow
        app.kubernetes.io/version: v0.11.1
        kustomize.component: knative
        role: activator
        serving.knative.dev/release: v0.11.1
    spec:
      containers:
      - env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: SYSTEM_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CONFIG_LOGGING_NAME
          value: config-logging
        - name: CONFIG_OBSERVABILITY_NAME
          value: config-observability
        - name: METRICS_DOMAIN
          value: knative.dev/internal/serving
        image: gcr.io/knative-releases/knative.dev/serving/cmd/activator@sha256:8e606671215cc029683e8cd633ec5de9eabeaa6e9a4392ff289883304be1f418
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: k-kubelet-probe
              value: activator
            path: /healthz
            port: 8012
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: activator
        ports:
        - containerPort: 8012
          name: http1
          protocol: TCP
        - containerPort: 8013
          name: h2c
          protocol: TCP
        - containerPort: 9090
          name: metrics
          protocol: TCP
        - containerPort: 8008
          name: profiling
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: k-kubelet-probe
              value: activator
            path: /healthz
            port: 8012
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
            memory: 600Mi
          requests:
            cpu: 300m
            memory: 60Mi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: controller
      serviceAccountName: controller
      terminationGracePeriodSeconds: 300
status:
  conditions:
  - lastTransitionTime: "2020-05-28T10:30:49Z"
    lastUpdateTime: "2020-05-28T10:30:49Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2020-05-28T10:30:49Z"
    lastUpdateTime: "2020-05-28T10:30:49Z"
    message: 'pods "activator-6dc4884-" is forbidden: unable to validate against any
      pod security policy: [spec.initContainers[0].securityContext.capabilities.add:
      Invalid value: "NET_ADMIN": capability may not be added]'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure
  - lastTransitionTime: "2020-07-01T14:09:46Z"
    lastUpdateTime: "2020-07-01T14:09:46Z"
    message: ReplicaSet "activator-57cc9fcb85" has timed out progressing.
    reason: ProgressDeadlineExceeded
    status: "False"
    type: Progressing
  observedGeneration: 2
  unavailableReplicas: 2
Manvi07 picture Manvi07 on 1 Jul 2020

@Manvi07 Can you delete the deployment and try deploying again?

yuzisun picture yuzisun on 1 Jul 2020

@yuzisun You mean deleting the autoscaler & activator deployments?

Manvi07 picture Manvi07 on 1 Jul 2020

@yuzisun You mean deleting the autoscaler & activator deployments?

yes

yuzisun picture yuzisun on 1 Jul 2020

@yuzisun, didn't work. The pods not showing up still. :(
Here are the logs.

manvi@manvi-pc:~/K8_cluster$ kubectl delete deployment activator -n knative-serving
deployment.extensions "activator" deleted
manvi@manvi-pc:~/K8_cluster$ kubectl apply -f activator.yaml -n knative-serving
deployment.extensions/activator created
manvi@manvi-pc:~/K8_cluster$ kubectl delete deployment autoscaler -n knative-serving
deployment.extensions "autoscaler" deleted
manvi@manvi-pc:~/K8_cluster$ kubectl apply -f autoscaler.yaml -n knative-serving
deployment.extensions/autoscaler created
manvi@manvi-pc:~/K8_cluster$ kubectl get pods -n knative-serving
NAME                                READY   STATUS    RESTARTS   AGE
autoscaler-hpa-68cc87bfb9-rndz7     1/1     Running   0          22d
controller-95dc7f8bd-6xbqr          1/1     Running   0          22d
networking-istio-5b8c5c6cff-x64ch   1/1     Running   0          22d
webhook-67847fb4b5-z6rf2            1/1     Running   0          22d
Manvi07 picture Manvi07 on 1 Jul 2020

@Manvi07 so activator.yaml does not have the istio injector annotation now? you might want to check the deployment status for activator and autoscaler to see if still the same error as before.

yuzisun picture yuzisun on 1 Jul 2020

@Manvi07 you can also try explicitly turning off the sidecar like in this PR.

yuzisun picture yuzisun on 1 Jul 2020

@Manvi07 so activator.yaml does not have the istio injector annotation now? you might want to check the deployment status for activator and autoscaler to see if still the same error as before.

Yes @yubozhao, it does not.

Manvi07 picture Manvi07 on 1 Jul 2020

@Manvi07 so activator.yaml does not have the istio injector annotation now? you might want to check the deployment status for activator and autoscaler to see if still the same error as before.

Yes @yubozhao, it does not.

@Manvi07 Try the above to set the annotation to false explicitly instead of removing it.

yuzisun picture yuzisun on 1 Jul 2020

@yuzisun I turned off the sidecar explicitly as mentioned in your PR, and that resulted in successfully starting the autoscaler however, activator has some issues:

kubectl get pods -n knative-serving
NAME                                READY   STATUS             RESTARTS   AGE
activator-596cd49775-pf5wp          0/1     CrashLoopBackOff   9          21m
autoscaler-7b58b49794-79vzl         1/1     Running            0          7s
autoscaler-hpa-68cc87bfb9-rndz7     1/1     Running            0          22d
controller-95dc7f8bd-6xbqr          1/1     Running            0          22d
networking-istio-5b8c5c6cff-x64ch   1/1     Running            0          22d
webhook-67847fb4b5-z6rf2            1/1     Running            0          22d
Manvi07 picture Manvi07 on 1 Jul 2020

@yuzisun I turned off the sidecar explicitly as mentioned in your PR, and that resulted in successfully starting the autoscaler however, activator has some issues:

kubectl get pods -n knative-serving
NAME                                READY   STATUS             RESTARTS   AGE
activator-596cd49775-pf5wp          0/1     CrashLoopBackOff   9          21m
autoscaler-7b58b49794-79vzl         1/1     Running            0          7s
autoscaler-hpa-68cc87bfb9-rndz7     1/1     Running            0          22d
controller-95dc7f8bd-6xbqr          1/1     Running            0          22d
networking-istio-5b8c5c6cff-x64ch   1/1     Running            0          22d
webhook-67847fb4b5-z6rf2            1/1     Running            0          22d

@Manvi07 still crashing? try delete and redeploy activator? you are close !

yuzisun picture yuzisun on 1 Jul 2020

Thanks a lot @yuzisun. The pods are up & running now. However, on redeploying the inferenceService I get IngressNotConfigured and the output of kubectl logs -l app=networking-istio -n knative-serving is: 

{"level":"info","ts":"2020-07-02T04:48:35.823Z","logger":"istiocontroller.ingress-controller","caller":"ingress/ingress.go:169","msg":"Creating/Updating VirtualServices","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","knative.dev/traceid":"b818c5f1-e7fb-4a05-b8ff-95ab4db2bb48","knative.dev/key":"default/sklearn-iris-predictor-default"}
W0702 04:48:52.675011       1 reflector.go:302] k8s.io/client-go/informers/factory.go:133: watch of *v1.ConfigMap ended with: too old resource version: 34968785 (34973892)
{"level":"info","ts":"2020-07-02T04:49:05.826Z","logger":"istiocontroller.ingress-controller.event-broadcaster","caller":"record/event.go:258","msg":"Event(v1.ObjectReference{Kind:\"Ingress\", Namespace:\"default\", Name:\"sklearn-iris-predictor-default\", UID:\"3d9fd72e-e55a-418c-9d9b-c5ca7e5069a7\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", ResourceVersion:\"34970598\", FieldPath:\"\"}): type: 'Warning' reason: 'CreationFailed' Failed to create VirtualService default/sklearn-iris-predictor-default-mesh: Timeout: request did not complete within requested timeout 30s","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-07-02T04:49:05.826Z","logger":"istiocontroller.ingress-controller.event-broadcaster","caller":"record/event.go:258","msg":"Event(v1.ObjectReference{Kind:\"Ingress\", Namespace:\"default\", Name:\"sklearn-iris-predictor-default\", UID:\"3d9fd72e-e55a-418c-9d9b-c5ca7e5069a7\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", ResourceVersion:\"34970598\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' failed to create VirtualService: Timeout: request did not complete within requested timeout 30s","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"warn","ts":"2020-07-02T04:49:35.828Z","logger":"istiocontroller.ingress-controller","caller":"ingress/ingress.go:134","msg":"Failed to update Ingress status","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","knative.dev/traceid":"b818c5f1-e7fb-4a05-b8ff-95ab4db2bb48","knative.dev/key":"default/sklearn-iris-predictor-default","error":"Timeout: request did not complete within requested timeout 30s"}
{"level":"error","ts":"2020-07-02T04:49:35.829Z","logger":"istiocontroller.ingress-controller","caller":"controller/controller.go:376","msg":"Reconcile error","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","error":"Timeout: request did not complete within requested timeout 30s","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:376\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:362\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func2\n\t/home/prow/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:310"}
{"level":"info","ts":"2020-07-02T04:49:35.829Z","logger":"istiocontroller.ingress-controller","caller":"controller/controller.go:363","msg":"Reconcile failed. Time taken: 1m0.0056637s.","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","knative.dev/traceid":"b818c5f1-e7fb-4a05-b8ff-95ab4db2bb48","knative.dev/key":"default/sklearn-iris-predictor-default"}
{"level":"info","ts":"2020-07-02T04:49:35.829Z","logger":"istiocontroller.ingress-controller.event-broadcaster","caller":"record/event.go:258","msg":"Event(v1.ObjectReference{Kind:\"Ingress\", Namespace:\"default\", Name:\"sklearn-iris-predictor-default\", UID:\"3d9fd72e-e55a-418c-9d9b-c5ca7e5069a7\", APIVersion:\"networking.internal.knative.dev/v1alpha1\", ResourceVersion:\"34970598\", FieldPath:\"\"}): type: 'Warning' reason: 'UpdateFailed' Failed to update status for Ingress \"sklearn-iris-predictor-default\": Timeout: request did not complete within requested timeout 30s","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-07-02T04:49:35.989Z","logger":"istiocontroller.ingress-controller","caller":"ingress/ingress.go:160","msg":"Reconciling ingress: &v1alpha1.Ingress{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"sklearn-iris-predictor-default\", GenerateName:\"\", Namespace:\"default\", SelfLink:\"/apis/networking.internal.knative.dev/v1alpha1/namespaces/default/ingresses/sklearn-iris-predictor-default\", UID:\"3d9fd72e-e55a-418c-9d9b-c5ca7e5069a7\", ResourceVersion:\"34970598\", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63729261809, loc:(*time.Location)(0x29d6020)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"serving.knative.dev/route\":\"sklearn-iris-predictor-default\", \"serving.knative.dev/routeNamespace\":\"default\", \"serving.knative.dev/service\":\"sklearn-iris-predictor-default\"}, Annotations:map[string]string{\"kubernetes.io/ingress.class\":\"contour-corp\", \"networking.knative.dev/ingress.class\":\"istio.ingress.networking.knative.dev\", \"serving.knative.dev/creator\":\"system:serviceaccount:kfserving-system:default\", \"serving.knative.dev/lastModifier\":\"system:serviceaccount:kfserving-system:default\"}, OwnerReferences:[]v1.OwnerReference{v1.OwnerReference{APIVersion:\"serving.knative.dev/v1alpha1\", Kind:\"Route\", Name:\"sklearn-iris-predictor-default\", UID:\"13972fd8-00dc-4d37-ab04-cde6a026681d\", Controller:(*bool)(0xc0017c6150), BlockOwnerDeletion:(*bool)(0xc0017c6151)}}, Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1alpha1.IngressSpec{DeprecatedGeneration:0, TLS:[]v1alpha1.IngressTLS(nil), Rules:[]v1alpha1.IngressRule{v1alpha1.IngressRule{Hosts:[]string{\"sklearn-iris-predictor-default.default.svc.cluster.local\", \"sklearn-iris-predictor-default.default.example.com\"}, Visibility:\"ExternalIP\", HTTP:(*v1alpha1.HTTPIngressRuleValue)(0xc001257be0)}}, Visibility:\"ExternalIP\"}, Status:v1alpha1.IngressStatus{Status:v1.Status{ObservedGeneration:0, Conditions:v1.Conditions{apis.Condition{Type:\"LoadBalancerReady\", Status:\"Unknown\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0xbfb777b7faf6ed52, ext:1953176998646523, loc:(*time.Location)(0x29d6020)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"NetworkConfigured\", Status:\"Unknown\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0xbfb777b7faf6e196, ext:1953176998643518, loc:(*time.Location)(0x29d6020)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"Ready\", Status:\"Unknown\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0xbfb777b7faf6d8bc, ext:1953176998641268, loc:(*time.Location)(0x29d6020)}}}, Reason:\"\", Message:\"\"}}}, LoadBalancer:(*v1alpha1.LoadBalancerStatus)(nil), PublicLoadBalancer:(*v1alpha1.LoadBalancerStatus)(nil), PrivateLoadBalancer:(*v1alpha1.LoadBalancerStatus)(nil)}}","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","knative.dev/traceid":"9483913d-2b54-41f6-9e1a-62943419495d","knative.dev/key":"default/sklearn-iris-predictor-default"}
{"level":"info","ts":"2020-07-02T04:49:35.989Z","logger":"istiocontroller.ingress-controller","caller":"ingress/ingress.go:169","msg":"Creating/Updating VirtualServices","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","knative.dev/traceid":"9483913d-2b54-41f6-9e1a-62943419495d","knative.dev/key":"default/sklearn-iris-predictor-default"}
Manvi07 picture Manvi07 on 2 Jul 2020

@Manvi07 Looks like it is not able to create the istio virtual service due to some networking issue on your cluster.

yuzisun picture yuzisun on 2 Jul 2020

@yuzisun I am also having similar issue,

I have installed kubeflow with minikube and then installed kfserving from repo, run same example 

ubuntu@td-kubeflow:~/kfserving/docs/samples/tensorflow$ kubectl apply -f tensorflow.yaml 
error: Error loading config file "/home/ubuntu/.kube/config": open /home/ubuntu/.kube/config: permission denied
ubuntu@td-kubeflow:~/kfserving/docs/samples/tensorflow$ sudo kubectl apply -f tensorflow.yaml 
inferenceservice.serving.kubeflow.org/flowers-sample unchanged
ubuntu@td-kubeflow:~/kfserving/docs/samples/tensorflow$ kubectl get inferenceservices
error: Error loading config file "/home/ubuntu/.kube/config": open /home/ubuntu/.kube/config: permission denied
ubuntu@td-kubeflow:~/kfserving/docs/samples/tensorflow$ sudo kubectl get inferenceservices
NAME             URL       READY     DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
flowers-sample                                                          28h
sklearn-iris                                                            71m
ubuntu@td-kubeflow:~/kfserving/docs/samples/tensorflow$

not seeing any URL or Ready Statues 

ubuntu@td-kubeflow:~/kfserving/docs/samples/tensorflow$ sudo kubectl get pods -n knative-serving
NAME                                READY     STATUS    RESTARTS   AGE
activator-78c895d774-tdpdj          2/2       Running   5          6d18h
autoscaler-7cfc48cf7-qvhmr          2/2       Running   7          6d18h
autoscaler-hpa-867dddf-w4vnf        1/1       Running   1          6d18h
controller-986646d76-bg4h2          0/1       Pending   0          6d18h
networking-istio-79b5d9cf79-qnqtz   0/1       Pending   0          6d18h
webhook-5dbc56cccf-csqw4            1/1       Running   1          6d18h
tiru1930 picture tiru1930 on 2 Jul 2020

@tiru1930 looks like your minkube does not have enough resources as you can see knative controller and networking-istio pods are in pending state

yuzisun picture yuzisun on 3 Jul 2020

@Manvi07 Looks like it is not able to create the istio virtual service due to some networking issue on your cluster.

@yuzisun any suggestions on how should I go about the problem?

Manvi07 picture Manvi07 on 3 Jul 2020

@yuzisun now i have increased cpu count

root@td-kubeflow:/home/ubuntu/kfserving/docs/samples/sklearn# kubectl get pods -n knative-serving
NAME                                READY     STATUS    RESTARTS   AGE
activator-78c895d774-tdpdj          2/2       Running   10         7d11h
autoscaler-7cfc48cf7-qvhmr          2/2       Running   12         7d11h
autoscaler-hpa-867dddf-w4vnf        1/1       Running   3          7d11h
controller-986646d76-bg4h2          1/1       Running   0          7d11h
networking-istio-79b5d9cf79-qnqtz   1/1       Running   0          7d11h
webhook-5dbc56cccf-csqw4            1/1       Running   3          7d11h
root@td-kubeflow:/home/ubuntu/kfserving/docs/samples/sklearn# ls -lrt
total 16
-rw-rw-r-- 1 ubuntu ubuntu  214 Jul  2 11:20 sklearn.yaml
-rw-rw-r-- 1 ubuntu ubuntu 1050 Jul  2 11:20 perf.yaml
-rw-rw-r-- 1 ubuntu ubuntu   82 Jul  2 11:20 iris-input.json
-rw-rw-r-- 1 ubuntu ubuntu 2568 Jul  2 11:20 README.md
root@td-kubeflow:/home/ubuntu/kfserving/docs/samples/sklearn# kubectl apply -f sklearn.yaml 
inferenceservice.serving.kubeflow.org/sklearn-iris unchanged
root@td-kubeflow:/home/ubuntu/kfserving/docs/samples/sklearn# kubectl get inferenceservices
NAME             URL       READY     DEFAULT TRAFFIC   CANARY TRAFFIC   AGE
flowers-sample                                                          44h
sklearn-iris                                                            10h
root@td-kubeflow:/home/ubuntu/kfserving/docs/samples/sklearn#

Where does these pods created ?

when i do kubectl get pods
No resources found.

tiru1930 picture tiru1930 on 3 Jul 2020

@Manvi07 Looks like it is not able to create the istio virtual service due to some networking issue on your cluster.

@yuzisun any suggestions on how should I go about the problem?

@Manvi07 this smells like firewall issue on your cluster, in this case knative is calling istio webhook istio-galley in istio-system namespace to create the virtual service but timed out. It is best to check your firewall setup.

yuzisun picture yuzisun on 3 Jul 2020
👍1

@Manvi07 let me know if I can close this?

yuzisun picture yuzisun on 7 Jul 2020

Yeah @yuzisun. Doesn't look like a Kubeflow issue now. Thanks for your help. :)

Manvi07 picture Manvi07 on 8 Jul 2020

/close

yuzisun picture yuzisun on 8 Jul 2020

@yuzisun: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot picture k8s-ci-robot on 8 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jeffgriffith picture jeffgriffith  路  9Comments
ontheway16 picture ontheway16  路  4Comments
kmr0877 picture kmr0877  路  5Comments
aka-toxa picture aka-toxa  路  4Comments
valentin-stefan-popa picture valentin-stefan-popa  路  3Comments