Kfserving: pvc problem with custom inferenceservice

[jeffgriffith]

/kind bug

What steps did you take and what happened:

I'm trying to deploy a 'custom' inferenceservice (just using the base tensorflow/serving image). while i am able to declare the volumeMounts within spec/default/predictor/custom/container, there does not seem to be a place to specify the volumes themselves. looking at the base InferenceService spec, only default and canary seem to be valid at the top level of the spec. Here i'm using a volume i know to work when creating a standalone pod (and it works in general with a tensorflow predictor when referenced with 'storageUri: pvc://...'

apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: nlu-confidence-exam
spec:
  volumes:
    - name: azure-file-volume
      persistentVolumeClaim:
        claimName: azurefile
  default:
    predictor:
      custom:
        container:
          image: tensorflow/serving
          env:
            - name: MODEL_NAME
              value: "nlu-confidence-exam"
            - name: MODEL_BASE_PATH
              value: "/data/nlu-models/confidence/exam"
          ports:
            - containerPort: 8500
          volumeMounts:
          - mountPath: /data
            name: azure-file-volume
            readOnly: true

What did you expect to happen:
Expected the volume to be mounted but instead I get this error:

Error from server: error when creating "nlu-confidence-exam.yaml": admission webhook "inferenceservice.kfserving-webhook-server.validator" denied the request: Custom container validation error: volumeMount has no matching volume: volumeMounts[0].name

Anything else you would like to add:

Environment:
Running with tag 0.2.1 on top of KF 0.7 base install.

[ellistarn]

This has no semantic meaning for KFServing

spec:
  volumes:
    - name: azure-file-volume
      persistentVolumeClaim:
        claimName: azurefile

Can you show the output of a kubectl command showing volumes?

[jeffgriffith]

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
pvc-7d47d436-0a4e-11ea-820d-eaecd6093de4   15Gi       RWX            Delete           Bound    default/azurefile         azurefile               3d21h

[jeffgriffith]

thanks @ellis-bigelow . the volume is above. as i said, it works fine from a tensorflow predictor. i realize from the InferenceService spec that volumes would have no meaning, but i don't see how one would ever reference the pvc from a custom container in that case (maybe i'm missing something obvious in k8s :-) )

[yuzisun]

Can you try this? there is a well-known environment variable STORAGE_URI for custom container which kfserving understands and create the pvc volume.

apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: nlu-confidence-exam
spec:
  default:
    predictor:
      custom:
        container:
          image: tensorflow/serving
          env:
            - name: MODEL_NAME
              value: "nlu-confidence-exam"
            - name: MODEL_BASE_PATH
              value: "/mnt/models"
            - name: STORAGE_URI
              value: pvc://{PVC_NAME}/export
          ports:
            - containerPort: 8500

[jeffgriffith]

perfect. thanks Dan! i'll give that a try.

[jeffgriffith]

Thanks @yuzisun @ellis-bigelow. This worked, however the semantics of the storage uri seems to be a bit different from a typical predictor like tensorflow in how it uses the meta.name to construct the base path as well as link the data from /mnt/pvc. anyway, i'll figure that out and try to document it separately. thanks again! i'll try to make some new samples based on this if you guys are interested.

[yuzisun]

@jeffgriffith feel free to open another issue to discuss how to improve the current behavior, looking forward to your samples !

[decewei]

Can you try this? there is a well-known environment variable STORAGE_URI for custom container which kfserving understands and create the pvc volume.

apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: nlu-confidence-exam
spec:
  default:
    predictor:
      custom:
        container:
          image: tensorflow/serving
          env:
            - name: MODEL_NAME
              value: "nlu-confidence-exam"
            - name: MODEL_BASE_PATH
              value: "/mnt/models"
            - name: STORAGE_URI
              value: pvc://{PVC_NAME}/export
          ports:
            - containerPort: 8500

Is there a way to write to pvc? The PVC mounted here is read only. But I confirmed this method is working for read only purpose pvc.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.