Kfserving: PVC model not found

[jeffgriffith]

/kind bug

KFserving question (issue?) as it pertains to PVC storage of TF saved models. i've created an azure PV and copied data to it, standalone ubuntu pod had no trouble mounting it and that's how i copied my model into it (I mounted the PVC into /data on that pod and then deleted the pod then verified the data is still really there with a new instance):

./dpe2e
./dpe2e/saved_model.pbtxt
./dpe2e/variables
./dpe2e/variables/variables.data-00000-of-00001
./dpe2e/variables/variables.index

Now when the predictor pod is initializing it prints this:

INFO:root:Initializing, args: src_uri [/mnt/pvc/dpe2e] dest_path[ [/mnt/models]
INFO:root:Copying contents of /mnt/pvc/dpe2e to local
INFO:root:Linking: /mnt/pvc/dpe2e/saved_model.pbtxt to /mnt/models/saved_model.pbtxt
INFO:root:Linking: /mnt/pvc/dpe2e/variables to /mnt/models/variables

But ultimately it fails because it cannot find any servable models under /mnt/models.

2019-11-12 20:01:47.406630: I tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config:  model_name: dpe2e model_base_path: /mnt/models
2019-11-12 20:01:47.406793: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2019-11-12 20:01:47.406812: I tensorflow_serving/model_servers/server_core.cc:561]  (Re-)adding model: dpe2e
2019-11-12 20:01:47.407275: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable dpe2e found under base path /mnt/models
2019-11-12 20:01:48.407418: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable dpe2e found under base path /mnt/models
2019-11-12 20:01:49.407532: W tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:267] No versions of servable dpe2e found under base path /mnt/models

Indeed that seems to be true because /mnt/models/saved_model.pbtxt is symbolically linked to /mnt/pvc which does not exist at the point where i bash into it. At SOME point however it seems to have found my model because it sees the .pbtxt extension vs .pb (which i proved to myself by fiddling with the names).

My YAML looks like this:

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
  name: "dpe2e"
spec:
  default:
    predictor:
      tensorflow:
        storageUri: "pvc://azurefile/dpe2e"
      minReplicas: 1
      maxReplicas: 1

Ideas? I feel like i have a naming/path issue somewhere or for some reason the PVC is released. The 'Copying contents of /mnt/pvc/... to local' seems odd considering it's actually linking to /mnt/pvc.

Environment:

Basic install on Azure AKS cluster using v0.7 kfctl.

[ellistarn]

Thanks for the bug report! This is one of our lesser tested codepaths. @rakelkar can you help?

[jeffgriffith]

@ellis-bigelow @rakelkar i managed to successfully replace 0.2.0 with 0.2.1 in my general KF install and get this to work from an azure files PVC. Gonna close it.