Tf-operator: Servable not found for request: Latest(mnist)

[zuowang]

I am trying to run model.py with tfjob. Below is my yaml file to create a tfjob.

apiVersion: "kubeflow.org/v1alpha1"
kind: "TFJob"
metadata:
  name: "example-job3"
  namespace: tfworkflow
spec:
  replicaSpecs:
    - replicas: 1
      tfReplicaType: MASTER
      template:
        spec:
          serviceAccountName: tf-job-operator
          containers:
            - image: tensorflow/tensorflow:1.5.1
              name: tensorflow
              imagePullPolicy: Always
              command:
               - /usr/bin/python
              args:
               - /data/pijie/mnist/model.py
              env:
               - name: TF_DATA_DIR
                 value: /data/pijie/mnist/data-12345
               - name: TF_MODEL_DIR
                 value: /data/pijie/mnist/model-12345
               - name: TF_EXPORT_DIR
                 value: modelf-12345
               - name: TF_TRAIN_STEPS
                 value: "200"
               - name: TF_TF_BATCH_SIZE
                 value: "100"
               - name: TF_LEARNING_RATE
                 value: "0.01"
              volumeMounts:
               - name: data-persistent-storage
                 mountPath: /data
          volumes:
          - name: data-persistent-storage
            persistentVolumeClaim:
              claimName: tfworkflow-pvc
          restartPolicy: OnFailure
    - replicas: 2
      tfReplicaType: WORKER
      template:
        spec:
          serviceAccountName: tf-job-operator
          containers:
            - image: tensorflow/tensorflow:1.5.1
              name: tensorflow
              imagePullPolicy: Always
              command:
               - /usr/bin/python
              args:
               - /data/pijie/mnist/model.py
              env:
               - name: TF_DATA_DIR
                 value: /data/pijie/mnist/data-12345
               - name: TF_MODEL_DIR
                 value: /data/pijie/mnist/model-12345
               - name: TF_EXPORT_DIR
                 value: modelf-12345
               - name: TF_TRAIN_STEPS
                 value: "200"
               - name: TF_TF_BATCH_SIZE
                 value: "100"
               - name: TF_LEARNING_RATE
                 value: "0.01"
              volumeMounts:
               - name: data-persistent-storage
                 mountPath: /data
          volumes:
          - name: data-persistent-storage
            persistentVolumeClaim:
              claimName: tfworkflow-pvc
          restartPolicy: OnFailure
    - replicas: 2
      tfReplicaType: PS
      template:
        spec:
          serviceAccountName: tf-job-operator
          containers:
            - image: tensorflow/tensorflow:1.5.1
              name: tensorflow
              imagePullPolicy: Always
              command:
               - /usr/bin/python
              args:
               - /data/pijie/mnist/model.py
              env:
               - name: TF_DATA_DIR
                 value: /data/pijie/mnist/data-12345
               - name: TF_MODEL_DIR
                 value: /data/pijie/mnist/model-12345
               - name: TF_EXPORT_DIR
                 value: modelf-12345
               - name: TF_TRAIN_STEPS
                 value: "200"
               - name: TF_TF_BATCH_SIZE
                 value: "100"
               - name: TF_LEARNING_RATE
                 value: "0.01"
              volumeMounts:
               - name: data-persistent-storage
                 mountPath: /data
          volumes:
          - name: data-persistent-storage
            persistentVolumeClaim:
              claimName: tfworkflow-pvc
          restartPolicy: OnFailure

But when I used tf-serving to serve the trained model, and predicate a number.

/usr/bin/tensorflow_model_server --port=9000 --model_name=modelf-12345 --model_base_path=/model-12345/export/modelf-12345

python mnist_client.py



md5-d26567af22ff05e72662829edbf4aa4b



root@77f321b127c5:/notebooks# python mnist_client.py
/usr/local/lib/python2.7/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Traceback (most recent call last):
  File "mnist_client.py", line 45, in <module>
    result = stub.Predict(request, 10.0)  # 10 secs timeout
  File "/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py", line 309, in __call__
    self._request_serializer, self._response_deserializer)
  File "/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py", line 195, in _blocking_unary_unary
    raise _abortion_error(rpc_error_call)
grpc.framework.interfaces.face.face.AbortionError: AbortionError(code=StatusCode.NOT_FOUND, details="Servable not found for request: Latest(mnist)")

However, it works when I follow the steps in here using argo.

[gaocegege]

/api v1alpha1
/kind question

[gaocegege]

I am not sure if it is the right place for the question since it is related to tf-serving.

/cc @jlewi

[gaocegege]

Hi @zuowang

I am not sure if the dir here is right:

/usr/bin/tensorflow_model_server --port=9000 --model_name=modelf-12345 --model_base_path=/model-12345/export/modelf-12345

Maybe it should be/data/pijie/mnist/model-12345. :thinking:

BTW, I think the error is not caused by tf-operator.

[zuowang]

@gaocegege The dir is right. /model-12345/export/modelf-12345 is the exported model.

[jlewi]

The error "Servable not found for request: Latest(mnist) indicates a problem with TFServing and the client. I believe this error basically means that the request specified a model of Latest(mnist) and this was not found.

What do the logs for TFServing show?

/cc @elsonrodriguez @lluunn

[elsonrodriguez]

Can you try changing --model_name=modelf-12345 to --model_name=mnist for the tf-seving command?

[zuowang]

@elsonrodriguez Thank you so much! You save me a lot of time!