Serving: Occasionally "Servable not found for request"

[quanhua92]

Hi,
I export a model with tf.contrib.session_bundle and serve it with tensorflow_model_server. It works but not always. I use the same client code without any changes and sometimes it gives me the error. For example, i sent 5 times and only 1 time makes the error.
Please tell me if anyone knows what is the problem.
Thanks in advance.

File "scripts/python_client.py", line 24, in <module>
    result = stub.Predict(request, 20.)
  File "/home/ubuntu/miniconda3/envs/work2/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 324, in __call__
    self._request_serializer, self._response_deserializer)
  File "/home/ubuntu/miniconda3/envs/work2/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 210, in _blocking_unary_unary
    raise _abortion_error(rpc_error_call)
grpc.framework.interfaces.face.face.AbortionError: AbortionError(code=StatusCode.NOT_FOUND, details="Servable not found for request: Latest(model)")

Here is my client python code.

host, port = "0.0.0.0:9000".split(":")
channel = implementations.insecure_channel(host, int(port))
stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)

img = imread("samples_data/dog02.jpg")

request = predict_pb2.PredictRequest()
request.model_spec.name = "model"

request.inputs["image"].CopyFrom(
    tf.contrib.util.make_tensor_proto(
        img.astype(dtype=float),
        shape=img.shape, dtype=tf.float32
    )
)

result = stub.Predict(request, 10.)
print(result)

[ashavish]

Hi,
You can check if you give the same model name ie. 'model' everytime when you launch the tensorflow server. If the model name doesn't match the one the client is looking for, then I have encountered this error.

[quanhua92]

Hi @ashavish,
I run tensorflow serving first with named "model". Then I run the client code in a python file 5 times without changing anything. This error just happened 1 or 2 times in total 5 times.
However, if I run the client code in a for loop without stopping the process, everything is fine. For example, I use a Flask Server to run this code in a API request and the error doesn't happen.

[kirilg]

Is it the first few requests that fail or do you get successful requests followed by NOT_FOUND errors? It's possible that the server did not finish loading the model when you sent your first request which failed with NOT_FOUND, but subsequent requests succeeded because loading finished.

[quanhua92]

Hi @kirilg,
The first request success but some later requests failed. Does tensorflow serving release the model after each client process finished? I think that because I run the second process as soon as the first process finished and the second process can't find the model.
However, if I run the client code in a Flask process, it always success.

[kirilg]

The server will not unload the model when a client stops querying it. It should retain it in memory.

You said the second process can't find the model. Are both clients identical? Do requests from the second client eventually start to succeed or does the second client repeatedly get NOT_FOUND errors? Assuming the requests they send to the model server are identical and the model server is still running and healthy, I can't see a reason why this would happen.

[quanhua92]

HI @kirilg,
I just ran the python file several times so the clients are completely identical. However, I don't face this issue in the latest build from master. So i suggest that we can close this issue.

[shahinkl]

I am also facing the same issue.

[kirilg]

@shahinkl could you please provide more details with instructions to reproduce the error locally?

[utgarda]

I just had it and found this issue. Turned out I had a second tensorflow_model_server running on the same port, but without the required model, in a tmux session I forgot about. Stopped the wrong one and the issue's gone.

[salman-ghauri]

for me, this worked when I change request.model_spec.name = model to the model_name that I specified in tensorflow serving docker container.

[cishimingyuan]

@salman-ghauri Hi, could you please explain how to change request.model_spec.name = model to the model_name, I got this same question, Thanks!
AbortionError: AbortionError(code=StatusCode.NOT_FOUND, details="Servable not found for request: Latest(deeplab)

[salman-ghauri]

You have to check the name of the folder which you keep under /models/{your-model}/{versions} in docker container for tf-serving. This {your-model} is the name you have to set to request.model_spec.name. In your case, it seems like you have set this property to deeplab, if that's so, then, check the contents under this folder in your container.