Serving: inception-client error with tensorflow-serving-apis, but works well with bazel built server

[TonyChouZJU]

Build label: 0.9.0
TF: 1.5.0 CPU
Python 2.7.14 :: Anaconda custom (64-bit)
Ubuntu 16.04 TLS

The following error arise when I try to run inception-client with python and inception-sever with tensorflow-serving-api.

tensorflow-serving-apis

tensorflow_model_server --port=9000 --model_name=inception --model_base_path=inception-export

python tensorflow_serving/example/inception_client.py --server=localhost:9000 --image=/path-to-image

The error is

Traceback (most recent call last):
File "tensorflow_serving/example/inception_client.py", line 60, in
tf.app.run()
File "/home/zyb/anaconda2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 124, in run
_sys.exit(main(argv))
File "tensorflow_serving/example/inception_client.py", line 55, in main
result = stub.Predict(request, 10.0) # 10 secs timeout
File "/home/zyb/anaconda2/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 310, in __call__
self._request_serializer, self._response_deserializer)
File "/home/zyb/anaconda2/lib/python2.7/site-packages/grpc/beta/_client_adaptations.py", line 196, in _blocking_unary_unary
raise _abortion_error(rpc_error_call)
grpc.framework.interfaces.face.face.AbortionError: AbortionError(code=StatusCode.INVALID_ARGUMENT, details="NodeDef mentions attr 'identical_element_shapes' not in Op handle:resource, flow:float; attr=dtype:type; attr=element_shape:shape,default=; attr=dynamic_size:bool,default=false; attr=clear_after_read:bool,default=true; attr=tensor_array_name:string,default=""; is_stateful=true>; NodeDef: map/TensorArray = TensorArrayV3_output_shapes=[[2], []], clear_after_read=true, dtype=DT_STRING, dynamic_size=false, element_shape=, identical_element_shapes=true, tensor_array_name="", _device="/job:localhost/replica:0/task:0/device:CPU:0". (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
[[Node: map/TensorArray = TensorArrayV3_output_shapes=[[2], []], clear_after_read=true, dtype=DT_STRING, dynamic_size=false, element_shape=, identical_element_shapes=true, tensor_array_name="", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]")

---

But it workes well when inception-client in python and inception-sever in bazel

bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=9000 --model_name=inception --model_base_path=inception-export

python tensorflow_serving/example/inception_client.py --server=localhost:9000 --image=/path-to-image

Any solutions? Thanks in advance.

[TonyChouZJU]

Especially, what the following details means?

StatusCode.INVALID_ARGUMENT, details="NodeDef mentions attr 'identical_element_shapes' not in Op handle:resource, flow:float; attr=dtype:type; attr=element_shape:shape,default=; attr=dynamic_size:bool,default=false; attr=clear_after_read:bool,default=true; attr=tensor_array_name:string,default=""; is_stateful=true>; NodeDef: map/TensorArray = TensorArrayV3_output_shapes=[[2], []], clear_after_read=true, dtype=DT_STRING, dynamic_size=false, element_shape=, identical_element_shapes=true, tensor_array_name="", _device="/job:localhost/replica:0/task:0/device:CPU:0".

[kirilg]

This is a forward compatibility error that typically happens when you train a model using code that's newer than what was used to compile the server (and there was a forward incompatible change in between). The reason it works when you build with Bazel is because then you pull the latest code which includes the commit that introduced this new attr.

If you were using TF 1.5.0 to train, is it possible you're using an older ModelServer? You can install the 1.5.0 TensorFlow Serving ModelServer (released yesterday) using apt-get install. Can you please give that a try?

[TonyChouZJU]

@kirilg I upgrade the tensorflow-model-server to 1.5.0. It works.
Now, the inception-client in python can work both with inception-sever in bazel and inception-server in tensorflow_model_server.
Thanks a lot.

[kirilg]

Glad that worked :)