Bert: Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary

[iiapache]

I am using bert-serving to deploy tf-serving: https://github.com/bigboNed3/bert_serving，eporting model work fine，however inference doesn't work，anybody knows how to fix this problem：

Traceback (most recent call last):
File "request_queryclassifier_client.py", line 183, in
tf.app.run()
File "/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "request_queryclassifier_client.py", line 167, in main
result = stub.Predict(request, 1000.0) # 10 secs timeout
File "/data/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 533, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/data/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "NodeDef mentions attr 'Truncate' not in Op y:DstT; attr=SrcT:type; attr=DstT:type>; NodeDef: bert/encoder/Cast = CastDstT=DT_FLOAT, SrcT=DT_INT32, Truncate=false, _output_shapes=[[?,1,16]], _device="/job:localhost/replica:0/task:0/device:CPU:0". (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).
[[Node: bert/encoder/Cast = CastDstT=DT_FLOAT, SrcT=DT_INT32, Truncate=false, _output_shapes=[[?,1,16]], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]"
debug_error_string = "{"created":"@1553569490.204528585","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1017,"grpc_message":"NodeDef mentions attr 'Truncate' not in Op y:DstT; attr=SrcT:type; attr=DstT:type>; NodeDef: bert/encoder/Cast = CastDstT=DT_FLOAT, SrcT=DT_INT32, Truncate=false, _output_shapes=[[?,1,16]], _device="/job:localhost/replica:0/task:0/device:CPU:0". (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.).nt [[Node: bert/encoder/Cast = CastDstT=DT_FLOAT, SrcT=DT_INT32, Truncate=false, _output_shapes=[[?,1,16]], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]","grpc_status":3}"

ps：the versions of tensorflow are both 1.12.0 in exporting model phase and inference phase.

[iiapache]

I have found the reason： the version of tf in exporting phase & the version of tf in inference phase & the version of tensorflow_model_server should be the same，

[devimonica]

@iiapache how do you know those versions ?

[lucky096]

@devimonica If the server is installed, you can check its version as:
tensorflow_model_server --version

[ZumelzuR]

On my docker container (where I running tensorflow serving) I have
TensorFlow ModelServer: 2.1.0-rc1
TensorFlow Library: 2.1.0

The problem is related with your local tensorflow version that you use for export your protobuf model. I know that if you export your h5 model with tf versions 1.14.0, 2.1.0 and 2.2.0 you will have this problem at the time of perform inference. You can try to use tf versions >1.15.0 and less than 1.8.0. I think this happen because some tensorflow version doesn't support a particular layer at the time of export.

For change your local tensorflow version you can do

pip install tensorflow==1.15.0

[zxy2020]

On my docker container (where I running tensorflow serving) I have
TensorFlow ModelServer: 2.1.0-rc1
TensorFlow Library: 2.1.0

The problem is related with your local tensorflow version that you use for export your protobuf model. I know that if you export your h5 model with tf versions 1.14.0, 2.1.0 and 2.2.0 you will have this problem at the time of perform inference. You can try to use tf versions >1.15.0 and less than 1.8.0. I think this happen because some tensorflow version doesn't support a particular layer at the time of export.

For change your local tensorflow version you can do

pip install tensorflow==1.15.0

I have found the reason： the version of tf in exporting phase & the version of tf in inference phase & the version of tensorflow_model_server should be the same，

Thank you ! I used the tensorflow=1.15.0 to get savedmodel , and ran savedmodel in the docker of tensorflow/serving:1.14.0-gpu, finally , I got the wrong! So, I pulled docker tensorflow/serving:1.15.0-gpu to solve it !