Serving: A protocol message was rejected because it was too big (more than 1073741824 bytes)

[DachuanZhao]

A same issue maybe https://github.com/tensorflow/serving/issues/1331

Bug Report

If this is a bug report, please fill out the following form in full:

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS7
• TensorFlow Serving installed from (source or binary): pip install tensorflow
• TensorFlow Serving version: latest from docker

Describe the problem

When I serve a model whose size is 1.5gb ， tf-serving raises an error :

2020-07-10 10:21:20.941918: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-07-10 10:21:20.941938: I tensorflow_serving/model_servers/server_core.cc:575]  (Re-)adding model: model_output
2020-07-10 10:21:21.042806: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: model_output version: 0}
2020-07-10 10:21:21.042855: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: model_output version: 0}
2020-07-10 10:21:21.042883: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: model_output version: 0}
2020-07-10 10:21:21.042993: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/model_output/0
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/io/coded_stream.cc:192] A protocol message was rejected because it was too big (more than 1073741824 bytes).  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in net/proto2/io/public/coded_stream.h.
2020-07-10 10:21:21.480829: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:364] SavedModel load for tags { serve }; Status: fail: Data loss: Can't parse /models/model_output/0/saved_model.pb as binary proto. Took 437835 microseconds.
2020-07-10 10:21:21.480865: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name: model_output version: 0} failed: Data loss: Can't parse /models/model_output/0/saved_model.pb as binary proto

What should I do ?
Why not change the default size 1GB to 2GB or more ?

[gowthamkpr]

@DachuanZhao As mentioned in this comment, try hardcoding INT_MAX at L399 in coded_stream.h (SetTotalBytesLimit(INT_MAX);). Thanks!

[DachuanZhao]

@gowthamkpr
Where is coded_stream.h while I install tf-serving by docker ?

[shadowdragon89]

This is caused by the default value specify in Tensorflow for the size limitation. One way you could walk around is to increase the default value and recompile from source. We could see if it is possible to make it easy to config from Tensorflow.

[DachuanZhao]

This is caused by the default value specify in Tensorflow for the size limitation. One way you could walk around is to increase the default value and recompile from source. We could see if it is possible to make it easy to config from Tensorflow.

ok, waiting for your good news ~~~

[shadowdragon89]

This should be fixed by the change: https://github.com/tensorflow/tensorflow/commit/dc3099c444d294b39cd79fe1d1a4bff59a0c6180