Serving: Protobuf does not allow model size > 1 GB

[estmit]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 10.3.6
• TensorFlow Serving installed from (source or binary): source
• TensorFlow Serving version: master

Describe the problem

Even though total_bytes_limit_ has been set to INT_MAX (2 GB) for a while now, I'm still running into the following error when trying to serve a 1.3GB .pb model:

[libprotobuf ERROR external/protobuf_archive/src/google/protobuf/io/coded_stream.cc:192] A protocol message was rejected because it was too big (more than 1073741824 bytes). To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in net/proto2/io/public/coded_stream.h.

And so I went into serving/.cache/_bazel_username/356b9fce2c05ae2b5521ed5aed432b0e/external/protobuf_archive/src/google/protobuf/io/coded_stream.h and manually set total_bytes_limit_ to 2147483647 and recompiled in bazel (tools/run_in_docker.sh bazel build --jobs 6 tensorflow_serving/...) but I'm still seeing the same error.

Exact Steps to Reproduce

git clone --recurse-submodules https://github.com/tensorflow/serving
cd serving
tools/run_in_docker.sh bazel build --jobs 6 tensorflow_serving/...
# change coded_stream.h in .cache folder
tools/run_in_docker.sh bazel build --jobs 6 tensorflow_serving/...
docker run -it -p 8500:8500 -v '/Users/username/serving:/app/serving' -v '/Users/username/serving/.cache:/app/serving/.cache' -v '/Users/username/my_model:/models/my_model' ubuntu
cd app/serving/.cache/_bazel_username/356b9fce2c05ae2b5521ed5aed432b0e/execroot/tf_serving/bazel-out/k8-opt/bin/tensorflow_serving/model_servers
./tensorflow_model_server --port=8500 --model_name=my_model --model_base_path=/models/my_model

Source code / logs

2019-04-25 18:12:19.724832: I tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config:  model_name: my_model model_base_path: /models/my_model
2019-04-25 18:12:19.727898: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2019-04-25 18:12:19.727956: I tensorflow_serving/model_servers/server_core.cc:561]  (Re-)adding model: my_model
2019-04-25 18:12:19.855120: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: my_model version: 1}
2019-04-25 18:12:19.855187: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: my_model version: 1}
2019-04-25 18:12:19.855213: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: my_model version: 1}
2019-04-25 18:12:19.855824: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:363] Attempting to load native SavedModelBundle in bundle-shim from: /models/my_model/1
2019-04-25 18:12:19.855923: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/my_model/1
[libprotobuf ERROR external/protobuf_archive/src/google/protobuf/io/coded_stream.cc:192] A protocol message was rejected because it was too big (more than 1073741824 bytes).  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in net/proto2/io/public/coded_stream.h.
2019-04-25 18:12:23.108367: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: fail. Took 3252443 microseconds.
2019-04-25 18:12:23.108462: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name: my_model version: 1} failed: Data loss: Can't parse /models/my_model/1/saved_model.pb as binary proto

[gowthamkpr]

@estmit This looks like a duplicate of the issue. Pls go through the issue and let me know if it helps. Thanks!

[estmit]

@gowtham-kp I've already seen issue #24 but I apparently I missed laotao's workaround at the bottom of the thread. I tried to implement workaround by manually changing coded_stream.SetTotalBytesLimit(...) in L518 of env.cc to coded_stream.SetTotalBytesLimit(2147483647) but I'm getting the following compiler/linker errors when running bazel build:

ERROR: /Users/user/serving/.cache/_bazel_user/356b9fce2c05ae2b5521ed5aed432b0e/external/org_tensorflow/tensorflow/core/BUILD:2373:1: Linking of rule '@org_tensorflow//tensorflow/core:lib_internal_impl' failed (Exit 1)
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::GetRegisteredFileSystemSchemes(std::vector<std::string, std::allocator<std::string> >*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::RegisterFileSystem(std::string const&, std::function<tensorflow::FileSystem* ()>)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Thread::~Thread()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Thread::~Thread()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::EnvWrapper::~EnvWrapper()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::EnvWrapper::~EnvWrapper()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Thread::~Thread()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::EnvWrapper::~EnvWrapper()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::GetMatchingPaths(std::string const&, std::vector<std::string, std::allocator<std::string> >*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::Env()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::Env()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::FlushFileSystemCaches()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::NewRandomAccessFile(std::string const&, std::unique_ptr<tensorflow::RandomAccessFile, std::default_delete<tensorflow::RandomAccessFile> >*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::NewReadOnlyMemoryRegionFromFile(std::string const&, std::unique_ptr<tensorflow::ReadOnlyMemoryRegion, std::default_delete<tensorflow::ReadOnlyMemoryRegion> >*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::NewWritableFile(std::string const&, std::unique_ptr<tensorflow::WritableFile, std::default_delete<tensorflow::WritableFile> >*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::NewAppendableFile(std::string const&, std::unique_ptr<tensorflow::WritableFile, std::default_delete<tensorflow::WritableFile> >*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::FileExists(std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::GetChildren(std::string const&, std::vector<std::string, std::allocator<std::string> >*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::DeleteFile(std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::RecursivelyCreateDir(std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::CreateDir(std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::DeleteDir(std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::Stat(std::string const&, tensorflow::FileStatistics*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::IsDirectory(std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::DeleteRecursively(std::string const&, long long*, long long*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::GetFileSize(std::string const&, unsigned long long*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::CreateUniqueFileName(std::string*, std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::LocalTempFilename(std::string*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::ReadFileToString(tensorflow::Env*, std::string const&, std::string*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::WriteStringToFile(tensorflow::Env*, std::string const&, absl::string_view const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::FileSystemCopyFile(tensorflow::FileSystem*, std::string const&, tensorflow::FileSystem*, std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::CopyFile(std::string const&, std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::WriteBinaryProto(tensorflow::Env*, std::string const&, google::protobuf::MessageLite const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::WriteTextProto(tensorflow::Env*, std::string const&, google::protobuf::Message const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::ReadBinaryProto(tensorflow::Env*, std::string const&, google::protobuf::MessageLite*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::ReadTextProto(tensorflow::Env*, std::string const&, google::protobuf::Message*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::FileSystemRegistryImpl::GetRegisteredFileSystemSchemes(std::vector<std::string, std::allocator<std::string> >*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::GetFileSystemForFile(std::string const&, tensorflow::FileSystem**)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::RenameFile(std::string const&, std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::FilesExist(std::vector<std::string, std::allocator<std::string> > const&, std::vector<tensorflow::Status, std::allocator<tensorflow::Status> >*)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::FileSystemRegistryImpl::Lookup(std::string const&)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::FileSystemRegistryImpl::Register(std::string const&, std::function<tensorflow::FileSystem* ()>)'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/0/env.pic.o: multiple definition of 'tensorflow::Env::GetExecutablePath()'
/usr/bin/ld.gold: bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/core/_objs/lib_internal_impl/_env.pic.o: previous definition here
collect2: error: ld returned 1 exit status

[gowthamkpr]

@estmit Can you please try this workaround by changing https://github.com/google/protobuf/blob/master/src/google/protobuf/io/coded_stream.h#L611 to 256 << 20 and then recompile. Let me know if that works . thanks!

[netfs]

upstream protobuf limit is INT_MAX (~2GB):
https://github.com/protocolbuffers/protobuf/blob/859d2da51b3b67b5dc389eb3c0d0d3cc9b9868d7/src/google/protobuf/io/coded_stream.h#L385

setting the first arg in SetTotalByteLimits() to INT_MAX in env.cc here:
https://github.com/tensorflow/tensorflow/blob/1495d2a47cd5de203f5b503f9a878d5c49d585dc/tensorflow/core/platform/env.cc#L518

should fix things for you. and fwiw works for me (read: tools/run_in_docker.sh bazel build tensorflow_serving/... succeeds).

[estmit]

@netfs I did try changing L518 in env.cc but I got a lot of linker errors during recompile (see previous comment https://github.com/tensorflow/serving/issues/1331#issuecomment-487162294). I had to resort to hardcoding INT_MAX at L399 in coded_stream.h (SetTotalBytesLimit(INT_MAX);).. it's not ideal, but it will do for now.

[gowthamkpr]

@estmit Closing this issue as it has been resolved. Please add additional comment and we can reopen the issue again. Thanks!

[DachuanZhao]

@estmit Closing this issue as it has been resolved. Please add additional comment and we can reopen the issue again. Thanks!

Hi , @gowthamkpr
I meet the same error :

2020-07-10 10:21:20.941918: I tensorflow_serving/model_servers/server_core.cc:464] Adding/updating models.
2020-07-10 10:21:20.941938: I tensorflow_serving/model_servers/server_core.cc:575]  (Re-)adding model: model_output
2020-07-10 10:21:21.042806: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: model_output version: 0}
2020-07-10 10:21:21.042855: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: model_output version: 0}
2020-07-10 10:21:21.042883: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: model_output version: 0}
2020-07-10 10:21:21.042993: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/model_output/0
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/io/coded_stream.cc:192] A protocol message was rejected because it was too big (more than 1073741824 bytes).  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in net/proto2/io/public/coded_stream.h.
2020-07-10 10:21:21.480829: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:364] SavedModel load for tags { serve }; Status: fail: Data loss: Can't parse /models/model_output/0/saved_model.pb as binary proto. Took 437835 microseconds.
2020-07-10 10:21:21.480865: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name: model_output version: 0} failed: Data loss: Can't parse /models/model_output/0/saved_model.pb as binary proto

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS7
TensorFlow Serving installed from (source or binary): docker tensorflow/serving:latest
TensorFlow Serving version: docker tensorflow/serving:latest
What should I do ?
Why not change the default size 1GB to 2GB or more ?