Serving: gRPC: Received message larger than max (32243844 vs. 4194304)"

[benjamintanweihao]

System information

• OS Platform and Distribution: Linux Ubuntu 16.04
• TensorFlow Serving installed from (source or binary): Binary
• TensorFlow Serving version: TensorFlow ModelServer: 1.13.0-rc1+dev.sha.f16e777
  TensorFlow Library: 1.13.1

Describe the problem

I get a Received messager larger than max (32243844 vs. 4194304) error when creating a predict_pb2.PredictReqest().

Source code / logs

host = "0.0.0.0"
port = 8500
model_name = "default"
signature_name = "serving_default"
timeout = 600

image_data = imageio.imread('test.png')

channel = grpc.insecure_channel("localhost:"+str(port))
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

request = predict_pb2.PredictRequest()
request.model_spec.name = model_name
request.model_spec.signature_name = signature_name

image_meta = ImageMeta.build(0, image_data.shape, image_data.shape, (0, 0, 1, 1), 1.0)
image_meta_shape = [1, 12]
image_meta = np.expand_dims(image_meta, axis=0)

image_shape = [1] + list(image_data.shape)
image_data = np.expand_dims(image_data, axis=0)
image_data = image_data.astype(np.uint8)

image_tensor = tf.make_tensor_proto(image_data,
                                    dtype=types_pb2.DT_UINT8,
                                    shape=image_shape,
                                    verify_shape=True)

image_meta_tensor = tf.make_tensor_proto(image_meta,
                                    dtype=types_pb2.DT_FLOAT,
                                    shape=image_meta_shape,
                                    verify_shape=True)

request.inputs['image'].CopyFrom(image_tensor)
request.inputs['image_meta'].CopyFrom(image_meta_tensor)

result = stub.Predict(request, timeout)

Results in:

Traceback (most recent call last):
  File "/home/benjamintan/workspace/darkrai/pipeline/client.py", line 70, in <module>
    result = stub.Predict(request, timeout)
  File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/grpc/_channel.py", line 565, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
    status = StatusCode.RESOURCE_EXHAUSTED
    details = "Received message larger than max (32243844 vs. 4194304)"
    debug_error_string = "{"created":"@1560406481.884288286","description":"Received message larger than max (32243844 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":190,"grpc_status":8}"

I've seen a similar issue (#288) but that seems to affect an older version of the model server.

When I perform the same request in JSON, it works fine.

[netfs]

our receive limit is set to 2GB here:

https://github.com/tensorflow/serving/blob/91bf584c452494dd1e62eed0f6d1da8a47df030b/tensorflow_serving/model_servers/server.cc#L306

its odd that you are getting error, with limit showing 4MB (4194304).

can you try if https://github.com/tensorflow/serving/pull/1350 fixes your issue (i doubt it will). if it does not, i am wondering if this has something to do with gRPC/Python. one way to test that will be to write a C++ gRPC client and see if that shows the same behavior.

[HFrost0]

I got the same question.

[EsmeYi]

I solved this problem by assigning the grpc.max_message_length when creating the gRPC stub
~
# create the gRPC stub
options = [('grpc.max_message_length', 100 * 1024 * 1024)]
channel = grpc.insecure_channel(server_url, options = options)
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
~

[benjamintanweihao]

Thanks for the pointer @Eloring ! I needed this option to get it to work:

options = [('grpc.max_receive_message_length', max_message_length)]

[gowthamkpr]

Closing this issue as it has been answered. Please add additional comments and we can open this issue again. Thanks!

[oscarriddle]

for grpc 1.25.0:
channel_opt = [('grpc.max_send_message_length', 512 * 1024 * 1024), ('grpc.max_receive_message_length', 512 * 1024 * 1024)]

[CoinCheung]

How to set this in c++ ?

[ubaidillahSriyudi]

does anyone know how to do it (increase the message size) on Dart Flutter ?
i got gRPC: Received message larger than max, fyi i use go for the server side

[PistonY]

Don't forget add options to server.

server = grpc.server(futures.ThreadPoolExecutor(max_workers=max_workers), options=channel_opt)

[DachuanZhao]

[('grpc.max_send_message_length', 512 * 1024 * 1024), ('grpc.max_receive_message_length', 512 * 1024 * 1024)]

1.33.2 can work , too .

[mys007]

How to set this in c++ ?

It's an argument of a channel (grpc/1.27.3)

auto cargs = grpc::ChannelArguments();
cargs.SetMaxReceiveMessageSize(1024 * 1024 * 1024); // 1 GB
cargs.SetMaxSendMessageSize(1024 * 1024 * 1024);
std::shared_ptr<grpc::Channel> channel = grpc::CreateCustomChannel("localhost:50051", grpc::InsecureChannelCredentials(), cargs);