Azure-functions-host: [Grpc]Increase MaxMessageLength to int.MaxValue

[F-Schmidt1]

Investigative information

Please provide the following:

• Timestamp: 2020-02-14
• Function App version (1.0 or 2.0): 2.0
• Invocation ID: 482c2b8e-ffbd-48fb-93ee-926bd17f96f8
• Region: West Europe

Repro steps

1. Create a function with a Blobtrigger binding and deploy to Azure (Linux, consumption plan):
2. Upload a file smaller than 120 MB to the Blob Storage Directoy declared in the functions.json
3. Upload a file larger than 120 MB to the Blob Storage Directory declared in the functions.json

Expected behavior

The functions-app should trigger on both files, and be executed twice (with two entries showing up in the monitoring tab)

Actual behavior

The functions-app only triggers on the smaller file, and is only executed once. One successful, and zero non-successful executions show up in the monitoring tab.

In The Application-Insights-Logs the entry

Executing 'Functions.BlobTrigger' (Reason='New blob detected: file.csv', Id=482c2b8e-ffbd-48fb-93ee-926bd17f96f8)

shows up under traces, but not under requests. There are no entries under exceptions.

When running the functions locally in the VS-code debugger, the error

Error Writing message type InvocationRequest to workerId

System.Private.CoreLib: Error sending from server

is thrown.

Related information

Running the equivalent functions-app using c# on windows runs without any issues, so this problem might be specifically related to python and/or linux under consumption plan.

[anthonychu]

There's a 128MB limit on the trigger and binding payload size for gRPC based workers (i.e., non-.NET languages). @anirudhgarg to confirm

The solution to this, currently, is to use an Event Grid trigger to start the function whenever file(s) are added to the container, then use the storage SDK to download the file. This also allows you to stream the file so you don't load it all into memory at once (which can be an issue especially on the consumption plan).

@craigshoemaker Can we add this to the docs? Probably should document that all out of proc languages have a binding payload size of 128MB, and specifically call this out in the blob trigger docs and suggest Event Grid alternative. Thanks!

[F-Schmidt1]

Thank you for your reply.
I just tested the EventGrid trigger, and it seems to be working fine. Is there a reason for why this limit for blobtriggers exists? As far as i know streaming the file to reduce memory workload is also possible with blobtriggers.

[paulehr]

just wanted to chime in that i hit this as well and it didnt matter if i was using powershell or python as a language, I kept hitting the same GRPC resource exhausting when trying to push a ~400MB file.

[pragnagopa]

MaxMessageLength is configured here: https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script.Grpc/Server/GrpcServer.cs#L19 for gRpc server. This effects all out-of-proc languages.

We should remove the size limitation.

Note: This would require all the language workers being updated to set max value for the channel message size.

[AnatoliB]

@pragnagopa Why don't we just make the host pass int.MaxValue to workers instead? This will require no changes to the workers (unless they don't actually use this value - this would be a bug that needs to be fixed in any case).

[pragnagopa]

This mostly being cautious on int.MaxValue might differ per runtime and architecture. Removing limit on the worker would be safest.

[AnatoliB]

I see, makes sense. However, updating all the workers may take some time. Do you think we could just increase the limit on the host to a large but safe enough number first? Something like 1 GB? (Actually, even int.MaxValue is only 2^31 - 1, which should be safe enough for all mainstream platforms/languages).

Eventually, we should just stop passing this value as a parameter, I agree with this long-term.

[pragnagopa]

However, updating all the workers may take some time.

I have created tracking items on all the workers that ship with the host. As this is a very small change, next release of the workers can include this change. Once the workers are updated, I will update the host.

[mhoeger]

@pragnagopa - I think I'm a bit confused. We can't remove max message length limits on the clients, because the default max is too low (~4MB for node) as we found here and here. As far as I know, this low default is still in place.

This means that each client has to define its own maximum - but that feels like it's going to be duplicated effort if we already have one central location to control the maximum (like @AnatoliB mentioned). Do we have evidence that each worker needs different maximum limits on the message sizes? I don't think it matters if int.MaxValue is different depending on runtime/architecture because that max value is not related to the throughput limits of the worker. I think until we know that different maximums are needed per worker (and what those maximums are), it doesn't quite make sense to have each client setting a maximum arbitrarily to it's own int.MaxValue?

[pragnagopa]

This work item is about removing the size limit and setting channel max receive and send size to the maximum allowed by the runtime, when host communicates over gRpc. Max size limit is security feature. As functions host only communicates over local host to gRpc client, we do not have to set this. Similar to C# functions, there is no limit on blob size that can be read by C# function. It is up to the user.

As I mentioned earlier in the issue, enforcing int.MaxValue from a .net host and requiring language workers to set that value for channel max receive and send size is not recommended as MaxValue can be runtime specific.

[mhoeger]

@pragnagopa - Got it! Sorry I was missing that we're just trying to get the largest possible values with the assumption that grpc.max_send_message_length is set with an "int" type (instead of another numeric type). This is what happens when you work in a non-typed language I guess :)

[amamounelsayed]

@pragnagopa Thank you so much for the update.
I have two questions, Please advice!
Why does send size is not recommended? as the host is the grpc server, so regardless of the language we should be aligned with the server message handle size, so whatever number you set we should obey, and I believe whatever number is picked should be supported by all run time by default.
Second question, as we are using grpc stream, does increase the message size will affect the throughput? My question in different way does the message size affect the grpc decision to when it flush the message?

[pragnagopa]

I believe whatever number is picked should be supported by all run time by default.

As I mentioned earlier, MaxValue for .Net might be different from MaxValue for Java. This issue is about removing the limit.

My question in different way does the message size affect the grpc decision to when it flush the message?

No. At at least nothing in the gRpc docs. You would see perf impact when the message size is large enough and is closer to max size set on the channel. This change will unblock customer scenarios who want their functions to read large function invocation payload without imposing arbitrary limits.

[amamounelsayed]

As I mentioned earlier, MaxValue for .Net might be different from MaxValue for Java. This issue is about removing the limit.

>

I agree about this part, but why does it matter? Whatever grpc max value set any language will set it. Meaning if MaxValue in .Net is x, Java will follow the x as x < the max java grpc message size(which I did not find any documentation for this value) Also specially we are setting to int.MaxValue which I believe it does not matter to be int.MaxValue or any other value from the worker side. Please advice.

[pragnagopa]

Java will follow the x as x < the max java grpc message size(which I did not find any documentation for this value)

Client and server configurations can be managed independently. .Net host can set to max value, and Java worker can choose to use that value or remove the limit and set to max value allowed by Java. During function invocation, if any of these limits are hit, function invocation will fail either because out of memory exceptions or gRpc channel hits limits.

[mhoeger]

@pragnagopa - how do max message lengths interact when set on both client and server? We have maxReceiveMessageLength and maxSendMessageLength set on the gRPC server in host here to ~134 MB. If Java worker, for example, sets its max message send and receive length to 1 GB, will the max then become 1 GB? Or is it constrained by what the functions host gRPC server sets to its max send/receive?

Also on the potential impact to throughput - I think we should make sure to test potential regressions to throughput before checking in this change. I know I've heard @amamounelsayed mention that there's still bottlenecks in our throughput that are unaccounted for, and this knob seems like it has potential to have impact? Seems worthwhile to verify (I will do so for the node worker)

[amamounelsayed]

Also on the potential impact to throughput - I think we should make sure to test potential regressions to throughput before checking in this change. I know I've heard @amamounelsayed mention that there's still bottlenecks in our throughput that are unaccounted for, and this knob seems like it has potential to have impact? Seems worthwhile to verify (I will do so for the node worker)

I agree, it worth to be safe. I will do the same for Java :)

[pragnagopa]

how do max message lengths interact when set on both client and server?

Documentation is limited around this. Found https://nanxiao.me/en/message-length-setting-in-grpc/, setting MaxReceiveLength to -1 removes the limit on the client. This simply means that Client can accept any message size sent by the server.

We have automated perf tests kicked off on the nightly dev build. The sooner you have your changes in , the longer we have to react to any regressions (if at all)

[amamounelsayed]

Documentation is limited around this. Found https://nanxiao.me/en/message-length-setting-in-grpc/, setting MaxReceiveLength to -1 removes the limit on the client. This simply means that Client can accept any message size sent by the server.

Yeap I checked this article today, unfortuantly grpc-java needs the value > 0. I believe this only valid to c.
I am planning to get my changes this week.

@pragnagopa are we going to stop sending the value or we will continue sending the value?

[mhoeger]

Sounds good - we should first verify that changing this value on just the client side will get us the message size we're looking for I think because that will impact whether or not each worker should be setting the value themselves.

[pragnagopa]

we should first verify that changing this value on just the client side will get us the message size we're looking for

Just to clarify, updating on the client will only remove the size limit on the client size. Host changes are needed to actually increase the Send/Receive size to support sending larger function invocation payload and receive larger function invocation response.

[amamounelsayed]

@pragnagopa @mhoeger I tested the throughput java worker with 4MB, 128MB, and the MaxValue MB, there is no impact so far. Thank you!

[mhoeger]

Thanks @amamounelsayed !!

After talking to @pragnagopa , if the grpc server on the functions-host side has a smaller max send/receive, this will be the bottleneck. This change is to make sure that clients are not sent a max message size that will be too large for them to safely handle. For JavaScript, a number is defined has a maximum "safe" value (meaning arithmetic with the value will not be correct), but the maximum integer value is 1.7976931348623157e+308. For my part, because the limit will always be constrained by the C# side, I'm going to close the Node.js issue because whatever the host sends will be a safe value.