Azure-functions-host: Language workers: gRPC max message length is defaulted to 0.

[peterhuene]

Repro steps

1. Start the functions host out of the dev branch (HEAD = d87c8b0a972800434455b74e8253bbf2675956b9) with a script root containing at least one language-worker based Azure Function (node, java, etc.; in my case, it was Rust).

Expected behavior

gRPC server is instantiated with a valid max message length and worker initialization succeeds.

Actual behavior

gRPC server is instantiated with a max message length of 0 and worker initialization fails.

In the case of my language worker, the worker panics because it fails to perform worker initialization with the host.

info: Worker.Rust.0fafa35f-81cd-4538-94b6-e7d88de70c44[0]
      thread 'main' panicked at 'failed to receive message: RpcFailure(RpcStatus { status: ResourceExhausted, details: Some("Received message larger than max (41 vs. 0)") }).', azure-functions/src/rpc/client.rs:140:33
info: Worker.Rust.0fafa35f-81cd-4538-94b6-e7d88de70c44[0]
      note: Run with `RUST_BACKTRACE=1` for a backtrace.
fail: Worker.Rust.0fafa35f-81cd-4538-94b6-e7d88de70c44[0]
      Worker encountered an error.
System.Exception: rust_worker process with pid 21102 exited with code 101

PR #3163 removed all calls to ApplyLanguageWorkersConfig:

https://github.com/Azure/azure-functions-host/blob/d87c8b0a972800434455b74e8253bbf2675956b9/src/WebJobs.Script/Host/ScriptHost.cs#L843-L874

As a result, MaxMessageLengthBytes in the script config is now defaulted to 0.

This causes the gRPC server and clients respecting the --grpcMaxMessageLength option to error with any messages that it sends or receives larger than 0 (i.e. every message).

Known workarounds

None unless there's some other way to get the host to respect languageWorkers.maxMessageLength.

Related information

Also related to the changes from PR #3163, I no longer get any logging from dotnet run for the WebHost (I get the Host log only if I do ASPNETCORE_ENVIRONMENT=Development. Is this by design or is there some additional work now required for proper host configuration? I was at least able to get the output with the "command line" host to assist in debugging the max message length issue.

[peterhuene]

Here's the error that occurs for the node worker:

fail: Worker.node.c7e39b56-2fc8-4f79-8dda-36f6b03f790c[0]
      Error: Connection info missing
info: Worker.node.c7e39b56-2fc8-4f79-8dda-36f6b03f790c[0]
          at Object.startNodeWorker (/Users/peterhuene/src/azure-functions-host/src/WebJobs.Script.Host/bin/Debug/netcoreapp2.1/workers/node/worker-bundle.js:28412:15)

This is because grpcMaxMessageLength was passed as 0 and !grpcMaxMessageLength is truth-y (don't you just love JavaScript?), so it bails thinking there's a missing argument.

Still, I'd expect the gRPC implementation to fail if grpcMaxMessageLength were respected as 0.

[paulbatum]

@peterhuene thanks for the report! We'll get this fixed before these bits are released and we should also identify and close the test gap.

@pragnagopa can you take a look

@brettsam FYI see the note at the bottom about logging changes. I assume this is something you're already tracking?

[pragnagopa]

Thanks. We are aware of this issue. @brettsam and @fabiocav are looking into fixing this ASAP. I will update the code to ensure we do not explicitly pass 0 for maxMessageLength

[brettsam]

Yes, a logging PR will be coming soon that will get everything wired up correctly.

[peterhuene]

Thanks everyone 😃 apologies if you're already tracking this work. I like to rebase my host repo to upstream frequently (and update the protos) to ensure I keep in sync with changes for my worker implementation. Cheers!

[fabiocav]

Resolved by #3253

[fabiocav]

@peterhuene thanks again for reporting!