Azure-functions-host: Simultaneous timeouts in JobHost worker cause "Number of initialized language workers exceeded"

[mhoeger]

Two functions can time out at the same time and cause the same worker to crash for independent reasons. If this channel is a webhost channel, we handle that correctly. However, if this is a jobhost-scoped channel, we do not handle for this condition:
https://github.com/Azure/azure-functions-host/blob/2d0ac22afb86041a5b81797b6aaf16049a673198/src/WebJobs.Script/Workers/Rpc/FunctionRegistration/RpcFunctionInvocationDispatcher.cs#L321

Per Yong:

1. App is NodeJS with HTTP triggered functions (could be any app).
2. Customer has two functions that trigger timeout exception (30 seconds) almost at the same time.
3. The same channel running those functions is restarted & disposed almost at same time.
4. Two node.exe process are started almost at the same time as well while listening on the same tcp port 52088, which is unexpected.
5. This leads to the final exception Number of initialized language workers exceeded:2 exceeded maxProcessCount: 1 while executing the function:

Note that any worker can get into this state after at least one fatal worker exception on the instance.

cc: @pragnagopa and @yojagad

[pambardekar]

I am hitting this bug pretty regularly since last couple of weeks. I am using Node JS Function App (v2). Node JS version is 8.11.2.

[radmike]

@mhoeger / @yojagad saw this was closed -- any idea when this will be released to Azure?

[shakirshakeelzargar]

I am facing this issue in recently in the App Service Plan.
Message Exception while executing function: Functions.etlQueProcessor Number of initialized language workers exceeded:7 exceeded maxProcessCount: 1
Exception type System.InvalidOperationException
Failed method Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcFunctionInvocationDispatcher+d__52.MoveNext

The function app is python based and is invoked by a queue message.
The issue never came u in last one year and nor restarting the function app resolved the issue.

[pragnagopa]

@shakirshakeelzargar - Can you please create a separate issue with details:

• Invocation Id
• Approx time range when you saw the exception
• Region where function app is running.