Azure-functions-host: Service Bus with Sessions triggered v2 Function scales down while incoming messages remain constant

[dwoldo]

Investigative information

Please provide the following:

• Timestamp: 2019-10-15 14:32 UTC
• Function App version (1.0 or 2.0): 2.0
• Function App name: velocity-func-sb-prod-eu2
• Function name(s) (as appropriate): velocity-sb-queue-proxy
• Invocation ID:
• Region: EUS2

Repro steps

We cannot reproduce this issue but have witnessed the pattern where a consistent number of messages are enqueued to the Service Bus Premium queue but the Azure function execution count begins dropping causing the queue to grow. After a period of long period of time, 20-30m at times, execution count spikes processing the queue until the next occurrence.

In last 7 days, occurrences:
2019-10-08 17:12 UTC
2019-10-11 18:27 UTC
2019-10-11 20:52 UTC
2019-10-12 01:02 UTC
2019-10-12 09:37 UTC
2019-10-12 12:27 UTC
2019-10-15 11:51 UTC
2019-10-15 14:31 UTC

Expected behavior

When a consistent number of items are enqueued, the instance(s) of the AzFx should keep up with the queue.

Actual behavior

During noted periods, the execution count for Service Bus triggered Azure Function begins to decrease causing the Service Bus queue active messages to grow quickly. There is no spike of incoming messages to the queue, it remains consistent. The execution count does not.

Known workarounds

Restarting the Azure Function (it takes ~10m for this to affectedly begin draining the queue in a positive way).

Related information

I do not believe the Azure Function is properly scaling the instances or continues to keep pace with the queue. The response time to restore normal queue processing operations is upwards of 20-30 minutes at times. We have no levers or knobs to pull that we are aware of.

We've had a case open with support for 10 days with Support and I am consistently told this is nominal. Case #: 119100524000400

[ankitkumarr]

@dwoldo, I can see that the support case was escalated to us. You should be receiving an analysis for the root cause in that ticket.

We found a rare bug in the scaling service, and have deployed a fix. It should be rolling out next month.
Please let me know if you have further questions.

[dwoldo]

@ankitkumarr ,

Thank you! One experience I had was regarding metrics. For several hours at a time I could not observe function metrics, no matter the metrics-- execution count, memory, requests, etc. Will this be resolved as well?

Thanks!

[ankitkumarr]

@dwoldo, I believe so. The actual issue caused metrics to not show up. I can confirm this with our team and get back to you.

[dwoldo]

Hello @ankitkumarr Do you have an update on this? Has this bug been resolved in all US Regions?

[fabiocav]

@ankitkumarr is there any follow up required here?

@dwoldo are you still experiencing issues?

[dwoldo]

@fabiocav We've remained committed to dedicated functions until this has been confirmed resolved. So I can't say whether or not we are affected. We are, however running out of runway with our ability to scale out. Thanks for following up with me.

[dwoldo]

@ankitkumarr @fabiocav Any update on this?

[ankitkumarr]

@dwoldo, apologies for the long delay here. That's entirely on me.
The deployment was completed and you shouldn't see such issues.

Please let us know if you have any further questions.

[dwoldo]

Thank you for the info, it is indeed great news. I will close this issue.