Azure-functions-host: azure timer function not executing all of sudden

[Hemantkaushik]

Investigative information

Please provide the following:

• Timestamp:2021-02-16T06:21:37.544
• Function App version:3
• Function App name: func-balance-adjustment-processor
• Function name(s) (as appropriate):AccountBalanceAdjustment
• Invocation ID:093becd6-f25c-4f49-a47c-635f9e0a6ba1
• Region: Australia East

on top mentioned runs are Manual runs for the function.
My function stopped running after 2021-02-14 04:00:01.111 UTC time.
It was scheduled for every 1 hour run, currently changed for every 5 min in configuration settings.
I have tried to redeploy it , same issue.
Please help

[v-bbalaiagar]

Hi @Hemantkaushik, I went through the logs for the time stamp - 2021-02-13 00:00:00 to 2021-02-15 23:59:00. I see the error "Singleton lock renewal failed for blob 'func-balance-adjustment-processo/host' with error code 409: LeaseIdMismatchWithLeaseOperation". Here is how you can troubleshoot the issue.
https://github.com/Azure/azure-webjobs-sdk-extensions/wiki/TimerTrigger#troubleshooting
Related information:
azure-functions-host/issues/7025
https://github.com/Azure/azure-webjobs-sdk-extensions/issues/249
azure-functions-host/issues/1864

[msftbot[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[tareksharbak]

Hi,

I'm having an issue similar to the one described above.
My function uses a Timer trigger (*/5 * * * *) to run every 5 minutes.

It works perfectly the first few times after deploying the function (I'm deploying using VS as a zip deploy), but then it stops triggering until I open the logs, where it continues working as usual. It seems like the timer trigger is failing unless I have the streaming logs open.

Here are the details of my function:

• Function Name: XboxCheckerFunction
• Function app name: XConsoleCheckerFunctionApp
• Region: West Europe

In the screenshot above, you can see how there was a gap. This happened between the time I closed the portal and opened it again and started the log stream.

Thank you,
Tarek

[v-bbalaiagar]

Hi @tareksharbak, I went through the logs for the time stamp- 2021-02-23 13:00:00 to 2021-02-23 15:15:00. I see the error
"The 'Xbo' function is in error: '/5 * * * *' is not a valid CRON expression". Can you please try with this expression: 0 */5 * * * *? Functions supports an extra field for seconds.
https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-timer?tabs=csharp#ncrontab-expressions

[tareksharbak]

Thanks @v-bbalaiagar!
The solution you posted works. However, the peculiar thing is that the "invalid" CRON expression seems to work fine when the log stream is open in the portal (it works at the correct intervals) and also while debugging with Visual Studio, which makes this very difficult to detect.

Is there a way for us to check the logs that you checked on our own? This will help debug and resolve a lot of issues faster.

Thanks again for you help.

[msftbot[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[v-bbalaiagar]

Hi @tareksharbak, Can you please go through here to view azure functions error logs.
To diagnose failures in your monitored applications, Please go through the document in the below link.
https://docs.microsoft.com/en-us/azure/azure-monitor/app/asp-net-exceptions#diagnosing-failures-using-the-azure-portal

[tareksharbak]

Thank you @v-bbalaiagar for the resources!

[v-bbalaiagar]

Hi @Hemantkaushik, Closing the issue, In case of any further concerns regarding this issue, please feel free to re-open the issue.

[Hemantkaushik]

Hi mate

I am still getting this error and trying to resolve.
I have improved my code to complete async, which wasn't earlier.
staging environment worked couple of days ago and It is not working now.

Error I am getting is :

Singleton lock renewal failed for blob 'func-balance-adjustment-processo/host' with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 2021-03-09T21:12:03.256Z (15172 milliseconds ago) with a duration of 4 milliseconds. The lease period was 15000 milliseconds.

Regards
Hemant

[anthonychu]

@tareksharbak @v-bbalaiagar The '*/5 * * * *' is not a valid CRON expression is a known issue that @mathewc has created a fix for. The issue is that the seconds portion of the cron expression is optional in the functions host, but required in the scale controller. This causes the error you see and also results in the app not scaling up when it's been scaled down to 0.

[Hemantkaushik]

It is running for me.
Thanks @v-bbalaiagar