Azure-functions-host: Timer Function on Consumption plan stops tiggering

[pgroene]

I have a Timer Function with a Consumption service plan. When I schedule the function to run once every hour it seems to stop after a few hours of running.

Repro steps

Provide the steps required to reproduce the problem

1. Create a function app with ARM templates (see arm used setting below)

2. Deploy the a timer function app with cron schedule 0 24 * * * * (deployed from vsts with Azure App Service Deploy)

3. Check once in while that if the timer is triggered.

Expected behavior

The Timer Function keeps running every hour

Actual behavior

After some hours the timers stops. This weekend it did not run till I opened the monitoring this morning and it started running again.

Known workarounds

It seems that when the interval is smaller (5 min) it keeps working. However this can also be because I do not know why it stops when using an hour.

Related information

• Programming language used
• Links to source

Arm templates:
Microsoft.Web/serverfarms:

{
            "name": "[variables('service_plan_name')]",
            "type": "Microsoft.Web/serverfarms",
             "sku": {
                "name": "Y1",
                "tier": "Dynamic",
                "size": "Y1",
                "family": "Y",
                "capacity": 0
            },
            "kind": "functionapp",
            "apiVersion": "2015-08-01",
            "location": "[resourceGroup().location]",
            "properties": { "name": "[variables('service_plan_name')]" }

        }

Microsoft.Web/sites:

{
            "type": "Microsoft.Web/sites",
            "kind": "functionapp",
            "name": "[parameters('functionapp_name')]",
            "apiVersion": "2015-08-01",
            "location": "[resourceGroup().location]",
            "properties": {
                "name": "[parameters('functionapp_name')]",
                "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('service_plan_name'))]",
                "siteConfig": {
                    "appSettings": "[variables('appsettings')]"
                }
            }       
        }

where variable appsetting:

  "appsettings":[
                        { "name": "FUNCTIONS_EXTENSION_VERSION", "value": "~1" },
                        { "name": "AzureWebJobsDashboard", "value": "[parameters('storage_connectionstring')]" },
                        { "name": "AzureWebJobsStorage", "value": "[parameters('storage_connectionstring')]" },
                        { "name": "WEBSITE_NODE_DEFAULT_VERSION", "value": "6.5.0" }
        ]

cron expression in timer:
"schedule": "0 24 * * * *"

The code for the Timer is in C#

[pgroene]

The plan used for the app is a consumption plan.

Screenshot from logging:

[davidebbo]

Are you still seeing this? How did you create the function? Was it through portal, or some other type of deployment?

Can you share your function app name (that's the app name, not function name!), either directly or indirectly? This will help us investigate. Thanks!

[pgroene]

The name of the apps we saw the behavior:

downloadSitecatalistFunctionp -> we saw the issue here, however the last we seems te be fine with "schedule": "0 */5 * * * *" (on next redeployment we will remove this app and redeploy a new app name)
dataimportfunctionst -> this one is currently redeployed and now running with "schedule": "0 24 * * * *" for testing to see if the error still occurs (just deployed)

We are deploying the application with these arm templates:

App Service plan/App Service Microsoft.Web.zip

Storage: Microsoft.Storage.zip (deploys storage with encryption enabled)

The templates are called as linked templates from an master template. The templates are deployed from an VSTS release pipeline. The App is deployed with Azure App Service Deploy VSTS task that just deploys the directory. blog post on vsts pipeline

[jhoerr]

I'm seeing this behavior too, much as described here. I would like my function to be executed once a day, however the timer is never triggered. If I modify the timer from the portal to something in the near-future (say 10 or 20 minutes from now) the timer triggers as expected. However if I set it to something further out (say 4 hours from now) the timer does not trigger.

Other notes:

• My function app name is iga-tracker-functions.
• My function app is on the Consumption plan.
• The app was created on the portal and is continuously deployed via GitHub

[davidebbo]

@pgroene not sure if you intended to trigger it every 24 hours, but what you have means every hour at 24 minute pas the hour (2:24, 3:24, ...). Of course, it should still work, but just making sure that's what you meant!

[pgroene]

The idea is that we have a window where we can pickup a file, every hour between x:20-x:30. The trigger should run 2:24, 3:24, ... indeed.

[davidebbo]

@pgroene Ok, from what I see, it is working fine. Here is the time I see of your last few executions (UTC):

2017-02-16 17:24:01.7661206
2017-02-16 18:24:01.5998242
2017-02-16 19:24:01.7349398
2017-02-16 20:24:01.6091199
2017-02-16 21:24:02.5931206

[davidebbo]

@jhoerr right now, you have your timer set to '0 */10 * * * 1-5', and looking at the log, it is working fine, firing consistently every 10 minutes.

If you have a function that is not currently working as expected, please let me know.

[jhoerr]

@davidebbo Sorry for the confusion. I changed the timer to 0 */10 * * * 1-5 in order to keep it alive. The original problematic value was 0 30 19 * * 1-5. Would it be helpful if I created a dummy function to repro the issue?

[davidebbo]

@jhoerr Yes, in order to investigate, I need to have an app in a state where it doesn't behave as expected.

But looking at your app, I think you're hitting #995, due to having set WEBSITE_TIME_ZONE. The fix for that is still a week or so away.

[jhoerr]

@davidebbo I'll add an additional function and follow up here when it's ready. In re: #995, am I correct in think that would be a separate (but related) issue?

[davidebbo]

@jhoerr yes, it's a separate related issue, which is fixed but not yet deployed. Until it is deployed, things will not work correct of you set WEBSITE_TIME_ZONE. So make sure your repro does not have that (unless that's the problem for you to begin with, in which case you really just want the other issue).

[pgroene]

@davidebbo It is correct that the apps are working fine now. This week we did run with 5 min intervals, and it worked fine. I changed it back to 1 hour intervals to see if we can have a repro. I expect that the app will be stopped Monday. However monitoring seems to trigger a resume of the timer, and keeps it alive. So best guess is that it will be stopped after the weekend when nobody is looking.

[jhoerr]

@davidebbo Thanks for the suggestion. I have a workaround in place so I'm reluctant to change the WEBSITE_TIME_ZONE setting. I think the prudent thing at this point is probably to wait until the time zone fix is deployed and then reassess whether the issue reported here persists. Thanks for your patience!

[davidebbo]

@pgroene Well, I expect it to keep firing reliably at 24 minutes past the hour. BTW, one thing you can do is add some queue or blob output trigger that will mark when the function is called. This way, you can easily check whether it's getting called (e.g. in storage explorer) without running into quantum observer situations ;)

[pgroene]

The last week all functions (3 instances) kept running in time, it seems that the problem is solved or was some kind of exception in the function app. I think this issue can be closed, if I see the behavior again we can reopen the issue and see if the can catch it when it is happening.

[davidebbo]

@pgroene thanks for confirming!

[sburman]

I am seeing this behavior on my timed trigger (on consumption plan).

At the moment, to shave back to the simplest implementation that was practical my function simply pings a slack webhook with a basic message.

Current schedule: "0 0 0/1 1/1 * *" - should run on the hour, every hour.

But the function doesn't run unless I have recently been active (via deployment, or logging into the azure portal and looking at my functions).

On this 'reactivation' it will immediately run the task once (I assume as a catchup for the missed trigger). And then if the top of the hour is close enough it may trigger successfully once more - but then never again.

The slack message is very basic but includes a timestamp from the server and this is my recent output:

FindNewFairplayXmlFiles.Run, from acutefunc. 2017-03-07T23:54:57.2507560+00:00
FindNewFairplayXmlFiles.Run, from acutefunc. 2017-03-08T00:00:00.0162141+00:00
FindNewFairplayXmlFiles.Run, from acutefunc. 2017-03-08T02:55:46.0141007+00:00
FindNewFairplayXmlFiles.Run, from acutefunc. 2017-03-08T03:00:00.2716406+00:00

Function app name is: acutefunc
Function name is: FindNewFairplayXmlFiles
Deployed via continuous Github as Precompiled Azure Functions.

[davidebbo]

@sburman the data in the scale controller shows this FindNewFairplayXmlFiles function as having an http trigger and not a timer trigger, so that is quite odd. That data dates back to Feb 16.

Was this function changed in the meantime? Can you try hitting the Refresh button in the lower left corner of the function portal UI?

[sburman]

It was changed, yes. Previously it was a http trigger. Have refreshed - not sure what this achieves on the backend as it was already listing as timer trigger in function.json and also the 'Integrate' menu option in the UI.

[davidebbo]

@sburman thanks, I will dig in and try to figure out what's going on. I'm wondering if it's somehow not liking your cron expression. Can you try 0 0 * * * * instead? That's a simpler way of saying the same thing (i.e. anytime the seconds and minutes are zero, meaning every hour). Issue may very well be unrelated, but let's try that to isolate. Thanks!

[sburman]

New schedule in place. Deployed and refreshed.

[davidebbo]

@sburman from what I can see, it seems to be working now, so maybe it was the cron that was throwing it off. Please leave it running this way so we can make sure it works consistently.

I'll then try to make sense of the cron issue.

[sburman]

I'll check my slack channel in a few hours to see for sure and report back here :wink:

[sburman]

Timer trigger behavior looks to be all good for now. Something funky in the way that original hourly schedule was being handled though.

[davidebbo]

@sburman Thanks for confirming. I opened #1275 to track this on its own.

[shudson302]

I am still having this behavior. Was there ever a fix? The name of my app is generatethumbs. It triggers off of a storage account when a blob is created, it calls computer vision api to generate a thumb and outputs that image back to the storage account.

It works fine while I have the azure portal open. But after a while it will quit triggering off of new blobs in the storage account. If I then open the azure portal and look at the logs, then all of the ones that didnt run will start and run successfully.

[davidebbo]

@shudson302 there are no known issues related to timer triggers. But what you describe sounds unrelated to timers, so I suggest opening a separate issue.