Azure-functions-host: Function not triggered by new queue item

[edspivak]

I have an Azure function with Azure Queue Storage trigger. After some time of inactivity (I usually notice it next day after it's been idle overnight), it stops responding to a new queue item. To revive, I have to go to that function in Azure portal, and it magically starts working again. Since I am on Consumption plan, I can't configure "Always On". It should just work, right?

Function App name: cm-image-ops
Function Name: image-merge

Please let me know if you need more information.

Thank you.

[Kimmen]

We have the same issue, but with Event Hubs instead. To recap: ours functions stops reacting on new events when there has been some time of inactivity (e.g. 20 minutes of inactivity). I've heard that the web host goes into idle mode when inactive. Once in idle mode it has hard time waking up.

Workaround
To prevent the web host to go into idle mode you can create a new Function with a Trigger-input. I've set the Timer to trigger once every third minute. The only thing the "keep-alive"-function does is writing to output.

[mathewc]

Yes, in Consumption things should "just work". Any hacks like creating a timer trigger input should of course not be needed :( We need to dig in and find out - sorry we missed this issue for so long guys. We'll take a look ASAP.

[davidebbo]

@edspivak can you give me an example time (preferably UTC) when you added an item to the queue and it didn't get processed? From what I can see, the scale controller is correctly monitoring the queue, so it's puzzling that it wouldn't work.

@Kimmen: I'd also like to look at your case if you can share the app name, function name and UTC time. Thanks!

[edspivak]

As an example:
We added an item to the queue on Fri, 03 Mar 2017 23:47:59 GMT. It was processed on Mon, 06 Mar 2017 17:17:58 GMT when I actually went to my function in Azure portal.
This was the message in the queue: c8138463d5604d68a4290ce2fbc51c33

There were many more messages stuck there. Prior to this, the last time the function observed working was Tue, 28 Feb 2017 17:34:29 GMT. In other words, if there are no messages in the queue for some time, function app shuts down.

Please let me know if you need more info.

[davidebbo]

@edspivak what's strange is that the only record I can see of the scale controller receiving your triggers is on March 6:

• First at 17:33, with the timer trigger set to 0 */1 * * * * (every minute)
• Then at 19:24, it got changed to 0 0 0/1 * * * (every hour)

Prior to that, how did you deploy your Functions? Was it all done via the portal? Do you remember on what date they were created?

Now that the scale controller has the information, I suspect you'll see everything working (no need for timer), but we should try to understand what happened earlier.

[edspivak]

@davidebbo I think you are looking at the wrong function. The function name is image-merge. The other function based on timer trigger is my work around experiment.

[davidebbo]

@edspivak No I know, I'm seeing both. I just didn't mention the other one because it didn't change between the two events. Those two times are the only ones where anything was received by the scale controller for your whole function app.

So two things:

1. Please verify that everything works now without the timer and with the portal close (and please do give it a few minutes to react, as that is normal in some cases)
2. If you recall, please share time of initial image-merge publishing, and method of publishing. Thanks!

[edspivak]

@davidebbo I published this function probably some time at the at beginning of February. I published it manually in portal and uploaded all the dlls by drugging and dropping.
I deleted my other function. Let's see if it works.

However, I can re-create the same scenario by creating other function app based on queue trigger. I will monitor and send you the result. Please keep this issue open.

Thank you very much for looking into it.

[davidebbo]

@edspivak one strange thing I found is that back on January 1st, the scale controller is showing errors related to the the Function App missing the AzureWebJobsStorage App Setting. But as of right now, you do have it on the App.

Do you remember how you created the Function App initially (e.g. was it via https://functions.azure.com/, or through the standard portal.azure.com creation workflow?)? And do you remember anything related to having to fix up those settings at the time? It started out without them, and somehow got them later.

[edspivak]

@davidebbo Function app was created through standard portal portal.azure.com. I think I remember having this error, and probably I fixed it. I do not remember all the details now, but I think I had to check that "slot setting" checkbox. I might be mistaken, though.

[Kimmen]

@davidebbo I can't reveal the name in public for the moment. Can I do "indirectly" somehow? Can I create a new dummy Function app on the same Consumption plan (doesn't seem like it inte the portal)?

But we deploy the Functions using ARM Templates and VSTS. Does it help you to see the template?

[davidebbo]

@Kimmen absolutely, the indirect method works too. And seeing the ARM template is probably not useful here.

[davidebbo]

@Kimmen See also https://github.com/Azure/azure-webjobs-sdk-script/wiki/Sharing-Your-Function-App-name-privately for alternate instructions that are a bit simpler for Function Apps.

[Kimmen]

@davidebbo Went for the alternate instructions.
First I have the last successful completion prior to closing the portal that executed as expected:
_2017-03-08T08:03:34.250 Function completed (Success, Id=e3b7299c-f22c-4dd3-bf17-5d6a0f55e94f)_

I verified that I still got the issue, so there are some other successful runs. But now I have one event that hasn't triggered yet. If you want to take a peek.

Can't remember when we generated environment last time, maybe January/February.

[davidebbo]

@Kimmen Looks like you are running into #360, triggering the error:

EventHub name should not be specified as EntityPath while using this Overload. Instead, please use 'path' parameter.

So you'll need to fix your connection strings accordingly, and that'll get you going. Clearly, we need to improve our error handling here. :)

Anyway, this is definitely unrelated to @edspivak's issue, so let's discuss separately to avoid threading the two in this issue.

[davidebbo]

@edspivak is your Function now getting correctly triggered?

[edspivak]

@davidebbo My function is working now because it is being used. It usually stops working after being idle for some time.
Just as an experiment, I created a separate Function app and function today's morning with queue storage trigger, and processed several messages. I want to see if my experimental function is still working after being idle for at least day and a half. I will let you know Friday.

[davidebbo]

@edspivak I suspect everything will run fine on your test app, and will also now run fine on your original app. I think the issue was caused by the earlier history of that app discussed above (missing app settings).

[Kimmen]

@davidebbo As you see I created a new issue.

[thusnjak]

Hi everyone,

I'm experiencing this issue, did some research and didn't find the solution. My queue triggered function app stops working very soon after I close the portal and the items get stuck in the queue until I log back in to the portal.

I would be very happy if someone could give me a hint or check this on my account - function app name ends with feedprocess and is on the same service plan with sa-v4-reference. There are 3 functions and the critical one is QueueProcessLivescore.

Please let me know if you need more information.

Many thanks.

[davidebbo]

@thusnjak I think issue might be that you are using a Connection String instead of an App Setting from the storage connection. Can you try switching to an App Setting?

[thusnjak]

@davidebbo thank you - now works as expected.

[feratur]

I now have the same problem as @edspivak . I have two sets of functions, the first set works as expected, but the functions in the second set suddenly started behaving just like @edspivak described: after a period of inactivity the function just doesn't get triggered by the messages in the queue until I select the function in the Azure Portal.

I described the problem in another issue:
https://github.com/Azure/azure-webjobs-sdk-script/issues/1858

[anhduc130]

I also have the same issue. Could anyone please explain what it means to use App Setting instead of Connection String? I think I do have my Connection String in Application Settings.

[thusnjak]

Use Add new settng (marked green) instead of Add new connection string (red).

You will know you did right if your function under integrate doesn't have blank under storage account connection dropdown... (marked green below must not be blank)

[anhduc130]

I really appreciate your quick answer. I did the same as you described.
However, over the weekend the queue was not triggered at all, and on Monday it seems to go to idle mode and stops working until I open it in the Portal.

[davidebbo]

Can you check Function App details per here, as well as a sample UTC time when it didn't fire and should have? Thanks!

[anhduc130]

Yes. Here is my function app's info and I am in Vancouver, Canada.
2017-10-17T18:15:17.197 Function started (Id=2d53b7e6-347c-45ab-936a-5afdbf122fdd)

Thank you so much.

[davidebbo]

@anhduc130 can you also provide the UTC time of one case where you had items on your queue and they didn't get picked up?

I wonder if having the queueName set to updatedtrackingitems%QUEUE_NAME_SUFFIX% somehow throws off the scale controller. @cgillum, do you think it might? @anhduc130 as a test, does it work if you hard code it to have the full name?

[anhduc130]

Unfortunately, I did not keep track of the UTC time where the items were on the queue and did not get picked up.

what happened was that on Monday, I made a POST to the webhook that puts data on the queue so that it can get picked up and executed. However, it seems like the data was not executed. Then I opened the portal, and suddenly the lots of logs pop up and it WORKED!!!!!

[davidebbo]

Ok, if it happens again, please keep track of the time (doesn't have to be exact) and let us know. This way we can look at our queue monitoring logs and try to figure out why it didn't detect it.

[adamwebbcollabware]

Experiencing the same issue with functions seemingly no longer polling queues for messages. Can provide function name and timestamps.

Happy to open another issue if that is the correct process.

Thanks.

[spr4ttel]

This is still happening, and it's stopping our entire production workflow.

We have had big problems with it over the past 6-8 months, seemingly working better Dec 2017 - Mar 2018, but now being a really big problem again. It's intermittent on different Resources/Resource Groups - some work, some don't. Identical deployments. Messages get into the Service Bus Queue, but are not picked up. No diagnostic logs are generated whatsoever.

If I run the function app manually via Azure Portal, it will pick everything up. Then dies again, doing nothing. Restarting the function app does not help. _Sometimes_, simply clicking around in the UI seems to wake it up (which is alarming).

[davidebbo]

@spr4ttel this thread is tracking Storage Queues, and there are no known issues at this time. Please open a new issue to track your case, and provide the app name/ UTC time info as above for investigation (feel free to add a link from this issue for tracking).

[spr4ttel]

Thanks, I will do so. But that's actually a good clue, isn't it: exactly the same symptoms for Service Bus Queue triggers.

[mpseidel]

Just found this issue. I posted about this in the forums over here (Function running in Linux Container Preview waiting for queue messages).

Making sure the runtime never goes idle by keeping it busy every 5 minutes seems to have fixed it for us but still this seems like a glaring bug or has it been fixed already?

[davidebbo]

@mpseidel please open a new issue (you can link it), making it explicit in the title that you are using the Linux Container Preview. Everything in this thread was with the Windows offering.

[mpseidel]

@davidebbo thanks for clarifying. I've already contacted Azure support and posted in a forum. I'd be very happy to see the Azure product teams are actively picking up and tracking issues like these themselves instead of relying on customers to create issues for them. I'm happy to help but I don't have time explaining issues over and over again.

[davidebbo]

@mpseidel It should be fine if you started a thread in the forum.

[iOnline247]

I'm experiencing this problem as well. I'm using a queue trigger and it's wired up using the application settings. What's strange is I'll see messages every the queue and then disappear as of it was picked up. The message doesn't go to the poison queue either.

[TurtleBeach]

Part of weird queue behavior is due to the invisibility period - some part of the framework removes (peeks at?) a queue message so it is not visible in the portal, but it also hasn't yet been processed by the function that is waiting to be triggered by the queue. There is a default time that the queue is monitored, which is I believe on the order of minutes, so a message can seem to be in limbo for a while. I can't immediately find the docs for the queue parameters (thanks, MS...) but they do exist; take a look - you can shorten the time.

[iOnline247]

I'll take a look, but I'm experiencing delays of up to an hour, in some cases. That doesn't seem right. The average processing time for the function is ~3 seconds.

[Ppiwow]

Hi,

We experiencing exactly the same issue. After 30+ minutes our function triggered by queue item stop working and we need to just log in int o portal.

We deploy function with VSTS (now Azure DevOps) release module with AppService block.

What information you need to can investigate the issue.

[TianqiZhang]

I met the same issue recently and according to #3600 , if you used MaxQueuePollingInterval in your host.json, you must set it to a timespan.

Yeah, tell me about it.

Hope this solves your issues.

[kapilmehtagit]

Hi, I have had the exact same issue to the queue triggered function. It seems to be not picking up the queue messages for several hours, until I logon to the portal and have a poke at the logs, and then suddenly (in a few seconds or sometimes minutes), it springs to life. After this, it is happy to continue processing any new messages.

It is a Visual Studio based C# queue triggered function, which is deployed from Visual Studio to the function app. The queue name is parameterised and picked up from App Settings. I was wondering if it would help to hard code this name in the function trigger definition rather than parameterising it, but it is a difficult one to test and prove as it could take a few days for the issue to resurface.

Always On is On

Function details:
2018-10-31T12:06:08.478 Id=d548e7d7-89ec-4cbd-b5dc-5963e05edf53. UK South

Also, if this is of interest, I have a timer function (again pre-compiled C#) which runs every 3 minutes or so in the same function app to do some processing, which theoretically, keeps the host loaded and warmed up. There is also an http triggered function in the same function app that stays pretty responsive across days.

Happy to help with any troubleshooting.
Thanks for your help.

[mpseidel]

Meanwhile migrated to the v2 runtime (running inside docker).

All seemed well now for my one function working off of a storage queue that is kept "busy" by pushing a message to the queue every 5 minutes.

Another function working off of a different queue in the same storage account seems to suffer from the same old issue again. Function at some point just stops getting triggered.

Is anybody looking at this?

[MoimHossain]

This is still an issue. Reading a lot (here and Stackoverflow) - and applying almost all the suggestions I found didn't make any difference.

It's a c# function (VS developed) and pushed with Azure Pipelines. After some inactive period, the storage queue trigger doesn't work. Login in to the portal solve this within couple of seconds. This is seriously a BUG - which I believe should be properly investigated. This is a annoying blocker for our applications.

Anybody looking into this? Happy to share more info as needed to trouble shoot this.

Thanks

[cgillum]

I strongly suggest opening separate issues for these, and providing as much detail as possible about your setup (deployment mechanism, region, time range, etc.). There are unfortunately lots of possible reasons one might see these symptoms and each one would need to be investigated separately. Creating new issues is the best way to ensure someone on the team investigates promptly (new active issues get looked at more quickly than old closed issues). Thanks for your understanding - and we hope to get these resolved!