azure-functions-host 🚀 - Function not triggered by blob

Can you confirm that you're on a Dynamic App Service Plan? Azure Functions can be used on both a classic app service plan (where you do need to manually turn on Always On, see https://github.com/projectkudu/AzureFunctionsPortal/issues/557), or a dynamic plan.

lindydonna on 20 Sep 2016

Yes, it is using a Dynamic App Service Plan. I created the function from the 'Azure Functions page https://functions.azure.com, so the service plan was auto-created. Here's the service plan:

azurefunctions1

And the Azure Function:
azurefunctions2

JohanKroese on 20 Sep 2016

@tohling will investigate, it definitely sounds like a bug.

lindydonna on 20 Sep 2016

I am also experiencing the same behavior for my function app with blob triggers.

bullsfan127 on 20 Sep 2016

@JohanKroese & @bullsfan127 can you check whether your storage account has diagnostic logs enabled for blob? The notification system relies on this.

To check:

in Portal, find your storage account
go to Diagnostics
Check that Status is On and 'Blob logs' is checked.

davidebbo on 20 Sep 2016

One more thing: if blob logging is enabled, you look see a Blob container named $logs in the storage account, and it should contain entries that match the time where you updated blobs.

davidebbo on 20 Sep 2016

👍1

I have 'Blob logs' checked and i had entries in $logs that matched the files that were placed

bullsfan127 on 21 Sep 2016

@bullsfan127 Can you share your function app name (that's the app name, not function name!), either directly or indirectly? This will help us investigate. Thanks!

davidebbo on 21 Sep 2016

Blob logging is enabled. The entry in $logs matches the time when the blob was inserted:
container: $logs\blob\2016\09\14\1200\000000.log - 14 sept - 12:56:33 GMT

In the storage account tied to the Azure Function I find the matching blob receipt:
container: azure-webjobs-hosts\blobreceipts...\myblobfile. Datetime: 15 sept - 20:11:05 GMT

Note that the storage account that is the trigger-source for the Azure Function is a different storage account than the one that the Azure Function uses internally (named azurefunctions6fde45d2).

JohanKroese on 21 Sep 2016

CC @mathewc Who was looking into it

lindydonna on 27 Sep 2016

Update on this: @jocawtho has identified the bug causing this. We have a fix, but it will take us a while to roll it out, since it is in core WebApps infrastructure, not the Functions runtime (i.e. not in this repo).

mathewc on 30 Sep 2016

Any update on this, getting this in a customer deployment right now, although seems a bit schizophrenic as sometimes it is okay and other times I have to go in to kick it off?

martinabbott on 20 Oct 2016

The fix is currently being rolled out into production. The upgrade should be completely finished by the end of next week.

jocawtho on 20 Oct 2016

👍1

@jocawtho Can you update this issue when the fix has been deployed?

lindydonna on 2 Nov 2016

The fix to this issue has now been deployed fully to production.

jocawtho on 2 Nov 2016

Sorry to bump this, but I am still experiencing this issue. Is the fix rolled out globally? My function is deployed in the west Europe region. BTW I don't have any dynamic plan available, instead I can choose between a consumption plan and an app service plan (mine is consumption). I suppose consumption and dynamic are the same..?

ThorvaldBoe on 1 Dec 2016

👍1

Reactivating. @jocawtho can you investigate?

@ThorvaldBoe can you share your function app name (that's the app name, not function name!), either directly or indirectly?

davidebbo on 1 Dec 2016

Hi All,

I remember this problem 6 weeks ago and was told it was being rolled out within the next week. Believe that is the comment on the issue...

So yeah, be good to know a timeline.

Did speak to Chris at the Summit and he did mention a workaround of using a timer trigger in the same Function App that fired every 3 minutes or so (from memory) but not tried it as we went a different way.

M

On 2 Dec. 2016, at 00:53, David Ebbo notifications@github.com wrote:

Reactivating. @jocawtho can you investigate?

@ThorvaldBoe can you share your function app name (that's the app name, not function name!), either directly or indirectly?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

martinabbott on 1 Dec 2016

The function app name is 'autobudget'

thanks!

ThorvaldBoe on 1 Dec 2016

@ThorvaldBoe can you also specify how you deployed your Functions into the app? e.g. was it using the Portal UI, git, WebDeploy, ...?

davidebbo on 1 Dec 2016

I used the portal and typed the code directly.

ThorvaldBoe on 1 Dec 2016

Can you try one thing: go to the Portal Function UI and click the 'refresh' button in the lower left corner. Let's see if that helps.

Please let us know once you have done that so we can check our logs. Thanks!

davidebbo on 1 Dec 2016

Done.

ThorvaldBoe on 1 Dec 2016

I do see a record of the update at 2016-12-01 19:03:27Z. My suspicion is that it will start working now. But what's not clear is why it was not already updated when you created your functions in the portal.

Anyway, please check whether things in fact work now, and we'll go from there.

davidebbo on 1 Dec 2016

Hi,
thanks for your effort.
Unfortunately, this is still not working.

I wanted to test a bit to be sure before I came to a conclusion. There were uncertainty related to a few aspects of my function, for example related to an async method I used to send email. I thought I might potentially have overlooked something related to how async behaves in azure functions, and made a programming mistake. So in order to be completely sure, I decided to write a new function as simple as possible, with no async, just to see if I could reproduce the error.

The function (called "BlobTrigger") takes a blob as a trigger, and writes a new entry to an azure table containing the date/time of the operation. The source code is given below so that anyone can test it out.

When testing, I found that while having the function app blade open in the portal, everything behaves nice. When I close it, it works for a minute or two, then the function seems to fall asleep, until I open the function app blade in the portal again.

This should be quite straightforward to test. I am surprised nobody else have reported this. Am I the only one using Azure functions with blob trigger, or am I doing something wrong...?

#r "Microsoft.WindowsAzure.Storage"

using System;
using Microsoft.WindowsAzure.Storage.Table;

public static void Run(Stream myBlob, TraceWriter log, ICollector<BlobUpload> outputTable)
{
    log.Info($"C# blob trigger function processed: {myBlob}");
    BlobUpload bu = new BlobUpload();
    Guid id = Guid.NewGuid();
    bu.PartitionKey = id.ToString();
    bu.RowKey = id.ToString();
    bu.UploadDate = DateTime.Now;
    outputTable.Add(bu);
}

public class BlobUpload : TableEntity {
    public DateTime UploadDate { get; set; } 
}

ThorvaldBoe on 2 Dec 2016

😕1 👍1

Hi,
I have a Event Hub Trigger function app with consumption plan. It has same problem. It only works when I open it in portal UI.

wuliang1982 on 2 Dec 2016

👍3

Yeah keeping the blade open is how I solved it. Still works using that when the steaming service goes to sleep.

On 3 Dec. 2016, at 05:57, ThorvaldBoe notifications@github.com wrote:

Hi,
thanks for your effort.
Unfortunately, this is still not working.

I wanted to test a bit to be sure before I came to a conclusion. There were uncertainty related to a few aspects of my function, for example related to an async method I used to send email. I thought I might potentially have overlooked something related to how async behaves in azure functions, and made a programming mistake. So in order to be completely sure, I decided to write a new function as simple as possible, with no async, just to see if I could reproduce the error.

The function (called "BlobTrigger") takes a blob as a trigger, and writes a new entry to an azure table containing the date/time of the operation. The source code is given below so that anyone can test it out.

When testing, I found that while having the function app blade open in the portal, everything behaves nice. When I close it, it works for a minute or two, then the function seems to fall asleep, until I open the function app blade in the portal again.

This should be quite straightforward to test. I am surprised nobody else have reported this. Am I the only one using Azure functions with blob trigger, or am I doing something wrong...?

r "Microsoft.WindowsAzure.Storage"

using System;
using Microsoft.WindowsAzure.Storage.Table;

public static void Run(Stream myBlob, TraceWriter log, ICollector outputTable)
{
log.Info($"C# blob trigger function processed: {myBlob}");
BlobUpload bu = new BlobUpload();
Guid id = Guid.NewGuid();
bu.PartitionKey = id.ToString();
bu.RowKey = id.ToString();
bu.UploadDate = DateTime.Now;
outputTable.Add(bu);
}

public class BlobUpload : TableEntity {
public DateTime UploadDate { get; set; }
}
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

martinabbott on 2 Dec 2016

👍1

@ThorvaldBoe Just want to report that I'm having the exact same behaviour as you. Blobtrigger only works when you've got the blade open.

I guess I'll work around it by inserting the object into a blob, and then adding the id to a regular storage queue.

EmilBechMadsen on 6 Dec 2016

👍1

Think it is safe to conclude that this is a bug. Hopefully MS will fix it soon. Your move, Microsoft! :)

ThorvaldBoe on 6 Dec 2016

@ThorvaldBoe I took a look at your app specifically and I can confirm that the issue is on our side and is not specific to your function code. Sorry about the issue - we'll keep investigating and try to figure out what's going on.

In the meantime, I'm attempting a mitigation by rebooting one of our infrastructure nodes to see if that gets your function to start triggering properly when the portal is not running. Let me know if you continue to see this behavior.

cgillum on 6 Dec 2016

👍1

No worries, thanks for looking into it.
I guess a few glitches are unavoidable in the beginning, but as it matures, Azure functions are really going to rock! :)

ThorvaldBoe on 6 Dec 2016

For what it's worth, we actually experienced the same issue when triggering on a azure queue instead. Maybe this is an EU West issue or something? This effectively renders function triggers pretty useless.

EmilBechMadsen on 7 Dec 2016

@EmilBechMadsen can you share your function app name (that's the app name, not function name!), either directly or indirectly? I'd like to confirm whether it's the same issue and, if so, your additional data may help us narrow this down more quickly. Thanks!

cgillum on 7 Dec 2016

@cgillum Hi Chris. Yep. It's "lasso-services". We've switched it to a queuetrigger, and then disabled it when it wasn't working in favour of a traditional webjob. The specific function is the export function.

EmilBechMadsen on 7 Dec 2016

@EmilBechMadsen Thanks for the quick response. I see you have two queue-triggered functions. The first one is _NewsReloaderFunction_ with a connection of _lassofunctions_STORAGE_, and our telemetry shows that this queue doesn't exist in your storage account. Perhaps this function was misconfigured? It's still enabled too.

The second function, _TargetGroupExportFunction_, has a similar problem though is a little more nuanced. In this case, our system is looking for a queue named _exportsqueue_ in the default storage account (_AzureWebJobsDashboard_). However, I also noticed that your function.json actually points to a different storage account, _LassoStorageNew_, which _does_ contain this queue. I'm guessing what happened is that you initially configured your queue trigger with the default storage account, but then switched to the new storage account. What can happen is if you make changes to function.json in a way that is not supported (e.g. editing the file directly in Kudu, etc) then our infrastructure won't notice the change. The way to fix this is to either publish your function using Git or WebDeploy, or you can navigate to the portal and either click the Refresh button for your bindings or you can make a simple no-op change to update the trigger registration. I suspect this function will work correctly if you simply re-enable it using the portal (that should cause a trigger registration update).

So just to clarify for anyone else who is following this thread, I think these are two different issues. ThorvaldBoe's issue is not yet understood and we're still investigating.

cgillum on 7 Dec 2016

👍2

@cgillum Aha. Well that would explain things. Thanks for the pointers. I changed the storage account this way because the interface does not allow you to configure it to use classic storage accounts.

Just in case you missed this point: It did actually work and triggered on the classic queue when I had the blade open, but I guess the automatic triggers outside the editor works differently.

EmilBechMadsen on 7 Dec 2016

@EmilBechMadsen Yeah, that's expected. Unfortunately there is a synchronization that needs to happen between parts of our system to ensure things continue working when the portal is closed. That synchronization is what doesn't happen when manual edits are made. Sorry about the confusion.

cgillum on 8 Dec 2016

I believe I've identified the issue with blob triggers and am working on a fix. I'm going to try and squeeze this into the next production deployment, which is starting very soon but may take about a week to complete. I'll circle back once the fix has made it into all of our production data centers. Thanks for your patience on this and for reporting the issue!

cgillum on 9 Dec 2016

👍2

Hi, I am still experiencing the exact same issue, except with the 'ServiceBusTopicTrigger-CSharp'. Is there a fix for this yet?

KeishaW on 13 Dec 2016

👍1

No fix yet I don't think but I did find a workaround...

If you create a TimerTrigger in the same Function App, and set it to run every 9 minutes then the app keeps running it seems.

Okay you get execution costs associated with it, but I literally just took the code as it is created in a TimerTrigger and it seems to work okay.

On 13 Dec. 2016, at 22:25, KeishaW notifications@github.com wrote:

Hi, I am still experiencing the exact same issue, except with the 'ServiceBusTopicTrigger-CSharp'. Is there a fix for this yet?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

martinabbott on 13 Dec 2016

@KeishaW This issue is specifically tracking blob triggers. Service Bus triggers won't be affected. One thing that folks commonly run into with Service Bus triggers is that their connection strings do not contain the Manage permission, which is currently required. In any case, I suggest you open a new item for that.

cgillum on 13 Dec 2016

👍1

@cgillum We've been discussing the ServiceBus AccessRights issue in an internal email thread, but I agree we should have a work item tracking this. I've opened https://github.com/Azure/azure-webjobs-sdk-script/issues/1048.

mathewc on 14 Dec 2016

👍1

Hi Folks
I am also facing the same issue for last 2 weeks. I have a Event Hub Trigger function app with consumption plan. It was working, when I open the portal in UI & continue to run for 3-4 hours , after that it automatically sleeps, when no request coming to event hub.
Next day when request come again , it is not triggering, until I open portal.

I have to swap forcefully from Consumption Plan to App Service Plan to run this.

When we can expect fix for Consumption Plan????

Thanks.

navilwayne on 3 Jan 2017

@navilwayne This issue only tracks functions triggered by blobs (see @cgillum's previous comment). If you are seeing something with other triggers, it is likely unrelated, and should be tracked separately.

davidebbo on 3 Jan 2017

@davidebbo - I have post the new request - Azure Function Not Triggering by Event Hub under Azure/Azure-Functions.

Link - https://github.com/Azure/Azure-Functions/issues/122.

Thanks

navilwayne on 3 Jan 2017

@navilwayne Thanks. Not that https://github.com/Azure/azure-webjobs-sdk-script/ is actually the correct repo for it.

davidebbo on 3 Jan 2017

Any update on this, my customer is still having to keep a browser window open just to fire the function, or go in regularly to process files?

martinabbott on 7 Jan 2017

@martinabbott just to make sure there is no misunderstanding (since there has been some in this issue), can you confirm that you are referring to a blob trigger and not something else? My understanding is that @cgillum got the fix deployed, but I'll let him confirm.

davidebbo on 7 Jan 2017

Hi David,

Yes blob trigger is the only issue I have.

Some more on this, I hadn't trigger my Function for some time but had deployed a Timer Tigger Function which had solved the issue previously.

When checking again this week, even opening the blade didn't work and the blob trigger would not fire at all, even though the Timer Trigger was running.

I know this was a problem because adding a new Blob Trigger worked so it's like my original Blob Trigger had become unregistered.

Cheers,

Martin

On 8 Jan 2017, at 02:40, David Ebbo notifications@github.com wrote:

@martinabbott just to make sure there is no misunderstanding (since there has been some in this issue), can you confirm that you are referring to a blob trigger and not something else? My understanding is the @cgillum got the fix deployed, but I'll let him confirm.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

martinabbott on 8 Jan 2017

If it doesn't run even going to portal, then it must be a different category of issues than the one Chris fixes (which was only about waking up the Function App). Does the blob function run correctly when you manually trigger it from the portal?

Can you share your function app name (that's the app name, not function name!), either directly or indirectly? As well as the blob function name within it.

davidebbo on 8 Jan 2017

As we found in the other issue, the problem for @martinabbott was that the function was disabled in the function.json.

Closing this issue now as things should be working for everyone.

If someone still sees the case of a blob triggered function that 'falls asleep' when portal is closed, please do let us know.

davidebbo on 9 Jan 2017

@davidebbo the issue raise in 1103 is different to this one (function becoming disabled after a long period of inactivity)

I can confirm that I am still able to see the particular issue described here in 691 in my Function App.

My function writes data to a database. If I close the browser and wait a period of time, I just waited 90 minutes, then upload a file, my database is not updated.

As soon as I open a browser window to the Function App blade my database gets updated.

So I don't believe this problem has been resolved.

Same function and Function App details as discussed previously.

martinabbott on 9 Jan 2017

I've encountered the same issue.
I was about to deploy an azure blob-triggered function to a client, but can't deploy it as long as this bug is open.

I've also created a small test case to be sure it's happening as described here:

Create a 1-minute-time-trigger function which writes it's operation logs to a storage account.
Create a blob-trigger function which monitors this storage account.
As long as the Azure portal was open - I could see invocations of the blob-trigger function every minute in "Manage" tab in the function app.
I closed all azure portal browser tab, waited for 15 minutes.
When I reopened the "Manage" tab, there were no invocations of the blob-trigger function, although there were invocations of the time-trigger-

Function-App names:
Time trigger: blob-trigger-upload-test (ignore the blob trigger function in this app)
Blob trigger: blob-trigger-test

Is there a timeline for fixing this issue ?

liricooli on 10 Jan 2017

@martinabbott your function app is definitely getting woken up, so the fix discussed above is working. The question is why it's not processing your blob function. Can you share the UTC time of one specific incident where you add a new blob and you don't see it getting processed?

davidebbo on 10 Jan 2017

@liricooli thanks for sharing your app names. Looking at blob-trigger-test, I do see many invocations of the BlobTriggerJS1 function, but I don't know whether it was when you had portal open or not. Can you also share the UTC time of an instance where you didn't have the portal open and the function failed to be called?

davidebbo on 10 Jan 2017

@davidebbo thanks for the quick reply.
I closed the portal yesterday, January 10th, at 12:15 PM UTC time and reopened it at about 12:40 PM.

liricooli on 11 Jan 2017

@liricooli but at which point did you add a new blob that needed to get processed? Note that after the app gets deactivated, it can take a longer for it to get restarted, and it could be that it would have worked on its own with a little more time.

Looking through our logs. I see that the scale controller detected some blob updates at 12:41:55, and woke up the site at that point. But note that when it detects it may be several minutes after it happens.

So the sequence was:

12:17: scale controller detected blob changes
Function app runs and processes everything
12:23: Function app idles out due to lack of activity
Around 12:33, you added a new blob (not sure exact time)
12:35: You went to portal, which woke it up
12:41: Scale controller finally detected the change and wakes up the app, which at that point has already done the work. But if you had not gone to portal, it would have done it on its own.

Bottom line:

I believe it would have worked even if you had not gone to portal. Please do verify :)
We're aware that the scale controller detection is too slow, and are looking at possible improvements going forward

davidebbo on 12 Jan 2017

Hi @davidebbo,
I'm quite sure that I can recreate this issue at-will and would be happy to do that while you're looking at the logs in real time, if needed.

The sequence of events you described is not quite what happened as far as I see it.

It's not that I added a new blob at ~12:23, but I was automatically and continuously adding new blobs every 1 minute, with a time-trigger function.
When the portal was open the blob trigger was triggered every minute, but once I closed the portal, the blob-trigger did not trigger even once.
The next time the blob-trigger triggered was only when I reopened the portal.

Here's the timeline as I see it:
~10:00 - 12:17: portal open, the time-trigger was running every 1 minute, the blob-trigger was running as expected, triggering every minute as a new file hits the blob storage.

~12:18: I closed the portal. the time trigger was still running every 1 minute from 12:18, and writing files that the blob-trigger is expected to monitor.

12:18 - 12:40: I reopened the portal. I verified that the time-trigger was indeed running during the whole time, and that the blob trigger did not trigger even once during that period, although there were new files in the monitored container.

12:40-~130:00 I left the portal open for some time and verified that the blob-trigger starts again.
This is why you see it started running in 12:41 - it was due to my manual opening of the portal.

as far as I grasp it without understanding anything about the system / architecture, it does not seem like and issue with slowness of a component like you described in the bottom line, but rather something that should be running but isn't.

liricooli on 15 Jan 2017

Yeah I'd echo that, I'm able to create at will if I don't have a Timer Trigger firing every 15 minutes that seems to keep it alive.

Currently at an event so I can't test but it's really easy to test anyway.

On 15 Jan 2017, at 18:52, liricooli notifications@github.com wrote:

Hi @davidebbo,
I'm quite sure that I can recreate this issue at-will and would be happy to do that while you're looking at the logs in real time, if needed.

The sequence of events you described is not quite what happened as far as I see it.

It's not that I added a new blob at ~12:23, but I was automatically and continuously adding new blobs every 1 minute, with a time-trigger function.
When the portal was open the blob trigger was triggered every minute, but once I closed the portal, the blob-trigger did not trigger even once.
The next time the blob-trigger triggered was only when I reopened the portal.

Here's the timeline as I see it:
~10:00 - 12:17: portal open, the time-trigger was running every 1 minute, the blob-trigger was running as expected, triggering every minute as a new file hits the blob storage.

~12:18: I closed the portal. the time trigger was still running every 1 minute from 12:18, and writing files that the blob-trigger is expected to monitor.

12:18 - 12:40: I reopened the portal. I verified that the time-trigger was indeed running during the whole time, and that the blob trigger did not trigger even once during that period, although there were new files in the monitored container.

12:40-~130:00 I left the portal open for some time and verified that the blob-trigger starts again.
This is why you see it started running in 12:41 - it was due to my manual opening of the portal.

as far as I grasp it without understanding anything about the system / architecture, it does not seem like and issue with slowness of a component like you described in the bottom line, but rather something that should be running but isn't.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

martinabbott on 15 Jan 2017

I set up a similar repro with one function in one app adding blobs on a timer, and the other function (different app) consuming them. For me, it is working very consistently with the portal closed. At most, the processing happens a few minutes late, and it is well understood why.

@liricooli in your case, I can see from our logs that you went to the portal at 12:35, not 12:41. 12:41 is when the scale controller detected the change and ping the app (which by that point was already up).

If you want to test this, please just keep it running without going to the portal, and observed whether things really are stuck, or end up getting processed a little late. Please let me know when the app is setup (and the two app names, if different from above). @martinabbott, same thing if you get a repro going.

davidebbo on 16 Jan 2017

@davidebbo I closed the portal now and would not open it in ~24 hours.
Will report back with another comment when I open it again.

liricooli on 16 Jan 2017

@davidebbo I reopened the portal now.
In the "Monitor" pane of my function I see that the last invocation was "a day ago", for this blob: functionlogs/blob-trigger-upload-test/2017/01/16/06/d6beed-6564.applicationLog.csv

I verified that there are newer blobs from the 17th.

Also, after refreshing the "Monitor" pane I started seeing invocations from blobs from today, the 17th.

liricooli on 17 Jan 2017

@liricooli so I'm looking at your app and trying to make sense of your functions.

In the blob-trigger-upload-test app, you have a TimerTriggerJS1 that is getting triggered every minute. But that function doesn't appear to do anything. i.e. it doesn't have any output bindings that would create a blob. Isn't that the Function that should be created a blob every minute? If not, who is doing that?

davidebbo on 17 Jan 2017

@davidebbo Sorry for not replying earlier - I was getting married :)
You are correct that TimeTriggerJS1 in blob-trigger-upload-test app is not doing anything.
However, this function is configured to write diagnostic logs to a container named - functionlogs in testfunctionlog storage account

This is the container our blob trigger function BlobTriggerJs1 in blob-trigger-test App service is monitoring.

liricooli on 23 Jan 2017

@liricooli First congrats!! 🍾 🎉

So I looked into your scenario in more details, and I now understand exactly what's going on. The problem is that the diagnostic logs settings you're using don't really work with Function apps. I just opened https://github.com/projectkudu/AzureFunctionsPortal/issues/885 to track improving this.

So effectively, the issue in your case is not that the function listening to the blobs (in target app) is not waking up, but is that no blobs are getting produced at all unless you have the portal opened on the source app. You can see that by looking at the blob container in Storage Exporer, but with and without the portal opened.

Bottom line: to really test this, you need to have your source function produce some blobs by using a blob output binding. And in my own test, this works very well even after closing the portal.

davidebbo on 23 Jan 2017

The fix for the blob trigger issue has been fully deployed to production (it's been there for a while now) and I haven't seen any new reports of problems (other than slowness or user config issues, which we can track elsewhere) so I think we can close this issue.

cgillum on 30 Jan 2017

So I’ve now tried this and it appears as though it is working.

I uploaded a file, the function simply writes out details including the current date and time to a storage Table.

The function did process the file around 6 minutes after I uploaded.

martinabbott on 31 Jan 2017

I think we're experiencing a similar issue in which files are not being picked up by a blob trigger function.

Today we added 5 files to a container which weren't processed for just over 30 mins. Someone then loaded the function blade and they were picked up.

Is there anything we can do to diagnose this? From following this thread - it looks as if logs were piped into the $logs container shortly after the files were added. But in the function storage account blob receipts and host logs were only added after someone loaded the function app up via the portal.

Nosmadas on 3 Feb 2017

@Nosmadas Can you share your function app name (that's the app name, not function name!), either directly or indirectly, as well as the UTC time of the incident? As far as we know, all cases are fixed now, so there are no know pending issues.

davidebbo on 3 Feb 2017

I am experiencing something similar with my bandata-complaint-photos-resize app. Logs was created in the log folder after uploading photos to blob, but no function running was occured. But then, after opening portal, photos were resized.

I went through this thread, by I haven't found anything that could help me run my function automatically. I would be grateful if you could help me in this matter.

I have Dynamic App Service Plan and Check that Status is On and 'Blob logs' is checked.

rmaziarka on 4 Feb 2017

@rmaziarka as per above, to investigate, we need the approximate UTC time where you uploaded a blob and didn't see the function run. Also, please include the name of the function. Thanks!

davidebbo on 4 Feb 2017

Taking a look, I can see that your app is getting woken up on Feb 3 at 21:14, 21:20, 22:35, 22:44, 23:43. So it is correctly detecting new blobs and doing the right thing from what I can see. Do keep in mind that when the app is cold, blob detection can be delayed for several minutes.

davidebbo on 4 Feb 2017

It was woken on Feb 3 because I opened portal. Photos were uploaded day before.

Today I uploaded 5 photos to my blob at 08:43 UTC (2017-02-04). Nothing happened. Then hour later I opened portal and automatically photos were resized (function has started), at 10:09 UTC (2017-02-04).

My app is bandata-complaint-photos-resize, function is complaint-photos-resize.

rmaziarka on 4 Feb 2017

@davidebbo The function app name is ... 5 files were added to the blob container used in the blob trigger at approx 10:10 UTC (2017-02-03), I loaded the function blade at around 10:43 and they were then picked up.

Thanks

Nosmadas on 4 Feb 2017

@rmaziarka so looking at your function, you are processing blobs in place, i.e. replacing the original image by the resized image.

Looking at our logs, it's a bit puzzling as I see the scale controller is detecting changes to the blob at 10:16, but not at 8:43. i.e. it appears to detect the change that is caused by the execution of your function, but not the change caused by you adding the new files in the first place.

Couple questions:

How exactly are you adding the new blobs? i.e. via code, some tool, ...?
I wonder if the fact that both your trigger and your output are the same blob is triggering some issue. If you don't mind, would you be able to try changing the output binding to go to a different container, to see if that makes a difference? Thanks!

davidebbo on 4 Feb 2017

👍1

@Nosmadas your case is different, as the scale controller does not have any information about your triggers. There was an issue that was fixed yesterday that could cause that to happen when deploying via VS (or msdeploy). Can you try clicking the 'Refresh' button in the Functions UI (in the lower left, below Quickstart). Thanks!

davidebbo on 4 Feb 2017

@davidebbo Done! Thanks for looking into it, will monitor going forwards.

Nosmadas on 4 Feb 2017

@Nosmadas yep, I see the registration for both functions. Normally, processing will now happen with having to go to portal.

davidebbo on 4 Feb 2017

👍1

@davidebbo
I created new function complaint-photos-resize-2 (app name bandata-complaint-photos-resize) which adds _temp_ at the end of the filename. And now it's working. Both from code (ASP.NET app) and Storage Exporer. I would not think about it :)

Would you be able to fix it somehow or should i need to create 2 functions:

first to resize photo and save with temp name
second to rename output file to previous name
?

rmaziarka on 6 Feb 2017

@rmaziarka I set up a similar scenario where the function updates the new blob inplace, and it's actually working fine for me.

My guess is that there is some alternate issue at play. Thinking about it, it's hard to see how the fact that you update inplace could break things, because the fact that you do an implace update doesn't come into play until your function gets called, and if it gets called, we don't have the issue in the first place. So the inplace part is very likely a red herring here.

davidebbo on 7 Feb 2017

Hi, Guys.

I am experiencing exactly the same issue but with function app triggered by the Queue. I am also on Consumption plan. App Name: cm-image-ops, Function Name: image-merge. Should I open a new issue or the same fix as for blob trigger can be applied in my case?

Thank you.

edspivak on 27 Feb 2017

@edspivak Please do open a new issue, as this one is specific to blob triggers. Thanks!

davidebbo on 27 Feb 2017

@davidebbo
I'm seeing this issue.
At around 4:00pm UTC we uploaded a blob. It's now 4:19 UTC and it still hasn't processed. At 4:20 UTC I will open the app. The app name is cjep-Blob-Efiling-partners.

Opened the app and it immediately processed.

ASeale on 6 Dec 2017

@ASeale it looks like there may be an issue with your connection string. I see an error on our backend: "The storage account connection string was null, empty, or not found." Can you check that the connection name is valid?

We're working on a way to better show these messages to the user, and this should be coming in the next month or two. /cc @tohling

davidebbo on 6 Dec 2017

@davidebbo ok so, the connection string names were accurate. As a test I moved them from the "Connection strings" section in the Application Settings to the "Application settings" section in the Application Settings. Everything seems to be working now. I will run some more delayed tests just to make sure.

ASeale on 6 Dec 2017

@davidebbo verified working. I waited 30 minutes before trying another test. It took it 8 minutes to finally process it, but it did end up getting it. Not sure why moving the connection strings from the connection string section to the application setting section worked, but it did.

ASeale on 6 Dec 2017

@ASeale I would say it's a bug that it didn't work with Connection Strings. But for now, yes, please do use App Settings (there is no drawback).

davidebbo on 6 Dec 2017

Facing the same issue now

kkanna on 3 May 2018

@kkanna please share your function app name (that's the app name, not function name!), either directly or indirectly, as well as a sample UTC time where it didn't fire and you think it should have. Also, indicate how long you waited.

davidebbo on 3 May 2018

Hello, I'm facing the same issue too
The connection string is in the app settings
'Blob logs' is checked
There are entries in $logs that matched the files
I have two function listening on the same container, but none of the two is triggering anymore.
The functions are in the same service plan where dummy6428 is
I been waiting for an hour now but nothing happened
I tried to run function manually with "Test" and the function worked
Can you @davidebbo help?
Thank you

agostinocinelli on 18 May 2018

@agostinocinelli I suspect there is some memory corruption that happened when the app last started around May 10th that put it in an unstable state.

Specifically, I see two kind of exceptions on the 10th and 11th:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.CreateVideoThumbCS ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.
   at System.IO.MemoryStream.set_Capacity(Int32 value)
   at System.IO.MemoryStream.EnsureCapacity(Int32 value)
   at System.IO.MemoryStream.Write(Byte[] buffer,Int32 offset,Int32 count)
   at System.IO.Stream.InternalCopyTo(Stream destination,Int32 bufferSize)
   at System.IO.Stream.CopyTo(Stream destination)
   at Submission#0.Run(Stream myBlob,Stream videoThumb,String blobname,String blobextension,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 107 
   End of inner exception
   at System.RuntimeMethodHandle.InvokeMethod(Object target,Object[] arguments,Signature sig,Boolean constructor)

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.CreateVideoThumbCS ---> System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation. ---> System.ComponentModel.Win32Exception : Only part of a ReadProcessMemory or WriteProcessMemory request was completed
   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Submission#0.CreateThumbnail(String temp,String tempOut,Int32 duration,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 247
   at Submission#0.GetThumbnail(String temp,String tempOut,Int32 duration,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 209
   at Submission#0.Run(Stream myBlob,Stream videoThumb,String blobname,String blobextension,TraceWriter log) at D:\home\site\wwwroot\CreateVideoThumbCS\run.csx : 126 
   End of inner exception
   at System.RuntimeMethodHandle.InvokeMethod(Object target,Object[] arguments,Signature sig,Boolean constructor)

Restarting the app would put it back into a clean state, but it may happen again is there is some issue with what the function is doing. Also, you have 34 sites running on a Small Basic VM, which may contribute to the memory issues. Try scaling up to a medium, or use multiple App Service plans.

davidebbo on 18 May 2018

Thank you @davidebbo for your help!
Most of the 34 sites are no longer or very rarely used, but we definitely need to organize them better :)
I will periodically check the log before moving the functions to production!
Thank you again

agostinocinelli on 21 May 2018

I'm encountering this same issue. If I load up the portal it seems to work, I can wait 2-3 mintues, upload again and it seems to work. If I wait around 10 minutes nothing triggers.

and I get the following :

2018-08-24T14:09:14 No new trace in the past 8 min(s).
2018-08-24T14:10:14 No new trace in the past 9 min(s).
2018-08-24T14:11:14 No new trace in the past 10 min(s).
2018-08-24T14:12:14 No new trace in the past 11 min(s).

mesani on 24 Aug 2018

Assume this is on a consumption plan?

Is it v1?

Just as a test, as this worked for me, try setting a timer trigger to fire every 5 minutes and see if that solves it.

Azure-functions-host: Function not triggered by blob

Most helpful comment

All 103 comments

r "Microsoft.WindowsAzure.Storage"

Related issues