Azure-webjobs-sdk: Function host fails for Blob Trigger watching a ADLSv2 account

[olegkapustin]

Repro steps

1. Create an ADLSv2 account
2. Create a Function App (Python 3.7, blob trigger) (vs code template func is sufficient):
3. Configure bindings to use the ALDSv2 account

Expected behavior
The blobs should be processed.

Actual behavior
Sometimes function host fails with the stack trace like this (may happen both locally and in Azure):

assembly,level,line,method
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",0,0,"System.ThrowHelper.ThrowKeyNotFoundException"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",1,0,"System.Collections.Generic.Dictionary`2.get_Item"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",2,183,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener+<GetLogsWithPrefixAsync>d__16.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",3,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",4,154,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener+<ListRecentLogFilesAsync>d__15.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",5,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",6,0,"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",7,55,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener+<GetRecentBlobWritesAsync>d__12.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",8,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",9,110,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.PollLogsStrategy+<ExecuteAsync>d__12.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",10,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",11,0,"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",12,126,"Microsoft.Azure.WebJobs.Host.Blobs.Listeners.ScanBlobScanLogHybridPollingStrategy+<ExecuteAsync>d__13.MoveNext"
"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",13,0,"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"
"Microsoft.Azure.WebJobs.Extensions.Storage, Version=3.0.10.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",14,147,"Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer+<RunAsync>d__14.MoveNext"

Known workarounds
recreate storage account

Related information
Related issue - https://github.com/microsoft/vscode-azurefunctions/issues/1725

Root cause: no check for the existence of the dictionary key. (I assume a presence of a metadata property is not guaranteed.)
https://github.com/Azure/azure-webjobs-sdk/blob/fca811638dd3c7dcf55905dea6837be5b30f5f15/src/Microsoft.Azure.WebJobs.Extensions.Storage/Blobs/Listeners/BlobLogListener.cs#L181

[venkatx5]

Any update on this?

[joalmeid]

+1

[joalmeid]

2404

[ekennedy6334]

Similar issue with C#:

System.Collections.Generic.KeyNotFoundException: The given key 'LogType' was not present in the dictionary.
at System.Collections.Generic.Dictionary2.get_Item(TKey key) at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener.GetLogsWithPrefixAsync(List1 selectedLogs, CloudBlobClient blobClient, String prefix, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.StorageBlobs\ListenersBlobLogListener.cs:line 175
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener.ListRecentLogFilesAsync(CloudBlobClient blobClient, DateTime startTimeForSearch, CancellationToken cancellationToken, Int32 hoursWindow) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.StorageBlobs\ListenersBlobLogListener.cs:line 150
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobLogListener.GetRecentBlobWritesAsync(CancellationToken cancellationToken, Int32 hoursWindow) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.StorageBlobs\ListenersBlobLogListener.cs:line 52
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.PollLogsStrategy.ExecuteAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.StorageBlobs\Listeners\PollLogsStrategy.cs:line 107
at Microsoft.Azure.WebJobs.Host.Blobs.Listeners.ScanBlobScanLogHybridPollingStrategy.ExecuteAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.StorageBlobs\Listeners\ScanBlobScanLogHybridPollingStrategy.cs:line 120
at Microsoft.Azure.WebJobs.Host.Timers.TaskSeriesTimer.RunAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Timers\TaskSeriesTimer.cs:line 147

[olegkapustin]

Looks like @brettsam has fixed this in dev:
https://github.com/Azure/azure-webjobs-sdk/blob/276f01d93e9ca42fa72a523337d73e7ae4400a57/src/Microsoft.Azure.WebJobs.Extensions.Storage/Blobs/Listeners/BlobLogListener.cs#L178

It would be nice to know when this will be in master...

[brettsam]

This will be a part of the Microsoft.Azure.WebJobs.Extensions.Storage 3.0.11 release, which will likely happen in the next couple of weeks.

However, it's already a part of the 4.0.0-preview1 release, which I've just pushed to Nuget this morning. That moves our storage extension up to newer Storage SDK nugets, so there are some breaking changes around that for C# functions (if directly using the Storage types), which you can see here: https://github.com/Azure/azure-webjobs-sdk/releases/tag/storage-v4.0.0-preview1

[CedricVandelaer]

Hi, is it possible to install this preview package or an updated version in python azure web app?
I can only find the option to specify the version number of the extension bundle in the host.json file, but the extension bundle doesn't to be updated yet either.

Thanks in advance

[lankaapura]

Hi @brettsam, When can we expect Microsoft.Azure.WebJobs.Extensions.Storage 3.0.11 in nuget?

I've created PR to Azure Functions Extention Bundle but it is failing since Microsoft.Azure.WebJobs.Extensions.Storage 3.0.11 is not out yet.
https://github.com/Azure/azure-functions-extension-bundles/pull/19

[evannielsen]

This is also a problem for a client of mine. Is there any way that we can get a timeline on when 3.0.11 will be released?

[brettsam]

I'm trying to work through this now -- we've had a series of deployment issues that have slowed a lot our progress down. But I'll see if i can get a release out soon.

[josiahvinson]

Published today!

[brettsam]

Yes, sorry for not updating this issue -- 3.0.11 was released a few days ago and the next bundles release will contain this also (if it's not already out).

[hiteshy]

Has the Microsoft.Azure.Functions.ExtensionBundle been updated. It seems that the VSCode still fails with the error as below for ADLSv2 Blob Trigger.
System.Private.CoreLib: The given key 'LogType' was not present in the dictionary

[brettsam]

@soninaren -- did the bundles release with 3.0.11 go out?

[hiteshy]

I think no, as the bundles latest release 1.1.1 in Git has version 3.0.10 for Microsoft.Azure.WebJobs.Extensions.Storage