Azure-functions-host: Unable to run Durable Functions with Azure Storage Emulator v5.8

Created on 15 Oct 2018  路  20Comments  路  Source: Azure/azure-functions-host

I was successfully running Durable Functions for a few weeks but now all of a sudden I am not able to run them and am getting the following output. Not sure what has changed -- any pointers on what to look at would be most appreciated. I have also included a related stacktrace below.

Azure Functions Core Tools (2.1.725 Commit hash: 68f448fe6a60e1cade88c2004bf6491
af7e5f1df)
Function Runtime Version: 2.0.12134.0
[10/14/2018 10:17:59 PM] Building host: startup suppressed:False, configuration
suppressed: False
[10/14/2018 10:17:59 PM] Reading host configuration file 'C:\Users\csheets\sourc
e\repos\FunctionApp4\FunctionApp1\bin\Debug\netstandard2.0\host.json'
[10/14/2018 10:17:59 PM] Host configuration file read:
[10/14/2018 10:17:59 PM] {
[10/14/2018 10:17:59 PM]   "version": "2.0"
[10/14/2018 10:17:59 PM] }
[10/14/2018 10:18:38 PM] A host error has occurred
[10/14/2018 10:18:38 PM] Microsoft.WindowsAzure.Storage: Server encountered an i
nternal error. Please try again after some time.
Listening on http://0.0.0.0:7071/
Hit CTRL-C to exit...

The following is a stacktrace that is being thrown:

System.AggregateException
  HResult=0x80131500
  Message=One or more errors occurred.
  Source=System.Private.CoreLib
  StackTrace:
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.<Dispose>b__8_0()
   at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility.RunWithoutSynchronizationContext(Action actionToRun)
   at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.Azure.WebJobs.Script.WebHost.BlobStorageSecretsRepository.<WriteToBlobAsync>d__22.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\BlobStorageSecretsRepository.cs:line 167
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.BlobStorageSecretsRepository.<WriteAsync>d__18.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\BlobStorageSecretsRepository.cs:line 128
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.<PersistSecretsAsync>d__26`1.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs:line 412
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.<GetHostSecretsAsync>d__11.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs:line 82
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.<GetOrCreateExtensionKey>d__6.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs:line 72
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetExtensionWebHookRoute(String extensionName) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs:line 64
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetUrl(IExtensionConfigProvider extension) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs:line 49
   at Microsoft.Azure.WebJobs.Host.Config.ExtensionConfigContext.GetWebhookHandler() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Config\ExtensionConfigContext.cs:line 55
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension.Microsoft.Azure.WebJobs.Host.Config.IExtensionConfigProvider.Initialize(ExtensionConfigContext context)
   at Microsoft.Azure.WebJobs.Host.DefaultExtensionRegistryFactory.Create() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\DefaultExtensionRegistryFactory.cs:line 38
   at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions.<>c.<AddWebJobs>b__1_0(IServiceProvider p) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs:line 50
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs:line 156
   at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 4530
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7719
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, Object serviceKey, IfUnresolved ifUnresolved, Type requiredServiceType, Request preResolveParent, Object[] args) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 290
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 210
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 195
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 96
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 79
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions.<>c.<AddWebJobs>b__1_4(IServiceProvider p) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs:line 83
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs:line 156
   at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 4530
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7719
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, Object serviceKey, IfUnresolved ifUnresolved, Type requiredServiceType, Request preResolveParent, Object[] args) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 290
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.ReflectionFactory.CreateExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7059
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6534
   at DryIoc.ReflectionFactory.CreateExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7059
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6534
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 210
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 195
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 96
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 79
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Azure.WebJobs.Script.ScriptHostBuilderExtensions.<>c.<AddScriptHostCore>b__2_5(IServiceProvider p) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\ScriptHostBuilderExtensions.cs:line 115
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs:line 156
   at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 4530
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7719
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 210
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 195
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 96
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 79
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Azure.Functions.Cli.Actions.HostActions.StartHostAction.<RunAsync>d__44.MoveNext()

Inner Exception 1:
StorageException: Server encountered an internal error. Please try again after some time.
picture csheets

Most helpful comment

Thanks - I downloaded 5.8 and observed the same problem. I have no idea why the Durable extension seems to be triggering this issue because the code path seems completely unrelated. I'll follow up with the Azure Functions Host team. I think this might possibly be a problem with extensions that use web hooks.

Workaround

Here is a workaround I identified from the Functions Key Management documentation. In local.settings.json, add a new setting called AzureWebJobsSecretStorageType and set it to "files".

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "AzureWebJobsSecretStorageType": "files"
  }
}

/cc @fabiocav

picture cgillum on 16 Oct 2018
👍30

All 20 comments

I switched from using local storage emulator to using Azure StorageAccount and the function app run ok -- so it is something to do with the Azure Storage emulator but I have no idea what. I have attempted to uninstall and re-install the storage emulator but the same error occurs after the uninstall/re-install process.

csheets picture csheets on 15 Oct 2018

Which version of the storage emulator are you using? This actually doesn't appear to be a Durable Functions issue, but rather an issue with the Azure Functions host itself, which uses storage for things like key management (key management frames appear in your callstack).

cgillum picture cgillum on 16 Oct 2018

I am using version: Windows Azure Storage Emulator 5.8.0.0 command line tool

I am able to run "regular" Azure Functions locally without any issues. Seems to be just Durable Functions that are having the problem.

csheets picture csheets on 16 Oct 2018

Thanks - I downloaded 5.8 and observed the same problem. I have no idea why the Durable extension seems to be triggering this issue because the code path seems completely unrelated. I'll follow up with the Azure Functions Host team. I think this might possibly be a problem with extensions that use web hooks.

Workaround

Here is a workaround I identified from the Functions Key Management documentation. In local.settings.json, add a new setting called AzureWebJobsSecretStorageType and set it to "files".

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "AzureWebJobsSecretStorageType": "files"
  }
}

/cc @fabiocav

cgillum picture cgillum on 16 Oct 2018
👍30

ok - thanks

prior to your reply i ended up uninstalling and re-installing VS 2017 and that resolved the issue -- looks like it re-installed Storage Emulator 5.6 in the process so just going to stay there for now

csheets picture csheets on 16 Oct 2018
👍1

Good to know there is a way to go back (I might consider doing that as well).

I've also opened a bug in the Azure-Functions-Host repo to track this since it will likely affect more than just Durable Functions: https://github.com/Azure/azure-functions-host/issues/3621

cgillum picture cgillum on 16 Oct 2018

Indeed, this is not specific to Durable Functions, but on a host code path it relies on.

This looks like an issue with the more recent version of the Emulator, we'll investigate.

fabiocav picture fabiocav on 16 Oct 2018

I also encountered the same error.
Also, the problem is temporarily resolved by the method @cgillum presented. Thanks.
https://github.com/Azure/azure-functions-durable-extension/issues/474#issuecomment-430337085

I expect that the error will be resolved.

tsubakimoto picture tsubakimoto on 18 Nov 2018

Moving this to host repo for triage and assignment

fabiocav picture fabiocav on 19 Nov 2018
🎉1

I experienced this issue after installing the Visual Studio 2019 Preview. Installing Storage Emulator 5.9 fixed it for me.

apsteger picture apsteger on 7 Jan 2019
👍1

I experienced the same issue. Thank you @cgillum for the workaround.

iwaldman picture iwaldman on 23 Jan 2019
👍1

I am also experiencing this issue. Using the workaround.

opyke picture opyke on 23 Jan 2019

UPDATE: said blog post

Alright. I'll be writing a blog post about this issue to make it more visible but basically, updating Storage Emulator to 5.9 solves your issue.

I was running Azure Durable Functions on .NET and NodeJS with the workaround for my samples:

I was running the v5.8 of Azure Storage Emulator. No idea what changed since the docs isn't updated yet. I've opened an issue to ensure that we get some notes.

So... updating to v5.9 fixed the issue in .NET as well as with NodeJS using the Windows Storage Emulator.

My Recommendation

Leave the workaround in place. People may have the 5.8 version which could lead to issues. There's no clear message that the parameter is missing which makes it hard to debug. Leaving the parameter in place doesn't seem to be causing any issues with 5.9 either.

Follow-Up

@fabiocav @cgillum Any point in making the message clearer for the users? Basically, 5.7 works, 5.9 works, 5.8 馃敟.

Obviously not an issue from Azure Functions point of view but it does look more like a breaking change in the Emulator's API.

Thoughts?

MaximRouiller picture MaximRouiller on 18 Mar 2019

Same thing is happening even if you have VS 2019 Installed and running Azure Storage Emulator 5.9. I had to add this entry to the local.settings.json to get the Azure Function to work.

chrislangston picture chrislangston on 30 Jun 2019
👍1

@chrislangston Which version of Azure Functions are you using?

If you run func on your CLI, can you copy the lines below the magical lightning? Mine looks like this:

Azure Functions Core Tools (2.7.1149 Commit hash: 476884e4a09c54686e460fd0c050d15139bf073e)
Function Runtime Version: 2.0.12438.0
MaximRouiller picture MaximRouiller on 8 Jul 2019

Adding this to triaged to verify whether there are issues when running with Storage Emulator 5.9.

yojagad picture yojagad on 16 Jul 2019

I'm still unable to run durable functions with Storage Emulator 5.10 and AzureWebJobsSecretStorageType set to "files" in the local.settings.json file.

If I remove the AzureWebJobsSecretStorageType setting (so it defaults to blobs) I get this exception in the Storage Account logs:

03/01/2020 09:53:22 [Error_debug] [ActivityId=919f1fa3-9cd5-441d-a07a-411fec2f6828] Exception raised during processing of request.
 Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.FatalServerCrashingException: 
The fatal unexpected exception 'Exception has been thrown by the target of an invocation.' encountered during processing of request. 
---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. 
---> System.InvalidOperationException: This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms.
   at System.Security.Cryptography.MD5CryptoServiceProvider..ctor()

This means I can't upload blobs to the storage emulator. I'm unable to disable FIPS compliant algorithms for encryption because i'm using a work machine. Looks like Durable Functions rely on blobs in the azure-webjobs-hosts and azure-webjobs-secrets containers. It's just a shame the AzureWebJobsSecretStorageType setting set to "files" doesn't overcome this issue. Wonder if this would be possible to fix in the future?

Of course I can workaround this by using a real storage account instead of the emulator..

r0bturner picture r0bturner on 3 Jan 2020

Looks like Durable Functions rely on blobs in the azure-webjobs-hosts and azure-webjobs-secrets containers. It's just a shame the AzureWebJobsSecretStorageType setting set to "files" doesn't overcome this issue. Wonder if this would be possible to fix in the future?

Of course I can workaround this by using a real storage account instead of the emulator..

Actually, I failed to successfully deploy my Durable Functions into Azure because of this same thing.
Manual removal of the blob's contents (the keys) solved the issue.

HQJaTu picture HQJaTu on 7 Feb 2020

Thanks - I downloaded 5.8 and observed the same problem. I have no idea why the Durable extension seems to be triggering this issue because the code path seems completely unrelated. I'll follow up with the Azure Functions Host team. I think this might possibly be a problem with extensions that use web hooks.

Workaround

Here is a workaround I identified from the Functions Key Management documentation. In local.settings.json, add a new setting called AzureWebJobsSecretStorageType and set it to "files".

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "AzureWebJobsSecretStorageType": "files"
  }
}

/cc @fabiocav

This solved issue for me. I had upgraded to VS 2019 and Emulator 5.1

vinayakmoh picture vinayakmoh on 16 Mar 2020

Yes, that will solve the problem for local dev. Deleting the Blobs in question will solve it when the same thing happens in Azure.

HQJaTu picture HQJaTu on 16 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

christopheranderson picture christopheranderson  路  4Comments
alaatm picture alaatm  路  4Comments
ElvenSpellmaker picture ElvenSpellmaker  路  3Comments
krillehbg picture krillehbg  路  3Comments
horihiro picture horihiro  路  3Comments