Azure-functions-host: DryIoc.ContainerException : Scope disposed{no name, Parent=disposed{no name, Parent=disposed{no name}}} is disposed and scoped instances are disposed and no longer available

[brettsam]

From https://stackoverflow.com/questions/58129379/azure-functions-scoped-instances-are-disposed-and-no-longer-available

I am receiving below error when function app is running on the run-time version "2.0.12701.0" (FUNCTIONS_EXTENSION_VERSION = ~2 in App Settings). As soon I change the FUNCTIONS_EXTENSION_VERSION to 2.0.12673.0 in App Settings. function starts working without any code change or deployment.

DryIoc.ContainerException : Scope disposed{no name, Parent=disposed{no name, Parent=disposed{no name}}} is disposed and scoped instances are disposed and no longer available. at DryIoc.Throw.It(Int32 error,Object arg0,Object arg1,Object arg2,Object arg3) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 8991 at DryIoc.Scope.TryGet(Object& item,Int32 id) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7881 at DryIoc.Container.InstanceFactory.GetAndUnwrapOrDefault(IScope scope,Int32 factoryId) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 1480 at DryIoc.Container.InstanceFactory.GetInstanceFromScopeChainOrSingletons(IResolverContext r) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 1466 at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType,IfUnresolved ifUnresolved) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 224 at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType,IfUnresolved ifUnresolved) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 195 at DryIoc.Resolver.Resolve[TService](IResolver resolver,IfUnresolved ifUnresolved) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 4742 at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.b__0(IResolverContext r) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs : 156 at DryIoc.Registrator.<>c__DisplayClass27_0.b__0(IResolverContext r) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 4541 at lambda_method(Closure ,IResolverContext ) at DryIoc.Factory.<>c__DisplayClass26_1.b__2() at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6596 at DryIoc.Scope.TryGetOrAdd(ImMap1 items,Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7840 at DryIoc.Scope.GetOrAdd(Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7825 at DryIoc.Factory.ApplyReuse(Expression serviceExpr,Request request) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6595 at DryIoc.Factory.GetExpressionOrDefault(Request request) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6555 at DryIoc.Factory.GetDelegateOrDefault(Request request) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 6625 at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7731 at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType,Object serviceKey,IfUnresolved ifUnresolved,Type requiredServiceType,Request preResolveParent,Object[] args) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 290 at lambda_method(Closure ) at DryIoc.Scope.TryGetOrAdd(ImMap1 items,Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7840 at DryIoc.Scope.GetOrAdd(Int32 id,CreateScopedValue createValue,Int32 disposalOrder) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 7825 at lambda_method(Closure ,IResolverContext ) at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType,IfUnresolved ifUnresolved) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 224 at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType,IfUnresolved ifUnresolved) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs : 195 at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedServiceProvider.GetService(Type serviceType) at C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\ScopedServiceProvider.cs : 25 at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.GetService(IServiceProvider sp,Type type,Type requiredBy,Boolean isDefaultParameterRequired) at lambda_method(Closure ,IServiceProvider ,Object[] ) at Microsoft.Azure.WebJobs.Host.Executors.DefaultJobActivator.CreateInstance[T](IServiceProvider serviceProvider) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\DefaultJobActivator.cs : 37 at Microsoft.Azure.WebJobs.Host.Executors.DefaultJobActivator.CreateInstance[T](IFunctionInstanceEx functionInstance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\DefaultJobActivator.cs : 32 at Microsoft.Azure.WebJobs.Host.Executors.ActivatorInstanceFactory1.<>c__DisplayClass1_1.<.ctor>b__0(IFunctionInstanceEx i) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\ActivatorInstanceFactory.cs : 20 at Microsoft.Azure.WebJobs.Host.Executors.ActivatorInstanceFactory1.Create(IFunctionInstanceEx functionInstance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\ActivatorInstanceFactory.cs : 26 at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.CreateInstance(IFunctionInstanceEx functionInstance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 44 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ParameterHelper.Initialize() at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 846 at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsyncCore(IFunctionInstanceEx functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 116

[brettsam]

Some initial questions in case anyone comes across this (we haven't been able to reproduce this yet):

1. What are the relationships between your scoped and singleton services? Do any Singletons hang onto the scoped services?
2. What are you doing with your scoped services in the function? Could you be storing them or accessing them after the function exits?

Thanks for the help.

[TechWatching]

I have the same issue as the one described in the stackoverflow post. My azure function stopped suddently to work, I guess due to the new Azure functions runtime release.

[rasmustherkelsen]

I am seeing this issue as well. For me it occurs in all my timer trigger functions. I cannot reproduce it locally as I seem unable to get the newest runtime SDK. Pinning the Azure Functions runtime to 2.0.12673.0 makes the code work again without any changes as reported by @TechWatching Fortunately only my dev environments has been updated so far. Can see prod still runs on 2.0.12673.0. As emergency action I need to pin the version in prod but in reality you could just have killed my prod environment without me being able to do anything. Please fix this asap

[brettsam]

/cc @fabiocav

@TechWatching / @rasmustherkelsen -- can either of you share with me the details of your DI setup? We don't yet have a repro for this error and we're trying to establish the pattern that causes it. We suspect the error is due to a fix in another DI area (specifically the fix for https://github.com/Azure/azure-functions-host/issues/3399, which was PR https://github.com/Azure/azure-functions-host/pull/4923).

If you're not able to share publicly, would you be able to email me (my microsoft email is in my profile) and share some of your code with us?

[vrossi]

@brettsam I was experiencing the same issue, but mine was with a Topic Trigger. The only dependency I was taking was an IHttpHandlerFactory. I'd be happy to send over some code if you'd like?

I was having another issued when I downgraded as well. One of my other Topic Trigger functions was taking a class parameter so the runtime would deserialize the message body into my class for me. We typically do this for all functions that are listening to topics. While this works for the others, this particular function was throwing a deserialization exception. I was forced to take a string parameter instead to get past this for the time being.

[brettsam]

@vrossi -- yes, if you're able to send me some code via email that would help immensely. Once we have a root cause we can figure out how we can get both issues resolved.

[JohanNorberg]

I also have this issue. It works locally but when deployed it to Azure I get this error.

[TechWatching]

@brettsam I just sent you an email with some code (object email is "GitHub issue 4989"). Tell me if you need there is other things you need to investigate.

[brettsam]

Thanks for the help @vrossi and @TechWatching! I've been able to reproduce this with a simple app that registers IHttpClientFactory with AddHttpClient() in my startup, and then uses that factory in my function. The key was that I had to invoke my function via the admin APIs (i.e. via the portal's "Run" button) _before_ the function ever executed "naturally". After that, it was broken forever. Conversely, if the function executed "naturally" first, it all ran okay, even if the admin API invoked it later.

Can anyone confirm if this is what you're seeing as well? It's possible there are other permutations of getting into this bad state, but this has worked 100% of the time for me.

The current workaround is to set the FUNCTIONS_EXTENSION_VERSION app setting to 2.0.12673.0 while we work on a fix.

[rasmustherkelsen]

Interesting. I will try to reproduce this behaviour tomorrow as it is getting a bit late here in Europe now.

[jahansha]

I had same issue. I applied the workaround recommended by brettsam it is working for now, until a fix is provided. Thanks brettsam!

[TechWatching]

I did not test it since, but I can confirm you @brettsam that I had the issue just after running the function through the "Run" button of the portal.

[brettsam]

Thanks all -- we have a fix in #4999 and will be working to roll it out.

[fabiocav]

We'll start the hotfix rollout this morning. Thanks again for reporting, for all the information shared and the patience while we investigated the problem!

[sidhuonline61]

I can confirm that with the latest run-time (Runtime version: 2.0.12742.0 (~2)), I am no longer receiving this error. https://github.com/Azure/azure-functions-host/issues/5060

[fabiocav]

This was globally deployed and resolved with the patch mentioned by @brettsam. Closing as resolved.

[x5engine]

Still an issue in December live on azure Portal after I created the function using the portal idea

Seriously please test your features!

Also the command terminal there is not working at all, no command works!

I wasted a whole day then switched to firebase which was seamless and fast to work with, even netlify functions are so easy to setup and work with no bugs

[a8775]

I have the same problem

[brettsam]

This is a fairly generic error that can occur for a number of reasons. We have another fix coming soon that addresses another code path that could hit this: https://github.com/Azure/azure-functions-host/pull/5330.

@a8775 -- if you have a repro that you can share with us (via github or even a gist), that would be very useful in seeing if this will be addressed with the fix. If so, could you please open a new issue and tag me in it?

[ilayn]

Could you please at least publish the full stack trace in the error message? We are basically clicking around until we can get lucky like a slot machine with error message.

[fabiocav]

@ilayn full exception details will be available in Application Insights (or console if local). What logs are you looking at?

[ilayn]

There is nothing related to logs or App Insight. In our case the issue was the azure web interface was not refreshing after we deleted one subfunction from the function app. So it was still trying to find the deleted app in the frontend. We have succesfully deleted the same app at least three times consecutively until something triggered a refresh. In the meantime browser and even PC restart didn't solve anything. But it could have told us that there was a problem with the deletion instead of this not-so-helpful tongue twister.

This error message can be just removed as it signals absolutely nothing to the user side other than disposed things are disposed. If it is for internal use then it can be avoided as a popup. Because it certainly doesn't help.