Azure-functions-host: DryIoc.ContainerException causes lots of azure function to fail few times every day

[abhiyadav1323]

We see few DryIoc.ContainerException failures throughout the day in multiple azure functions in both our test and prod bed. So, I am hoping to get some guidance on figuring out why these failures are happening and what can we do to stop/reduce them.

Investigative information

• Timestamp: 2020-09-08T22:13:12.0028507Z
• Invocation ID: 5f7fe852-f2f8-45ae-b6bf-84478fd64b71
• Region: West US 2

Repro steps

These functions are either timer triggered or queue triggered.

Actual behavior

We are seeing lots of azure functions failing in our test code few times a day with DryIoc.ContainerException.

Message:

Container is disposed and should not be used: Container is disposed. You may include Dispose stack-trace into the message via: container.With(rules => rules.WithCaptureContainerDisposeStackTrace())

Call Stack:

DryIoc.ContainerException:
   at DryIoc.Throw.It (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 8990)
   at DryIoc.Container.ThrowIfContainerDisposed (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 410)
   at DryIoc.Container.WithCurrentScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 436)
   at DryIoc.ResolverContext.OpenScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 2699)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.WebHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\WebHostServiceProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 51)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver.CreateChildScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\ScopedResolver.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 42)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope (Microsoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=null: 104)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInstanceWrapper.get_InstanceServices (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInstanceWrapper.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 44)
   at Microsoft.Azure.WebJobs.Host.Executors.DefaultJobActivator.CreateInstance (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\DefaultJobActivator.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 32)
   at Microsoft.Azure.WebJobs.Host.Executors.ActivatorInstanceFactory`1+<>c__DisplayClass1_1.<.ctor>b__0 (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\ActivatorInstanceFactory.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 20)
   at Microsoft.Azure.WebJobs.Host.Executors.ActivatorInstanceFactory`1.Create (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\ActivatorInstanceFactory.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 26)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.CreateInstance (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 44)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ParameterHelper.Initialize (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 846)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsyncCore>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35Microsoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.csMicrosoft.Azure.WebJobs.Host, Version=3.0.19.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: 122)

Related information

• Programming language used: C#, .net core

[abhiyadav1323]

Hi @pragnagopa, @brettsam, @fabiocav , would it be possible for you guys to provide a brief explanation as to why this is happening? We have an azure function based infrastructure for Microsoft Sports and we see this exception almost every day across all azure functions across all apps in both test and prod environment which sometimes causes us to miss updates during live games.

[fabiocav]

@abhiyadav1323 this could happen for a few different reasons, so it's hard to diagnose without looking more closely at the details of how you're using the services. The vast majority of cases with this issue we review are caused by function code holding onto services beyond their lifetime scope (e.g. a function execution caching a service/factory or the service provider across different invocations).

[zmarty]

@fabiocav We do use DI singletons for cosmos db clients, http client factory, etc. as per recommendations in the Azure Functions documentation. What information do you need to debug this further?

[uprightbass360]

This also looks to be related : https://github.com/Azure/azure-functions-host/issues/5240