Azure-functions-host: ScopedResolver in V2 is not thread safe.

[mlankamp]

Investigative information

Please provide the following:

• Timestamp:
• Function App version (1.0 or 2.0): 2.0.12115.0
• Function App name:
• Function name(s) (as appropriate):
• Invocation ID:
• Region: local emulator

Repro steps

1. Inject IServiceProvider into a custom Extension
2. In the IBinding.BindAsync call the CreateScope on the IServiceProvider
3. Use the custom Extension in function (for instance an httptrigger)
4. Make sure the function is called multiple times, this will cause an error in the CreateChildScope.

The pattern to call CreateScope is used in various Dependency Injection extensions:

• https://github.com/cResults/azure-functions-v2-dependency-injection/blob/master/InjectBinding.cs
• https://github.com/DibranMulder/azure-functions-v2-dependency-injection/blob/master/AzureFunctionsV2.DependencyInjection/InjectBinding.cs

All these extensions are based on a solution for V1, because in V1 the problem didn't exist.

Expected behavior

The ScopedResolver from the host uses a HashSet<ServiceScope>, this is not thread safe.

Actual behavior

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Function1 ---> System.InvalidOperationException: Exception binding parameter 'sampleRepo' ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at System.Collections.Generic.HashSet`1.SetCapacity(Int32 newSize)
   at System.Collections.Generic.HashSet`1.IncreaseCapacity()
   at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value)
   at System.Collections.Generic.HashSet`1.Add(T item)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver.CreateChildScope(IServiceScopeFactory rootScopeFactory) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\ScopedResolver.cs:line 55
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 101
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.CreateScope(IServiceProvider provider)

Known workarounds

• None

[doggy8088]

Azure Functions 2.0 is just GA. https://github.com/Azure/app-service-announcements/issues/138

Is there any workaround on this issue for now?

[mlankamp]

Only use singleton and don't call CreateScope

[jeromekol]

Are you seeing these exceptions in AI? Because we receive random 500s which are not logged to AI.

[paulbatum]

I was just able to repro this randomly while I was investigating another issue, just using HttpTrigger. So the scope is much broader than what is reported above. Its a pretty bad bug that will impact the reliability of all V2 function apps that use HTTP trigger.

We will prioritize getting this fixed in the next functions V2 release (deployment will likely start around 10/3).

In my case the stack trace was:

System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Collections.Generic.HashSet`1.AddIfNotPresent(T value)
   at System.Collections.Generic.HashSet`1.Add(T item)
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.ScopedResolver.CreateChildScope(IServiceScopeFactory rootScopeFactory) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\ScopedResolver.cs:line 55
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.CreateScope() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 101
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesFeature.get_RequestServices()
   at Microsoft.AspNetCore.Http.DefaultHttpContext.get_RequestServices()
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass4_1.<UseMiddleware>b__2(HttpContext context)
   at Microsoft.AspNetCore.Builder.UseWhenExtensions.<>c__DisplayClass0_1.<UseWhen>b__1(HttpContext context)
   at Microsoft.Azure.WebJobs.Script.WebHost.Middleware.HomepageMiddleware.Invoke(HttpContext context) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Middleware\HomepageMiddleware.cs:line 34
   at Microsoft.AspNetCore.Buffering.ResponseBufferingMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.Azure.WebJobs.Script.WebHost.Middleware.HttpExceptionMiddleware.Invoke(HttpContext context) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Middleware\HttpExceptionMiddleware.cs:line 23
   at Microsoft.Azure.WebJobs.Script.WebHost.Middleware.AppServiceHeaderFixupMiddleware.Invoke(HttpContext httpContext) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Middleware\AppServiceHeaderFixupMiddleware.cs:line 35
   at Microsoft.Azure.WebJobs.Script.WebHost.Middleware.ScriptHostRequestServiceProviderMiddleware.Invoke(HttpContext httpContext, WebJobsScriptHostService manager) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Middleware\ScriptHostRequestServiceProviderMiddleware.cs:line 36
   at Microsoft.Azure.WebJobs.Script.WebHost.Middleware.HostWarmupMiddleware.Invoke(HttpContext httpContext) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Middleware\HostWarmupMiddleware.cs:line 34
   at Microsoft.Azure.WebJobs.Script.WebHost.Middleware.HostAvailabilityCheckMiddleware.Invoke(HttpContext httpContext, IScriptHostManager scriptHostManager) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Middleware\HostAvailabilityCheckMiddleware.cs:line 47
   at Microsoft.Azure.WebJobs.Script.WebHost.Middleware.EnvironmentReadyCheckMiddleware.Invoke(HttpContext httpContext, IScriptWebHostEnvironment webHostEnvironment) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Middleware\EnvironmentReadyCheckMiddleware.cs:line 29
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

[fabiocav]

A targeted fix for this was just made. There's ongoing work for a set of tests exercising this code path.

Just something to keep in mind with the implementations linked to above; you should exercise some care when exposing the runtime service provider. We have work in progress right now to expose DI capabilities built into the host (which would eliminate the need for a custom extension to address those scenarios), including support for constructor injection. As part of that work, we'll be adding some runtime validation and documenting types you can safely depend on (services and their lifetime scope). The V1 approach used by many extensions would be a safer alternative for now (with an extension scoped service provider only) and would also avoid the bug you just hit.