Azure-functions-host: Problems with scoped lifetime services (AddScoped)

[heikkilamarko]

Hi,

We are using durable functions with dependency injection. In our Startup.cs file, we register our dependencies as follows:

services.AddScoped<IMyService>(serviceProvider => new MyService(...))

In our activity function, we are using normal constructor injection to get IMyService instances.

The problem is that even if we are using AddScoped for registering the service, during the activity function run, each class that asks for the service, gets a different instance of IMyService. This breaks our app logic, because IMyService users won't see each other's changes.

As a workaround to the earlier Azure Functions runtime DI issues, we had the following pinning in place FUNCTIONS_EXTENSION_VERSION = 2.0.12673.0. The pinned version is not supported anymore by Azure, so our function DI is now broken again.

[heikkilamarko]

I find it hard to believe this is by design. If it is so, documentation should state that very clearly.

[heikkilamarko]

The problem is not durable function specific. Same happens with normal TimerTrigger functions also.

[heikkilamarko]

I made more experiments and it seem that the DI problem occurs only when I have used services.AddHttpClient() to register HttpClients. If I comment out services.AddHttpClient() calls, AddScoped works as expected.

[dhanapalschandran]

Hi @heikkilamarko I am also facing the same issue, when this issue happens last week I have pinned to older runtime version, then it started working, now again breaking. I agree it's not an issue with durable function. Even I am using the same DI pattern for HttpClients, Do we have any workaround for httpclient register or waiting for team to provide global fix

[unt1tled]

@dhanapalschandran Same thing on my side. Could it be breaking again due to #5096?

[fabiocav]

@heikkilamarko can you please share repro code? Is this happening with factory registrations only? Some more details/repro will ensure we're looking exactly at the issue you're running into.

@unt1tled / @dhanapalschandran if you can share the code you're having issues with, we can validate with your scenarios as well.

Thanks!

[heikkilamarko]

Hi. See below for a repro code (see the comment block in the function code):

using Microsoft.Azure.Functions.Extensions.DependencyInjection;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using System.Net.Http;

[assembly: FunctionsStartup(typeof(Demo.App.Startup))]

namespace Demo.App
{
    public class Startup : FunctionsStartup
    {
        public override void Configure(IFunctionsHostBuilder builder)
        {
            var services = builder.Services;

            services.AddHttpClient<MyHttpClient>();

            services
                .AddScoped<IMyService, MyService>()
                .AddScoped<IMyServiceSetter, MyServiceSetter>()
                .AddScoped<IMyServiceGetter, MyServiceGetter>();
        }
    }

    public class MyHttpClient
    {
        private readonly HttpClient httpClient;

        public MyHttpClient(HttpClient httpClient)
        {
            this.httpClient = httpClient;
        }
    }

    public interface IMyService
    {
        void Set(int value);

        int Get();
    }

    public class MyService : IMyService
    {
        private int value = 0;

        public int Get()
        {
            return value;
        }

        public void Set(int value)
        {
            this.value = value;
        }
    }

    public interface IMyServiceSetter
    {
        void SetValue(int value);
    }

    public class MyServiceSetter : IMyServiceSetter
    {
        private readonly IMyService myService;
        private readonly IMyServiceGetter myServiceGetter;

        public MyServiceSetter(IMyService myService, IMyServiceGetter myServiceGetter)
        {
            this.myService = myService;
            this.myServiceGetter = myServiceGetter;
        }

        public void SetValue(int value)
        {
            myService.Set(value);
        }
    }

    public interface IMyServiceGetter
    {
        int GetValue();
    }

    public class MyServiceGetter : IMyServiceGetter
    {
        private readonly IMyService myService;
        private readonly MyHttpClient myHttpClient;

        public MyServiceGetter(IMyService myService, MyHttpClient myHttpClient)
        {
            this.myService = myService;
            this.myHttpClient = myHttpClient;
        }

        public int GetValue()
        {
            return myService.Get();
        }
    }

    public class DemoFunction
    {
        private readonly IMyServiceSetter myServiceSetter;
        private readonly IMyServiceGetter myServiceGetter;

        public DemoFunction(
            IMyServiceSetter myServiceSetter,
            IMyServiceGetter myServiceGetter)
        {
            this.myServiceSetter = myServiceSetter;
            this.myServiceGetter = myServiceGetter;
        }

        [FunctionName("Demo")]
        public void Demo([TimerTrigger("0 */5 * * * *", RunOnStartup = true)]TimerInfo timerInfo, ILogger log)
        {
            int value1 = 123;
            myServiceSetter.SetValue(value1);
            int value2 = myServiceGetter.GetValue();

            // Logs '123 - 0'
            // It means that MyServiceGetter and MyServiceSetter got different IMyservice instances.
            // Should be the same since we are using AddScoped to register services.
            // ---
            // If you comment out MyHttpClient injection from MyServiceGetter, it logs '123 - 123'.
            log.LogInformation($"{value1} - {value2}");
        }
    }
}

[t-l-k]

I believe this is the same as my issue #4914 - clearly something wrong with scoped registrations.

@fabiocav can we get an update on these tickets? Are they being investigated? Thx

[APIWT]

@fabiocav Hope you are having a nice day :) We are also hopeful that this issue will be fixed soon. We have spent a lot of time writing code to wire up our services and we really love the way scoped services work in ASP.NET Core. Let me know if there is anything I can do to help get this moving!

[buyckkdelaware]

We have experienced the same problem, but I also noticed this:
It's only the first time after the function started/triggered, that the scoped services are 'wrong'. The second time, it seems to be fine. Has anyone else noticed this? (I was able to reproduce it with the code posted here by @heikkilamarko.)

This is a pretty big problem for us as well, any progress on this issue?

Some additional information:
Azure Functions Core Tools - 2.7.1846
Function Runtime Version - 2.0.12858.0

[t-l-k]

@buyckkdelaware during load testing I have found multiple instances of same service type within the same request scope, it occurs frequently for me, at least 1 in 100 requests. I don't know if the _weight_ of the type is relevant, but my DbContext subclasses are definitely affected - I get missed saves because entities get written to an incorrect DbContext, injected into either repositories, or other infrastructure code. I haven't yet managed to run more than 500 requests due to the DryIOC container scope exploding (which appears to be due to DefaultHttpClientFactory, also likely implicated in the HttpClient errors you've witnessed).

Detecting these duplicates is quite easy, using an AsyncLocal static collection in the affected class's constructor to track when more than 1 distinct hashcode has been found in. DbContexts are not thread safe ofc so can't be used by multiple threads anyway.

I haven't yet "handled" it, but I'll probably just throw an exception and fail the request thread using hashcodes as the detection basis. Without handling it, my code sometimes doesn't know that it has failed, it just misses data.

[darrenhull]

Are there any workarounds for this issue?

[fabiocav]

@darrenhull the workarounds will require some code refactoring at the moment. This is being assigned to a sprint for investigation, so we hope to have an update on this issue soon.

[darrenhull]

Hi @fabiocav,

Do you know if there is an update on this issue yet?

Sorry to keep asking but It’s a pretty big blocking issue for my company’s migration which we are pushing further and further away without being able to suggest when we can resume it.

Thanks

[darrenhull]

Hi @fabiocav,

I see this has been triaged and had the p2 tag. What exactly does the p2 mean, guessing priority2?

Can you give us an update please?

Thanks

[fabiocav]

@darrenhull yes, this is a priority label. The issue has been triaged, but not yet assigned. We'll have the milestone updated once this is assigned to a sprint.

[daniel-white]

@fabiocav: this really sounds like a .net core bug/issue. can you confirm?

i'm running into something similar using a vanilla asp.net core service.

[bartdk-be]

@fabiocav : Not to blame anyone, but how can a basic feature like Scoped or not being labeled as a p2 ? This seems like a rather basic functionality ?
Can you point us to workarounds to resolve this issue ?

[daniel-white]

This appears to be a problem on at least .NET Core 2.1, but is fixed in 3.x. I've raised it here: dotnet/runtime#34586

[bartdk-be]

@daniel-white
Any pointer on where it should be fixed in 3.1 ? Because we faced a similar issue in 3.1 (using the functions DI setup). Thanks !

[daniel-white]

@bartdk-be: im not sure - in my miminal example at dotnet/runtime#34586 make sure your Microsoft.Extensions.Http is the latest version.

[rodhemphill]

Hi @t-l-k ... I guess you don't have a work around then? Same significant problem here - fighting with it for months. It seems to happen on all but trivial systems - certainly with three of our serverside blazor systems with between 30 and 50 services. We are thinking of writing a state memory solution - creating a "StateMemoryService" as a singleton and having it register the instances of each other services against the transaction's SessionId. An in-memory register of . It's a LOT of effort for what is essentially a bug. @fabiocav @daniel-white

[melbappdev]

Hey @fabiocav, if you are up for fixing this I can help. I have three client projects that have the problem and I can easily produce a situation that doesn't get the problem. I can't give you client repos but I have a testing repo that I can share with you. The problem is easy to reproduce and demonstrate in that repo. This repo includes EF and various service classes, some of which cascade inject other services ... which are the most likely candidates for the problem. Whatever causes it the symptoms are that all "scoped" services get translated to "transient".

[t-l-k]

@rodhemphill sticky tape and plasters I'm afraid. Basically failing requests and relying on the runtime to execute the request again with fingers crossed for subsequent attempts. We detect the problem in constructors of classes we know should be scoped, and thus when two instances are created in a logical execution scope (determined by AsyncLocal) we throw an exception and log it. Makes a mess of my logs - see https://github.com/Azure/azure-functions-host/issues/4914#issuecomment-601251057.

It caused quite an embarrasing false start to our UAT phase for sure (deadlines and all), customer was quite alarmed at the amount of failing transactions, and that was as I was chasing it here to get it fixed. I had to refactor in the end, as is the advice per @fabiocav. even changing the order of parameters in constructors reduced the pain! It's still failing on some non-critical code paths, but we can live with that.

Agree- anything non-trivial causes it to fail. We built a polyglot solution, but one component was full DDD stack based on Microsoft's own patterns advice due to the domain complexity. Our simpler Functions Apps (just plain HTTP and read/write storage) work without issue.

We've dropped functions from our list of candidate dotnet technologies for future products & projects in favour of plain ASP.NET Core until this is fixed. Obviously we lose the serverless benefits.

Still, I managed to get money back from Azure for the development headaches it caused, as it is a GA product. If you're suffering the same I would recommend you to also raise support requests. Of course, it didn't come close to covering the amount of expense, inconvenience, and damage to our reputation. I had a very professional support experience with Microsoft and their Functions support team were amazing. But they couldn't fix the underlying issue - that's clearly an engineering problem.

_Perhaps they might treat this more seriously if having to refund more customers via support requests ......... 😉_

[espray]

I have had all of my teams stopped all .Net Core AzFunc development, this Scoped DI issue was the final straw for me. If basic .Net Core DI dose not work who knows what other problems may show up at the wrong time, the risk is too great.

May be send a message/tweet to Damian Edwards @DamianEdwards and/or Scott Hanselman @shanselman about your AzFunc development experience.

[AnthonyIacono]

I think the thing that bothers me about this issue is that it has been around in some form or another for like a year and a half. The original bug was a two line fix, and I imagine the issue we are all facing currently is similar. Can we please get an update from engineering?

[darrenhull]

We too have had to stop using @azfunc as the scoping issue makes them unusable for our solution. This is very frustrating for a team that have embraced Azure functions from day one. I’ve been very disappointed with how long this issue has taken to be resolved @fabiocav , oh wait it’s still active! We too are seeking money back on our subscriptions.

[melbappdev]

I have wasted about 4 days solid on this issue and with impending delivery dates and (soon to be) an unhappy client ... I am glad you responded @t-l-k as you've saved me a lot more wasted time. Yes I agree @darrenhull that it sounds like it has taken too long to fix, but a more important question is: what has it cost? How many people like me have fallen into this hole? @fabiocav

[Arash-Sabet]

This issue is a huge blocker and addressing it has to be expedited. If there's a P1 label indicating a higher priority than P2, that one should be applied instead of P2. We cannot proceed with our project gracefully without a fix! @fabiocav

[fabiocav]

Assigning this to the next sprint. @Arash-Sabet, if you can open an issue detailing your scenario, we might be able to assist with a workaround until this is resolved.

[fabiocav]

The original bug was a two line fix, and I imagine the issue we are all facing currently is similar.

@anthonychu thank you for the patience, can you please share more details on what fix you're referring to?

[Arash-Sabet]

@fabiocav thanks for your reply. I will try to get you a bare-minimum project tomorrow. The only workaround that I could think of was to avoid the "scoped object" dependency injection and instead pass the "scoped" object's instance by adding an interface property to the consumer/service object. I was just wondering, what workaround you may be thinking of so that I could think about it ahead of time? Thanks.

[vitalybibikov]

Impossible to implement Domain Events with Azure functions without this functionality properly...
Can't believe it does not work. Still....

I'm asking to change the priority, as this is a blocker.

Thanks

[vitalybibikov]

Hi @fabiocav
Any news on this?

Can I somehow assist to speed this one up?

Thanks

[vitalybibikov]

As my solution is heavily using DDD, it was not possible for me to ignore absence of this core functionality.

I am using Bindings a lot, so moving to App Service is also not an option at the moment.
For anybody who is planning to use AF in prod, I'm strongly advising not to do so, as they are definitely not in Production Ready state for anything except for "web hooks".

The solution for now is:

To emulate scoped by using AddSingleton and AddTransient, so it will be possible to store all the connections in a Singleton dictionary and grab them from their when necessary.

Connection Key: FullTypeName + Request HashCode (or anything you want)

In startup it looks like this:

services.AddScopedContext<ITenantDbContext, TenantDbContext>();

Some boilerplate is required in order to dispose everything on exit.
My solution is to use DisposeContextFilter class that encapsulates all the disposal logic.

As filters in AF are not in a ready-to-use state also, it looks ugly. Though it works.
I can share the code if necessary.

[Boaz101]

I am also experiencing issues with an Azure function which uses scoped services combined with AddHttpClient. If I comment the AddHttpClient from startup it works correctly. I am moving the HttpClient out of dependency injection, however that has potential pitfalls, too. Hopefully, this is fixed soon because this is causing us a lot of headaches.

[roszell]

@fabiocav any news on this? I see it was assigned to Sprint 75 that was marked closed, but this was the only issue not resolved as part of that. Is this actively being worked on still?

[Schlebusch]

Sooooo ... any news? Glad to find this issue early enough to stop using scoped dependencies in my Azure Functions project.
It is frustrating to read that this issue is open for 9 months now and nothing indicates that it will be solved at all ...
MSFT should create a list of features in their SDK's that are not working.

[APIWT]

@fabiocav Hope you are having a nice day :) Any updates on this?

[t-l-k]

@fabiocav please can you advise as to any updates for this issue? We've some gaps opened up in our roadmap in August in which we're looking forward to paying down some technical debt, and revisiting this issue is on our tracker alongside migrating to dotnet 3+.

Is this recognised as a potential https://github.com/dadhi/DryIoc issue? Is there a correlating ticket with the DryIOC project?

Thanks

[espray]

@jeffhollan @brettsam any update on this issue?

[alexanderbikk]

hi guys
I'm using .net core 3.1 and looks like Scoped services work well but not for the "cold start".
So, when I run the function app either on local or Azure env and the function does the "cold start", my scoped services will be injected more than once per request. When I do the requests after "cold start" seems like all good.

So, for example EF DbContext will be created more that once per request for the "cold start". But for all next requests it will be only the one instance per request.
I'm wondering does it the same issue or I'm doing something wrong?

btw same for my custom services. I create the UoW which includes the EF DbContext, and registered it as Scoped. And again got the same issue.

Thanks

[rodhemphill]

Hi @alexanderbikk. One thing to check is whether you have a "prerender" process running that creates the initial instance. This is a known problem with net core Blazor apps for example.

[alexanderbikk]

Hi @rodhemphill I'm not sure that I get you.

As I said the Scoped lifitime works sometimes, but sometimes I get different instances per request. I can reproduce it on "cold start". Then everything works fine. After some idle the function does a "cold start" again and I again get this issue.

Thanks

[brettsam]

@alexanderbikk -- would you be able to throw a simplified repro up somewhere (in a github repo or even just a gist)? That'd help get this set up...

[alexanderbikk]

hi @brettsam

I believe I have not so much time for it. But I can describe the details and provide some code.
You can try the following scenario:

• Create the Azure function v3 with latest sdk version.
• Create two or more "Services" and UoW which can be configured as Transient in function Startup
• Add EF core DbContext which will be configures as Scoped by default. I used separate project with EF Core and just configuring the DbContext in function Startup.
• Use created DbContext in "Services" and UoW (DI in constructors)
• Call UoW and "Services" in az function to check the injected DbContext instance
• Also you can add some kind of logic in Services and using UoW to save changes. But it will be enough to check the injected DbContext InstanceId

The expected behavior is that the DbContext instance should be the same per one function call
But in my case sometimes(I reproduced it on "cold start") the "Context" instance is different for different Services, so I got some kind of Transient behavior.

Here is possible example of Startup.cs and Services

public class Startup : FunctionsStartup
{
    public IConfigurationBuilder ConfigurationBuilder { get; }

    public override void Configure(IFunctionsHostBuilder builder)
    {
        builder.Services.AddLogging();

        builder.Services.AddHttpClient();

        builder
            .Services
            .AddOptions<ApplicationSettings>()
            .Configure<IConfiguration>((settings, configuration) => { configuration.Bind(settings); });

        ApplicationSettings config = builder.Services.BuildServiceProvider().GetRequiredService<IOptions<ApplicationSettings>>().Value;
        builder.Services.AddDbContext<DatabaseContext>(options => options.UseSqlServer(config.WebSql.ConnectionString));

        builder.Services.AddTransient<IService1, Service1>();
        builder.Services.AddTransient<IService2, Service2>();
        builder.Services.AddTransient<IUnitOfWork, UnitOfWork>();

    }
}

public class Service1 : IService1
{
    private readonly DatabaseContext dbContext;

    public Service1(DatabaseContext dbContext)
    {
        this.dbContext = dbContext;
    }
}

public interface IService1
{
}

public class Service2 : IService2
{
    private readonly DatabaseContext dbContext;

    public Service2(DatabaseContext dbContext)
    {
        this.dbContext = dbContext;
    }
}

public interface IService2
{
}

public class UnitOfWork : IUnitOfWork
{
    public DatabaseContext DbContext { get; }

    public UnitOfWork(DatabaseContext dbContext)
    {
        DbContext = dbContext;
    }

    public void BeginTransaction()
    {
        DbContext.Database.BeginTransaction();
    }

    public async Task BeginTransactionAsync()
    {
        await DbContext.Database.BeginTransactionAsync();
    }

    public void Commit()
    {
        DbContext.Database.CommitTransaction();
    }

    public void Rollback()
    {
        if (DbContext.Database.CurrentTransaction != null)
        {
            DbContext.Database.RollbackTransaction();
        }
    }

    public void SaveChanges()
    {
        DbContext.SaveChanges();
    }

}

UPD: And one important note that I'm using bot UoW and Services not directly in function but inside the other class, let's call it Manager. Which injected and called from azure function.

public class Manager : IManager 
{
    private readonly IService1 _service1;
    private readonly IUnitOfWork _unitOfWork;

    public StateMachineActionsInvoker(IService1 service1,
        IUnitOfWork unitOfWork)
    {
        _unitOfWork = unitOfWork;         
        _service1 = service1;
    }
    public void Call()
    {           
       //do something         
     }
}

public class TestFunction
{
    private readonly IManager _manager;

    public TestFunction(IManager manager)
    {
        _manager= manager;
    }

    [FunctionName("TestFunction")]
    public async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest request,
        ILogger log)
    {
        _manager.Call()
    }
}

UPD2: @brettsam I have tried to reproduce it and no luck

I think I found the issue. At least I found the case when it happens. I called the dbContext.SaveChanges() inside my services and outside of services I called UoW, to use the transaction. In this case the injected dbContext (InstanceId) was different for UoW and my Service. But again it was only on "cold start" really strange behavior.

I have refactored my code, and called dbContext.SaveChanges() outside of the services from my UoW and I can't reproduce the issue. Seems like all good and nothing to check

Thanks a lot.

[pragnagopa]

@brettsam - Any updates here? Should we move this next sprint?

[brettsam]

I haven't been able to investigate yet -- but I've moved it to the next sprint so it comes back on @fabiocav's radar.

[fsalmeida]

I've been facing the same problem and I've found a workaround.
While I was testing some scenarios, I found out that the problem only happens on classes that expect the HttpClient as DependencyInjection.
For example:

`
public class SomeClass
{
private readonly MyHttpClient myHttpClient;
private readonly MyScopedDependency myScopedDependency;

public SomeClass(MyHttpClient myHttpClient, MyScopedDependency myScopedDependency)
{
    this.myHttpClient = myHttpClient;
    this.myScopedDependency = myScopedDependency;
}

public void SomeMethod()
{
    myHttpClient.DoSomething();
}

}
`

In this scenario, the "MyScopedDependency" would not be resolved as Scoped, but if I change my code to something like this:

`
public class ClassToHoldHttpClient
{
public MyHttpClient MyHttpClient { get; }

public ClassToHoldHttpClient(MyHttpClient myHttpClient) => MyHttpClient = myHttpClient;

}

public class SomeClass
{
private readonly ClassToHoldHttpClient classToHoldHttpClient;
private readonly MyScopedDependency myScopedDependency;

public SomeClass(ClassToHoldHttpClient classToHoldHttpClient, MyScopedDependency myScopedDependency)
{
    this.classToHoldHttpClient = classToHoldHttpClient;
    this.myScopedDependency = myScopedDependency;
}

public void SomeMethod()
{
    classToHoldHttpClient.MyHttpClient.DoSomething();
}

}
`

With this code, "MyScopedDependency" is now beeing resolved as Scoped again.

Is it a great solution? Of course not, but as this problem is not solved for about to 1 year, maybe this workaround may save some people.

[GuhBigardi]

I've been facing the same problem and I've found a workaround.
While I was testing some scenarios, I found out that the problem only happens on classes that expect the HttpClient as DependencyInjection.
For example:

`
public class SomeClass
{
private readonly MyHttpClient myHttpClient;
private readonly MyScopedDependency myScopedDependency;

public SomeClass(MyHttpClient myHttpClient, MyScopedDependency myScopedDependency)
{
    this.myHttpClient = myHttpClient;
    this.myScopedDependency = myScopedDependency;
}

public void SomeMethod()
{
    myHttpClient.DoSomething();
}

}
`

In this scenario, the "MyScopedDependency" would not be resolved as Scoped, but if I change my code to something like this:

`
public class ClassToHoldHttpClient
{
public MyHttpClient MyHttpClient { get; }

public ClassToHoldHttpClient(MyHttpClient myHttpClient) => MyHttpClient = myHttpClient;

}

public class SomeClass
{
private readonly ClassToHoldHttpClient classToHoldHttpClient;
private readonly MyScopedDependency myScopedDependency;

public SomeClass(ClassToHoldHttpClient classToHoldHttpClient, MyScopedDependency myScopedDependency)
{
    this.classToHoldHttpClient = classToHoldHttpClient;
    this.myScopedDependency = myScopedDependency;
}

public void SomeMethod()
{
    classToHoldHttpClient.MyHttpClient.DoSomething();
}

}
`

With this code, "MyScopedDependency" is now beeing resolved as Scoped again.

Is it a great solution? Of course not, but as this problem is not solved for about to 1 year, maybe this workaround may save some people.

\o/ It worked for me, thank u @fsalmeida, but I hope that fix coming soon :)

[espray]

@brettsam @fabiocav @jeffhollan @shanselman @DamianEdwards this was part of Spring 84, but was not completed. Now its not part of Sprint 85! WTF! Its coming up on 1yr for this P2 issue. Not building a lot of confidence in the community this way.

[vitalybibikov]

@espray I guess that's some kind of development, when you can just drop things out of a scope forever and ever.
As I can see 2 or 3 things in each of their sprints, that are not closed and just hang there without being tackled.

I've been waiting almost for a year for this thing to be closed.
But I guess something like Scoped is not a priority, so I've just created my own Scoped extension out of Singleton and Transient for one of my customers.

Now, as an architect in my organization I'm advising client's not to use Azure Functions, as they are still not ready for production use.

[espray]

@vitalybibikov same situation here. Actively not recommend AzFunc as well. Mainly because of HOW this issue has been handled.

[fabiocav]

Apologies for the delay on this. We have been, we have a relatively small team, and principally during the current circumstances, it has been challenging to address some of the issues in our sprints, and this was unfortunately, one of them.

I've been working with @brettsam and @anthonychu to resume the work on this. I also apologize for the lack of communication here. We'll keep this issue updated as we move forward.

Thank you for the patience.,

[fabiocav]

Providing an update here at the end of the week. @brettsam and I have made some progress on the different scenarios reported, testing and investigation, hopefully we'll have something ready before the sprint is over, but will continue to provide updates here for awareness.

[APIWT]

@brettsam Glad to see the fix was very simple. Would you be able to provide a timeline on deployment?

[brettsam]

The code change was small, but changes like this can be pretty impactful, which is why we decided to hide it behind a feature flag and get that change out ASAP. We wanted to let everyone try it and let us know if they run into issues while we continue testing scenarios. If all goes smoothly, we will make this the default everywhere in a future release.

As for the timeline -- it's currently in validation and the deployment is starting this week. We don't like giving exact timetables (hotfixes, unrelated incidents, etc can delay a deployment)... but it generally takes 6-ish business days to roll out worldwide once it has started.

[APIWT]

@brettsam Sounds good! Is there an easy way to be notified when the runtime on our functions app resource has support for the feature flag?

[darrenhull]

Thanks for fixing the issue and for the feedback @brettsam and @fabiocav. I too would like to know when it is available.

[t-l-k]

A quick GitHub search: https://github.com/Azure/azure-functions-host/wiki/Configuration-Settings -

AzureWebJobsFeatureFlags: Set to a comma delimited list of beta features to enable. Beta features enabled by these flags are not production ready, but can be enabled for experimental use before they go live.

I'm putting this in the AppSettings of my test deployments _posthaste_ and grabbing the popcorn:

AzureWebJobsFeatureFlags = EnableEnhancedScopes

Meanwhile I'll be watching https://github.com/Azure/azure-functions-host/releases and https://github.com/Azure/azure-functions-core-tools/releases like a hawk waiting for this to drop.

[fabiocav]

The validation phase has been completed and the release is starting to rollout globally. We don't usually publish the Core Tools until after the deployment, but we're making some changes to the process and will see if we can get this up ASAP so you can test locally as well.

Again, thanks for the patience with this!

[fabiocav]

We'll actually keep this issue open (the original intention was to keep it open) so we can continue to use this for feedback/validation.

[ebwinters]

Guys I have the solution after hours of investigation:
Instead of async void for function return type, do async Task. Not sure why this fixes it, any explanation would be great. @fabiocav

[jeffhollan]

In C# you should only use async void when you have to (event handlers, which functions are not in that sense), so likely due to that https://stackoverflow.com/questions/13636648/wait-for-a-void-async-method

[ishepherd]

Not sure why this fixes it, any explanation would be great.

@ebwinters your function would return when it hits the first await. I suppose the functions host has no way of telling there is still work pending, because you did not return a Task for it to wait on.

The async void case is a “fire and forget”: You start the task chain, but you don’t care about when it’s finished. When the function returns, all you know is that everything up to the first await has executed. Everything after the first await will run at some unspecified point in the future that you have no access to.
https://devblogs.microsoft.com/oldnewthing/20170720-00/?p=96655

[ebwinters]

Good to know

[t-l-k]

This is now working great for me, my failed request rates and exception rates have basically dropped to zero, and applications are working better than ever. This brings Azure Functions back to the table for projects in flight!!!

p.s. I skipped local validation using the func.exe tools and just went for the docker image mcr.microsoft.com/azure-functions/dotnet:3.0.14785, which worked great as well.

[brettsam]

@t-l-k -- to be clear, you're running with the new feature flag?

[t-l-k]

@brettsam indeed! Unless my eyes deceive me!

[fabiocav]

@t-l-k great! Thanks for providing feedback on this! And again, thank you for the patience with this! If you find any other issues while enabling this toggle, please let us know as we'll be making this the default once we have some good validation.

For others who were experiencing the same issue, please provide feedback if possible.

Guys I have the solution after hours of investigation:
Instead of async void for function return type, do async Task. Not sure why this fixes it, any explanation would be great.

@ebwinters the information provided by @ishepherd is correct, the pattern you were using will trigger the behavior you're describing and other potential side effects as well (e.g. early termination of your function if the host is deallocated). Without a Task that represents the work being executed, the calling code doesn't have any information about when that work completes and the invocation will be completed as soon as the method returns. By the way, this is not unique to Azure Functions, but applies to .NET in general. We do have checks and warnings in the runtime (if you were to look at you app diagnostics, you'd see a warning about async void usage) to make you aware of that.

[ebwinters]

@fabiocav that makes sense, thanks for explaining. I saw some code snippets in this thread with a void return type, so this may still help them (assuming warnings are being suppressed)

[b10-dslappendel]

@fabiocav Applying the feature flag helped in our case as well.

Our story:
Last month, we noticed some odd behavior regarding scoped services. In our case, a scoped service was responsible for setting up one database connection per request if necessary, and disposing it once the scoped service was disposed. However, after a longer period of time without restarts, the function app ran out of ports. After some (long) investigation we realized that scoped services were sometimes not disposed correctly.
Then I came across this issue 2 weeks ago and we applied the feature flag to the function app that showed this behavior. After that we let it run for a while, and it seems that we no longer leak connections, which suggests that the scoped service is being disposed of properly now.

After 2 months since the last response, is there any idea of when this will be available by default without having to add the feature flag?

[brettsam]

Thanks for providing that data, @b10-dslappendel! This is very useful.

We'll discuss this and see if we have enough usage and data to make it the default.

[simongullberg]

I can tell that adding "AzureWebJobsFeatureFlags": "EnableEnhancedScopes" solved the problem for us when injecting a scoped service.

We were injecting a scoped EF DbContext and this context was injected in two transient services (let's call those Service1 and Service2). Service1 fetched some entity from the database with the injected DbContext and then sent the entity to Service2 where the entity was modified and a call to SaveChangesAsync() was made on the injected DbContext in Service2. The update did not work because there was no change when looking in the database. I suspected that something was wrong with the DbContext so I added an instance variable of type Guid and called Guid.NewGuid. I then expected to see the same Guid on the DbContext injected in Service1 and Service2 but they were different. After adding the feature flag EnableEnhancedScopes the Guids were the same in both Service1 and Service2 and the entity was updated in the database.

[danielreuterwall]

We've had similar issues. Change tracking didn't seem to work between EF DbContext injected directly in function code and EF DbContext injected in scoped services. Using the feature flag did solve that problem for us which is great.

However, in a service added using AddHttpClient in our Startup.cs we injected IHostEnvironment environment to determine if we were in a production environment or not. After the feature flag was set IHostEnvironment environment was no longer available. We've solved it using a regular configuration variable, but the reason for why it's no longer available is unclear. We also tried to revert by removing the feature flag configuration but is seems like once it's set it's there for good.

So two questions:
Any reason or explanation why IHostEnvironment environment is not longer available?
Is it possible to remove the feature flag?

[VaclavK]

stumbled upon this while investigating issue with incomplete ef core (3) persistance..

our pattern is that we have multiple repos supposedly using (scoped) DbContext and we call save changes on one of them - this generally works (is meant to) but in one function app we have http client injected into DI

we started seeing comply partial data persistence (from the repo that was saved, not from the others) which this issue explains

this feature flag - is this PROD ready - assuming this is considered beta flag so it would not be recommended for production usage??? @fabiocav

reproducible in azure (v3 functions) and locally with azure functions CLI - 3.23.0

interestingly enough - we were trying to understand why we would not see this issue consistently/with high enough frequency - our (affected) function is triggered in parallel by SB trigger and behaviourally it seems not to be reproducible with message pump invoking function in parallel (in default config up to 16 messages)

[t-l-k]

@VaclavK I wouldn't consider Azure Functions "production ready" when running _without_ this flag TBH, especially if you're running a design with a properly encapsulated domain model with plenty of service abstractions.

@brettsam Is it enabled by default yet?

[fabiocav]

@t-l-k I'm sending the changes to make this the default in the current sprint. Next release will have the new behavior.

[VaclavK]

@fabiocaFabie just so I understand this - This will be the default behaviour without need to configure anything in?

How do I know that when that release will be the release running in azure for my functions?

[swettstein]

@fabiocav there was a release 2 days ago (3.0.15733). can you confirm this was part of that release?

[fabiocav]

@swettstein are you referring to the change in defaults? If so, the change was not made because of concerns with a few other items landing right before //build and some announcements.

My goal is to have the defaults change with the following release.