Hangfire: Asp.net core DI and ServiceScope

[dazinator]

I have been using Hangfire job server, in my asp.net core application.

In startup.cs I use a custom Activator so that I can inject any registered services, into jobs:

 public void Configure(IApplicationBuilder app)
            {

 // shortened for brevity
 GlobalConfiguration.Configuration.UseActivator(new ServiceProviderJobActivator(app.ApplicationServices));


}

The activator looks like this:

   public class ServiceProviderJobActivator : JobActivator
    {
        private IServiceProvider _serviceProvider;

        public ServiceProviderJobActivator(IServiceProvider serviceProvider)
        {
          // Service provider from IApplicationBuilder.ApplicationServices.
            _serviceProvider = serviceProvider;
        }

        public override object ActivateJob(Type type)
        {
            var implementation = serviceProvider.GetRequiredService(type);
            return implementation;

        }
    }

I am then injecting a DbContext into some of my jobs.

I have hit the issue, that the same instance of the DbContext was being accessed concurrently from different threads, because the same instance was being injected into multiple jobs running in concurrently : https://github.com/aspnet/EntityFramework/issues/8450#issuecomment-301117278

Two possible suggestions are:

1. Register my DbContext as transient. This means each hangfire job will get a new instance. But it also impacts the rest of my application running in the same process (i.e my MVC controllers will now have a transient instance rather than scoped to http context) which is not desirable.

2. Create a service "scope" at an appropriate level that ensures hangfire will only run a single thread at a time, within that same service scope.

I'd like to solve the problem using number 2. However I am not sure how to appropriately create and manage these service "scopes" for Hangfire.

Has anyone else solved this already that would care to share a solution?

[dazinator]

I thought perhaps creating a new scope, when activating each job, might solve the issue. i.e like this:

  public override object ActivateJob(Type type)
        {

            var scopeFactory = _serviceProvider.GetService<IServiceScopeFactory>();
            var scope = scopeFactory.CreateScope();
            var scopedContainer = scope.ServiceProvider;           

            var implementation = scopedContainer.GetRequiredService(type);
            return implementation;

        }

This would be similar I believe to what happens in MVC and services scoped to a "Request". I.e a request comes in, and MVC middleware creates a new ServiceScope for the lifetime of that request. The service scope is then disposed of at the end of the Request.

But the problem I have with Hangfire, is that: I should dispose of the scope after the job has finished funning. Not sure of an appropriate hook to do that part. Should I be managing this scope somewhere else?

[dcorbett-fivebridgesllc]

Hangfire uses this to create an instance of the class on which it's going to call the method to run the job. Shouldn't hangfire call dispose on that instance after the job completes, and wouldn't that work for you? You can implement your own dispose override in that class to ensure that your DbContext is disposed properly.

[dazinator]

Not really.

So the issue is, when the job is activated, I need to first create a new ServiceScope and resolve the job from that scope, so that the DbContext that is injected into the job classes constructor is a scoped instance. If I don't do this, and resolve the job from the ServiceProvider without creating a scope, then multiple jobs running in parallel will get the same DbContext instance injected, leading to errors as DbContext is not thread safe. Disposing of the DbContext in a Dispose method, within the job, doesn't solve that problem. I understand all this, however the question now becomes, if I create a new ServiceScope within the ActivateJob method, at what point can I then Dispose() of this ServiceScope. I need to dispose of the ServiceScope after the job has finished running. Disposing of the ServiceScope will also dispose of all of the services, including the DbContext that was injected for the lifetime of that job. This means that overriding / implementing Dispose in the job itself, to dispose of it's services that were injected (like the DbContext) becomes unnecessary. This is in line with what happens for RequestServices in the MVC stack. When a http request is received, a new ServiceScope is created for the lifetime of that request, and the HttpContext.RequestServices IServiceProvider is owned by that scope. This is set by middleware in the request pipeline. Services are then injected into your MVC classes during the request. At the end of the request, the middleware disposes() of the ServiceScope, which disposes of all of those scoped services. This is why you don't need to Dispose() of a DbContext in your MVC controller classes etc.
So I'm trying to find an equivalent place in Hangfire to mange this service scope - i.e I want to

1. Create the scope
2. Activate a job
3. Run the job
4. Dispose of the scope.

I can't inject ServiceScope into a job (and I don't think I would want to either.) and so I can't get the job to dispose of the ServiceScope in it's dispose method.

[burningice2866]

Since the scope is an implementation detail of your code and not anything Hangfire is aware of i don't see it as unreasonable for you to have the full responsibility of disposing it again.

How you would do it is tricky, since the only reference to the scope is on the second line of your method so you would somehow need to pass that as a reference to your job as well for it to dispose of when its finished. You could do it by adding a IServiceScope argument to your job-constructor and add the scope-instance to your container so your serviceprovider knows how to resolve it and inject it into your job.

[dazinator]

@burningice2866 - Yeah it's a good suggestion and I gave this one some thought (i.e passing the scope into the job) and I agree with you it's not unreasonable to expect custom code to manage the scope, and thats what I am trying to acheive. I don't think it's unreasonable to expect an appropriate hook that allows us to manage this scope. Injecting the service scope into the job, and making the job dispose of it, on dispose I guess would works as long as:

1. The life of a service scope = the lifetime of a single job.

I'm not sure if the above is optimal though, The appropriate lifetime for a service scope (i.e at what point you would want to create it, and dispose of it) seems to be dependent on how hangfire is going to schedule / execute it's jobs. I need to ensure only a single thread is going to be using services from the scoped container at one time, but that doesn't mean that's it's best to create a new scope for each thread, or for each job. For all I know, Hangfire may have some concept of separate "job runners" that process a single job a time, and therefore the appropriate place to create and hold onto this "service scope" would be at that "runner" level, not per job instance. This is just a contrived example, as I don't know enough about how hangfire works. So, perhaps creating new Scoped Container per job instance is the only way to do things, or perhaps it would be a better fit with Hangfire, if a more limited number of Service scopes were created, and cached / re-used. What do you think?

[dazinator]

Discovered global filters: GlobalJobFilters.Filters.Add(new LogEverythingAttribute());
Thinking this might be better than than trying to inject scope, and requiring job to dispose. Will give it a go.

[burningice2866]

Have you looked at JobActivator.BeginScope?

It looks like Hangfire is creating an explicit scope around each job being executed and its up to your implementation to return a scope here which Hangfire can dispose of again.

[dazinator]

Gave up trying to use a global filter to dispose of the service scope. The activator only has a Type for the job. So if it creates a scope, it can only store it under the jobs Type. Ideally it would need to be stored under an Id for the job, so that the Global Filter could dispose of the scope for the particular job ID after the job as finished executing in OnPerformed.

[dazinator]

I noticed that the Activator as a BeginScope() method. Could this be related? What is it for?

[burningice2866]

As referred to, Hangfire wraps the execution of each job in a BeginScope which is disposed at the end. The job-instance is resolved using the scope, which in turn can do its magic or just default back to calling ActicateJob on JobActivator which happens in the default SimpleJobActivatorScope implementation.

Soo, by the looks of it, you should override BeginScope and return a custom implementation of JobActivatorScope.

[dazinator]

Soo, by the looks of it, you should override BeginScope and return a custom implementation of JobActivatorScope.

Looks good, i'll try that.

[dazinator]

Hmm.. despite me doing this:

  GlobalConfiguration.Configuration.UseActivator(new ServiceProviderJobActivator(app.ApplicationServices));

My activator isn't being called. In the stack trace from within the job's constructor, I am seeing:

>   Hangfire.AspNetCore.dll!Hangfire.AspNetCore.AspNetCoreJobActivatorScope.Resolve(System.Type type)   Unknown

I am registering a job like so:

  RecurringJob.AddOrUpdate<TestJob>("Test Job", (x) => x.DoSomeTestJobAsync(1, JobCancellationToken.Null), "*/1 * * * *");

Any ideas why my Activator isn't invoked?

These are the hangfire packages I have installed:

[dazinator]

.. it seems there is already an implementation that manages service scope by doing exactly what we suggested: https://github.com/HangfireIO/Hangfire/blob/a436bb8fcf590c6172517c202097d747326c03c0/src/Hangfire.AspNetCore/AspNetCore/AspNetCoreJobActivator.cs

So now I need to figure out, why I am still seeing multiple DB context instances being accessed from my concurrent jobs.

[dazinator]

Ok going to close this, as got to the bottom of it at last.
Thank you for all your help and patience.

Hangfire handles this in 1.6.12 by registering a service that does what we suggested: https://github.com/HangfireIO/Hangfire/blob/129707d66fde24dc6379fb9d6b15fa0b8ca48605/src/Hangfire.AspNetCore/HangfireServiceCollectionExtensions.cs#L85

I was looking at this, because I was using 1.6.7 and this didn't exist in 1.6.7. Upgrading has sorted everything.

Although I may have noticed a separate issue, in that AddHangfire() seems to register the specialised Activator for asp.net core now, and my call to UseActivator<> wasn't honoured. Perhaps its order dependent. Not a big issue for me as I no longer need a custom Activator as supported out of the box but may catch someone else out.

[burningice2866]

Looking at the code here https://github.com/HangfireIO/Hangfire/blob/master/src/Hangfire.Core/Obsolete/StartupConfiguration.cs#L73 or here https://github.com/HangfireIO/Hangfire/blob/master/src/Hangfire.Core/GlobalConfigurationExtensions.cs#L49 i would say the order matters and last one wins since its a static Current-property which can only hold one value at a time.

[stajs]

Glad you have a way forward, but I gotta ask: why were you trying to dispose of your scope at all?

Our situation is probably a little different to yours. We are running a self-hosted (using Kestrel) website in a Windows Service, but we have been using a very similar job activator class to your second comment since the DNX days and we are still on Hangfire 1.6.6:

public class HangfireJobActivator : JobActivator
{
    private readonly IServiceProvider _serviceProvider;

    public HangfireJobActivator(IServiceProvider serviceProvider)
    {
        if (serviceProvider == null)
            throw new ArgumentNullException(nameof(serviceProvider));

        _serviceProvider = serviceProvider;
    }

    public override object ActivateJob(Type jobType)
    {
        // Ensure jobs run in their own "request" scope. Please.
        var scopeFactory = _serviceProvider.GetService<IServiceScopeFactory>();
        var scope = scopeFactory.CreateScope();
        var job = scope.ServiceProvider.GetService(jobType);

        if (job == null)
            throw new Exception($"Could not activate a job of type '{jobType.FullName}'.");

        return job;
    }
}

We too felt the pain of concurrent DbContext access from concurrent jobs and this has been solid in our experience for the last ~2 years.

[dazinator]

By not disposing of the Service Scope, and just letting it go out of scope, it just means you are relying on the GC to run at some unknown point in the future and call Dispose on it for you. In theory, if the GC ran and disposed of the scope whilst a job was running and using an injected DbContext, id expect that DbContext could suddenly be disposed of whilst your job is still using it. The fact you say there are no issues may be because your jobs finish before the GC gets to disposing of the scope. That aside, imho its still better to dispose of disposables rather than rely on the GC to do it at some point in the future imho, as you can free up resources earlier.

[stajs]

Yeah, for sure. I don't disagree with you, I just find these discussions intriguing. I wouldn't try and out guess the GC unless I could measure a problem, but I agree with the spirit of your point for the same reason you might wrap something in a using.

Interesting theory about the GC disposing a scope mid-job: is that possible? Naively I thought there was a reference to the scope for the life of a job, but perhaps you are right and we have never hit it.