Hangfire: Recurring job that runs on a non default queue is retried on default

[tstojecki]

When a recurring job that is scheduled to run on a non default queue fails it is retried on the default one:

The job is scheduled this way:
app.UseHangfireServer(new BackgroundJobServerOptions { Queues = new[] { "WEB" } })

In hangfire console, the job shows the following:

Enqueued
Triggered by DelayedJobScheduler
Queue:
DEFAULT

Failed
An exception occurred during performance of the job.

Enqueued
Triggered by recurring job scheduler
Queue:
WEB

Is this a bug? Seems to me, a job should be retried on the same queue as it was originally scheduled.

[pbolduc]

I have seen this problem as well. I have one set of HangFire tables in my database supported two related applications. One app uses queue A and default. The other app uses queue B. Occasionally, a job in queue B will fail and it is rescheduled on the default queue. The retried job always fails again because the first app does not have permissions/configuration to support the failed job.

I am running: 1.6.6

[pbolduc]

When DelayedJobSchedule enqueues the job, it does not set the queue in the EnqueuedState. The default constructor for EnqueuedState sets the queue to DefaultQueue (default).

In my database, all of my jobs are in state Enqueued, however, in the JobQueue table, the queue is default. I am adding my job using this code:

RecurringJob.AddOrUpdate<IJobType>(name, job => job.Run(), Cron.Hourly(), queue: "employee");

Here is the log:

[mikeholczer]

I don't think this is limited to recurring jobs. I'm using 1.6.8 and seeing the same thing for one-time jobs I created with BackgroundJobClient.Create and specify a non-default queue. If the jobs fail, they get re-tried on the default queue. I grant that jobs seem to be retried as scheduled jobs, so maybe it could be considered a scheduled job specific problem.

[mikeholczer]

@pbolduc looks like you diagnosed the issue pretty specifically, have you spent any time trying to fix it. I was thinking about giving it a stab.

[pbolduc]

I tried look at it, however, where it gets re-queued, the original queue is not available. I looked into how I could get the original queue name, but could not figure it out in the time I had. Eventually, I just created a separate database for the 2nd application. :-(

[kbrimington]

I panicked a bit when reading over this issue. We're transitioning to a microservices approach, and were depending on queues to provide separated concerns between apps while sharing a single Hangfire.

I figured out a workaround for you to consider. Here is a sample job. Note the QueueAttribute on the Run method.

public class CheeseJob
{
    [Queue("root")]
    public void Run()
    {
        throw new System.Exception("Things are not happy");
    }
}

// ...
RecurringJob.AddOrUpdate<CheeseJob>("Cheese", job => job.Run(), Cron.Hourly(), queue: "root");

After giving this a run and letting it retry a few times, I queried the Job table and got this:

I gave it another try, this time removing the QueueAttribute from the job method. This produced the queue-hopping symptom.

public class CheeseJob
{
    public void Run()
    {
        throw new System.Exception("Things are not happy");
    }
}

// ...
RecurringJob.AddOrUpdate<CheeseJob>("Cheese", job => job.Run(), Cron.Hourly(), queue: "root");

From this information, it would seem that you can work around the queue hopping issue by applying a QueueAttribute to the job method.

Even so, I would really like to see a fix for the queue hopping behavior for two reasons: the behavior is unexpected, and it is prohibitive to folk who have some reason not to perma-assign the job to a single queue using an attribute.

[brunopessanha]

There's an attribute in Requeue to initial queue #502 that also changes the behaviour to enforce the original queue rather than the default one.