Hangfire: Performance problems with huge amount of recurring jobs (1.500.000 Jobs)

[TheUltimateC0der]

As posted before here (https://github.com/HangfireIO/Hangfire/issues/247), I have about 1.6 million recurring jobs.

I have the problem, that these recurring jobs, wont trigger in time, and I think I have found the couse for this.

It is coused by the Execute function of the RecurringJobScheduler (https://github.com/HangfireIO/Hangfire/blob/master/src/Hangfire.Core/Server/RecurringJobScheduler.cs#L57)

var recurringJobIds = connection.GetAllItemsFromSet("recurring-jobs");

scince I am using SQL Server as Database, the query looks like this :

select Value from HangFire.[Set] where [Key] = 'recurring-jobs'

This query fetches ALL (in my case 1.637.623) results from the database at once. So it takes some time to fetch all the results from the database. Even results, which are not necessary to fetch, are getting processed by the foreach.

I think about removing the LastExecution, NextExecution, as well as the TimeZoneId from the Hash table, and add them as a column to the Set table. So you can perform a query to not fetch jobs, that are already enqueued, and dont has to be enqueued.

[odinserj]

O_O what do you do with them? Have you tried to group them together? Have you tried to use fire-and-forget or delayed job instead?

[odinserj]

I have ideas on how to improve performance of delayed and recurring jobs, but they will come in next versions.

[TheUltimateC0der]

Fire-And-Forget, or delayed jobs are not an option for me. I run these jobs, for pulling metadata from an API. Sometimes the metadata changes, and i have to save the changes to my database.

What do you mean by group them together ? If you mean, by let this happen in a single job, even this is not an option for me.

[odinserj]

Can you provide examples of your jobs: what they are doing, how often, etc?

[TheUltimateC0der]

My jobs are pulling metadata from an REST-API, where every job represents an specific object, to be updated. This happens every 2 days.

[octaviordz]

Hi @S3NS4Ti0N
I'm also using Hangfire to send emails, not nearly that many but my design is different I have 1 recurrent job EmailProcess, in your case it would run every minute, then that process get the emails that need to be send and create a background job for every email that it needs to send.
Like:

c# BackgroundJob.Schedule( () => new EmailProcess().SendEbill( JobCancellationToken.Null, "[email]", "[message]"), TimeSpan.Zero );

And I haven't had a problem with the amount of schedule/background job but in your case probably you would use a random delay to help with spread the load.

[TheUltimateC0der]

Hey @octaviordz

I just got a single job for sending mails. The other jobs are pulling data from an REST-API, and save them in my database.

[odinserj]

Just, want to be sure, 1500000 recurring jobs are pulling metadata on a recurring basis?

[TheUltimateC0der]

Sounds strange, i know, but actually ... yes :+1:

[yngndrw]

Obviously I know little about the problem you're solving, but it sounds to me as if you should have a single recurring job which runs every 2 minutes and is responsible for scheduling 1,500,000 fire-and-forget jobs for the REST API queries ? (Or possibly using batches, although if this is running every 2 minutes then it doesn't sound like each individual job is critical so fire-and-forget should work fine ?)

[TheUltimateC0der]

@yngndrw I created the recurringjobs so distributed, that they not run all at the same time. I created them with random generated cron expression for day of week, and time of day.

The key is, that not all jobs run at once on our servers (20 Servers with 5 workers), so we dont have that much workload.

If i would run them all at once, it would take about 4-6 days, to complete all of them

[yngndrw]

Ah I see. So going back to what Sergey suggested, could you divide up the jobs into 10,080 buckets using the modulo operator ? (I.e. One per minute of the week)

In other words there would be 10,080 recurring jobs, each of which would schedule ~160 jobs. Or around 160 jobs would be queued every minute and each individual job would be re-queued every week.

I'm not sure where I got the 2 minutes from but you mentioned that the jobs occurs every 2 days and I'm not sure how that ties into the day of week part, so perhaps the numbers I'm using would need tweaking but I'm hoping my working shows a more manageable scale of numbers.

[odinserj]

@S3NS4Ti0N, it is impossible now to handle such a workload using recurring jobs. As I already mentioned, I know how to optimize it, but the changes will be released with 1.6.0 only. Before that release, this workload will work only with small amount of recurring jobs that create large amount of delayed jobs.

[odinserj]

Implementation of the recurring job scheduler was re-written in latest 1.7.0 betas to pick only those jobs that should be scheduled based on the current time. So now it's capable to handle large workloads much better.

[TheUltimateC0der]

Thx for your efford :)

[jmrnilsson]

Hmm, makes me think..