Agenda: If server is shutdown, unfinished jobs won't resume

[armellarcier]

Using

function graceful () {
    agenda.stop(function(){
        process.exit(0);
    });
}
process.on("SIGTERM", graceful);
process.on("SIGINT" , graceful);

Locked jobs are unlocked BUT when server restarts agenda does not restart jobs that weren't finished. These have a lastRunAt prop set :

{
    "_id" : ObjectId("58c12bc6b3302977a218891c"),
    "name" : "send notification",
    "data" : "58c11db58757bacd99ebc132",
    "type" : "normal",
    "priority" : 0,
    "nextRunAt" : null,
    "lastModifiedBy" : null,
    "lockedAt" : null,
    "lastRunAt" : ISODate("2017-03-09T10:17:44.331Z")
}

How to handle jobs that should be restarted ?

[SymphonyKenanBerbic]

+1 We have same problem with agenda.

[SymphonyKenanBerbic]

And this is how we temporarily fixed that issue, we execute this part of code on server start

db.mongoLib.connection.on('open',function () {
    db.mongoLib.connection.db.collection('agendaJobs', function (err, collection) {
        collection.update({lockedAt: {$exists: true }, lastFinishedAt:{$exists:false} }, { $unset : { lockedAt : undefined, lastModifiedBy:undefined, lastRunAt:undefined}, $set:{nextRunAt:new Date()}},{multi:true},function (e,numUnlocked) {
            if(e)
                console.error(e);

            console.log(utils.String.format("Unlocked #{0} jobs.",numUnlocked));
        });
    });
});

[mirkods]

+1 same problem

[gowridev]

db.mongoLib.connection.on('open',function () {
^

ReferenceError: db is not defined.

could please tell me how configure db.mongoLib. does it point to mongo databaseuri or agenda job instance ?

[SymphonyKenanBerbic]

Sorry man I didn't saw message on the time, "db" is some type of db context, where I put all of my mongoose models. MongoLib is mongoose driver. What you can do in your example

const mongoose = require('mongoose');
mongoose.connect("address of your mongo server");
mongoose.connection.on('open',function () {
    mongoose.connection.db.collection('agendaJobs', function (err, collection) {
        collection.update({lockedAt: {$exists: true}, lastFinishedAt: {$exists: false}}, {
            $unset: {
                lockedAt: undefined,
                lastModifiedBy: undefined,
                lastRunAt: undefined
            }, $set: {nextRunAt: new Date()}
        }, {multi: true}, function (e, numUnlocked) {
            if (e)
                console.error(e);
            console.log(utils.String.format("Unlocked #{0} jobs.", numUnlocked));
        });
    });
});

or something similar

[lushc]

336, #375, and maybe others are due to the fact that job definitions are being declared dynamically. https://github.com/agenda/agenda/issues/109#issuecomment-60286993 explains why this doesn't work, although the example given is slightly wrong since agenda.every() does enforce a single job instance, which is probably where the confusion is stemming from and causing feature requests like #209. I think the docs need improving in this regard, because every() is quite high up and is presented as the de-facto way of scheduling a repeating job, but then it's easy to miss this caveat:

Every creates a job of type single, which means that it will only create one job in the database, even if that line is run multiple times.

So then people equate job instances to definitions and think it's a 1:1 relationship, so different data means you'll need a different (dynamic) definition. I'll admit it took me a while to realise that I had to manually work with instances to have "shared" repeating jobs, because the higher level APIs are enticing but don't really solve this common use-case:

// basically agenda.every() but without setting the job type to "single" & multiple definition scheduling
const every = (interval, name, data, options) => {
  return new Promise((resolve, reject) => {
    agenda.create(name, data)
      //.schedule(interval) if you don't want the job to run right away
      .repeatEvery(interval, options)
      .save((err) => err ? reject(err) : resolve());
  });
};

agenda.define('userJob', (job, done) => {
  console.log('Hello user ' + job.attrs.data.userId);
  done();
});

const job1 = every('1 minute', 'userJob', {userId: 1});
const job2 = every('5 minutes', 'userJob', {userId: 2});

// don't forget save is async
Promise.all([job1, job2])
  .then(() => console.log('scheduled jobs'))
  .catch((err) => console.error(err));

[bradwbradw]

I'm not creating jobs dynamically, and I've implemented calling stop() on sigterm. When i have jobs queued and i hit ctrl-c, then restart, i can see they start running, then eventually complete, as expected. however, if a job is running when i hit ctrl-c and reboot, it appears to be stuck in this state and never finishes. I'm looking at the state (running, queued, completed, etc) by using Agendash with very fast refresh interval (0.2 seconds)

In my situation, the "dead jobs" aren't locked, they are "running" (but not actually running, because they never finish)

[lushc]

@bradwbradw when a job is picked up and ran the lockedAt attribute is set to the current timestamp, but a job won't be unlocked if it's interrupted while running (by setting lockedAt to null), so Agenda's auto-unlock mechanism will kick in by picking up jobs with a lockedAt time older than 10 minutes (a default value which can be changed).

If you interrupt a running job, restart your application, and then manually change the interrupted job's lockedAt time to 10+ minutes earlier (using the command line or something like Robomongo, you should find the job is picked up again.

If your job doesn't have a lockedAt time set then could you please paste the affected job document to help us debug this issue?

[bradwbradw]

sure @lushc

{
    "_id" : ObjectId("595168c133b4c071b8cf361f"),
    "name" : "test job",
    "data" : {
        "num" : 0
    },
    "type" : "normal",
    "priority" : 0,
    "nextRunAt" : null,
    "lastModifiedBy" : null,
    "lockedAt" : null,
    "lastRunAt" : ISODate("2017-06-26T20:04:22.892Z")
}

that's very strange, because this job isn't being picked up even though lockedAt is null (i assume that qualifies for "10+ minutes earlier" ) ... i haven't changed the name of the job either

[bradwbradw]

@lushc perhaps it's possible that the job actually completed, but the dashboard didn't update (indicated it's a bug in agendash, not agenda) ?

[lushc]

@bradwbradw sorry, didn't see you replied with your job object. When the job is completed it has a lastFinishedAt field set, which yours hasn't got, so didn't complete. If you check out https://github.com/agenda/agenda/issues/410#issuecomment-310526028 or this commit there's a query you can implement in your application to "unlock" these interrupted jobs. This should make its way into Agenda at some point I think.

[franciscofsales]

@lushc what if we use mongo's system.js collection to store the definitions? I am considering doing it but I'm not sure how "bad" it is.. since its not recommended to store application logic function. I have critical jobs that can't fail and they're being ran in kubernetes, so restarts can happen... not sure yet how to tackle this..

[nhim175]

Don't know if similar issue but I have 1 recurring job running every hour but everyday it stops at midnight, and stops forever until I restart the worker.

[nicolocarpignoli]

My problem looks the same. I correctly schedule dynamic jobs:

   var agenda = new Agenda({ mongo: Mongo.getMongoConnection() });
    agenda.define(jobName, function (job, done) {
        job.alert = alert;
        func(job, done);
    });
    agenda.on("ready", function () {
        agenda.schedule(formatDate(date), jobName, data);
        agenda.start();
    })

This works if I keep my node server running. If i restart It, no job is scheduled (outdated jobs and jobs with future nextRun time). At server startup I do:

var agenda = new Agenda({ mongo: Mongo.getMongoConnection() });
    agenda.on("ready", function () {
        agenda.start();
        console.log("Starting agenda scheduler...");
    })

I also attach a 'find' query result of one of the jobs:

{ "_id" : ObjectId("598f9f957390ae1e79b3ec89"), "data" : { "hasToBeRemoved" : false }, "lastModifiedBy" : null, "name" : "Power", "nextRunAt" : ISODate("2017-08-13T02:45:00.472+02:00"), "priority" : 0, "repeatInterval" : "*/15 * * * *", "repeatTimezone" : null, "type" : "single" }
Can you help me find the problem? I cannot modify locketAt or other fields cause my document doesn't have them as you can see above.

I'd like any possible workaround: this issue unfortunately hits the major benefits of using this great module

[simison]

@nicolocarpignoli do you have changes to test if branch feature/optimize-find-and-modify (PR #476) fixes this issue?

[nicolocarpignoli]

@simison Yea I tried, but It doesn't seems to work either. Now my jobs are saved in mongo with 13 fields, like this one:

{
    "_id" : ObjectId("5990ade67390ae1e79b3ec8d"),
    "data" : {
        "hasToBeRemoved" : false
    },
    "disabled" : false,
    "lastFinishedAt" : ISODate("2017-08-13T21:52:19.157+02:00"),
    "lastModifiedBy" : null,
    "lastRunAt" : ISODate("2017-08-13T21:52:06.833+02:00"),
    "lockedAt" : ISODate("1970-01-01T02:00:00.001+02:00"),
    "name" : "Power",
    "nextRunAt" : ISODate("2017-08-13T21:53:06.833+02:00"),
    "priority" : 0,
    "repeatInterval" : "1 minute",
    "repeatTimezone" : null,
    "type" : "single"
}

But when I restart the server the job is not scheduled

[marcelinhov2]

Hey, do you guys have some update about this issue?

I already finished my implementation with Agenda at my project ( great module by the way ) but I'm with this same problem, when I need to restart my server, all my jobs stop to work.

[bastianwegge]

@marcelinhov2 we got it working with the latest version of agenda, contact me personally if you need help

[OmgImAlexis]

@bastianwegge can you share what you did so we know if we need to add better documentation or if there's a bug we're unaware of.

[bastianwegge]

Okay so first things first. We installed agenda on a bare minimum setup. I usually go for a mongoose.js file that holds all the config information. After using nodemon to watch for file-changes whenever I'm editing my scheduler.js, I noticed that after 1-2 restarts agenda stopped working. We dug deeper using the debug described in an issue I read before using env DEBUG="agenda:*" node dist/scheduler.js. Then we noticed that the job seemed to be locked.

First we updated mongoose to the latest Version (4.11.7 as of this writing)
Then we defined the following:

mongoose.connection.on('open', () => {
  mongoose.connection.db.collection('agendaJobs', (err, collection) => {
    collection.update({ lockedAt: { $exists: true }, lastFinishedAt: { $exists: false } }, {
      $unset: {
        lockedAt: undefined,
        lastModifiedBy: undefined,
        lastRunAt: undefined
      },
      $set: { nextRunAt: new Date() }
    }, { multi: true }, (e, numUnlocked) => {
      if (e) { console.error(e); }
      console.log(`Unlocked #{${numUnlocked}} jobs.`);
    });
  });
});

mongoose.connect(mongoUri, {
  useMongoClient: true,
  socketTimeoutMS: 0,
  keepAlive: true,
  reconnectTries: 10,
});

This is kind of the same what was defined earlier with the subtle difference that the event is defined before the actual MongoDB connection is opened. After doing so, the initial "bug" was gone and we could continue.

Any questions left?

[simison]

I'm doing something similar each time our app is restarted:

https://github.com/Trustroots/trustroots/blob/master/config/lib/worker.js#L175-L228

I'm also attempting to gracefully stop Agenda:
https://github.com/Trustroots/trustroots/blob/master/config/lib/worker.js#L238-L255

...and haven't had issues with jobs not running since that change. Previously jobs which were run repeatedly and often (every couple minutes) would get stuck. The shorter the frequency, more likely they seemed to get stuck.

[pruhstal]

Will this ever make it into the core library? The solutions above are helpful, but not elegant and require an additional module to "fix this issue". Btw, I've used agenda for years and am so happy it's still being maintained after having come back to it for a new project.

[OmgImAlexis]

@pruhstal we're looking into it.

[ChrisLahaye]

Any updates?

[pcyu]

Thank you so much for all the work you guys have put in so far. It's just so refreshing to see that this discussion has already taken place.

[bonesoul]

any updates on this?

[Ni55aN]

agenda.lockLimit(1); allows to avoid a problem for me, but concurrency not working correctly

[kamleshkarwande]

Thank you for the support on this issue folks. Has this issue been resolved?
I am using v2.0.2, the issue still persists.

[lkho]

Hi everyone, the issue is 2 years old already.
I am using v2.0.2.

The quickest solutions I found so far (without doing extra things like running queries on startup) are:

1. DO NOT perform the graceful shutdown, just exit the process and let the lock stays there. So that agenda will pick up the job again after the lock expired. Remember, DO NOT call done() inside the handler, or else lockedAt will be reset and the job will either marked as success or fail.

   Note: this will only work if you are going to exit process afterwards, since this hack will keep the job in the internal state of agenda, and you will not be able to restart agenda in the same process.

2. DO perform graceful shutdown of agenda (agenda.stop()). Inside the handler, set job.attrs.nextRunAt to now, and call done() directly (with optional error to indicate cancel), so agenda will unlock the job, and the job is re-queued to be run immediately.

   Note: for the mechanism of how to notify the handler to cancel the job is up to you (maybe emitting a 'cancel' event on the agenda object?)

Anyone have better idea on this?

[Ni55aN]

Anyone have better idea on this?

I'm using pg-boss with my wrapper https://github.com/timgit/pg-boss/issues/102

[sam13591980]

Okay so first things first. We installed agenda on a bare minimum setup. I usually go for a mongoose.js file that holds all the config information. After using nodemon to watch for file-changes whenever I'm editing my scheduler.js, I noticed that after 1-2 restarts agenda stopped working. We dug deeper using the debug described in an issue I read before using env DEBUG="agenda:*" node dist/scheduler.js. Then we noticed that the job seemed to be locked.

First we updated mongoose to the latest Version (4.11.7 as of this writing)
Then we defined the following:

mongoose.connection.on('open', () => {
  mongoose.connection.db.collection('agendaJobs', (err, collection) => {
    collection.update({ lockedAt: { $exists: true }, lastFinishedAt: { $exists: false } }, {
      $unset: {
        lockedAt: undefined,
        lastModifiedBy: undefined,
        lastRunAt: undefined
      },
      $set: { nextRunAt: new Date() }
    }, { multi: true }, (e, numUnlocked) => {
      if (e) { console.error(e); }
      console.log(`Unlocked #{${numUnlocked}} jobs.`);
    });
  });
});

mongoose.connect(mongoUri, {
  useMongoClient: true,
  socketTimeoutMS: 0,
  keepAlive: true,
  reconnectTries: 10,
});

This is kind of the same what was defined earlier with the subtle difference that the event is defined before the actual MongoDB connection is opened. After doing so, the initial "bug" was gone and we could continue.

Any questions left?

Thanks it was really good point! Just one question perhaps not really related to this topic, is it possible to use Agenda with mongoose? So far based on my understanding it does not support mongoose; however, what you did here was using mongoose to change the field of lock. Thanks!

[lkho]

@sam13591980 Actually Mongoose is just a wrapper around the raw MongoDB native driver. They are two separate things, and can be used independently and simultaneously.

The code you quoted is just using Mongoose to 'open' the connection, and after that get the 'raw' mongodb.Db instance from it and perform a 'raw' mongodb query. That's nothing to do with Mongoose at all.

You can also reuse any existing Mongoose connection in Agenda from the beginning. Just pass the mongodb.Db instance inside Mongoose to the Agenda constructor:

const mongooseConnection = mongoose.createConnection();
await mongooseConnection.openUri('mongodb://........');
const agenda = new Agenda();
agenda.mongo(mongooseConnection.db, 'collectionName', cb);

Connection.db from Mongoose docs.

*Note: this will not make any connection between mongoose and agenda, i.e. no hooks in mongoose will be run when Agenda updates objects, and no schema will be registered in mongoose.

If you are wishing to use Mongoose as a real ODM, Agenda currently does not have such code for you (as far as I know). You can find the schema structure from Agenda's source and build your own Mongoose schema, link the corresponding Agenda methods to Mongoose schema methods.

**Note2: this will ALSO NOT run mongoose hooks when Agenda updates objects, as Agenda will still use the raw mongodb.Collection to perform its queries.

[SergeyVatz]

Hello I have the same problem, after restart server agenda doesnt rerun jobs.

mongoose.connection.on('open', () => {
  mongoose.connection.db.collection('agenda', (err, collection) => {
    collection.update({ lockedAt: { $exists: true }, lastFinishedAt: { $exists: false } }, {
      $unset: {
        lockedAt: undefined,
        lastModifiedBy: undefined,
        lastRunAt: undefined
      },
      $set: { nextRunAt: new Date() }
    }, { multi: true }, (e, numUnlocked) => {
      if (e) { console.error(e); }
      console.log(`Unlocked #{${numUnlocked}} jobs.`);
    });
  });
});

When Im trying to connect to db again, agenda collection have updated nextRunAt field, but I doesnt run my job :(
What is problem? And How I can solve it?

[GeorgiPopov1988]

Hello!
I have the same problem when executing this code:

// Agenda config file
// All Agenda properties (defaultConcurrency, processEvery, lockLimit, defaultLockLimit, defaultLockLifetime) are with default values.

agenda.on('ready', async () => {
    mongoose.connection.db.collection('agendaJobs', function (err, collection) {
        collection.updateMany({ lockedAt: { $exists: true }, lastFinishedAt: { $exists: false } }, {
            $unset: {
                lockedAt: undefined,
                lastModifiedBy: undefined,
                lastRunAt: undefined
            }, $set: { nextRunAt: new Date() }
        }, { multi: true }, function (e, numUnlocked) {
            if (e)
                console.error(e);
            console.log("Unlocked #{0} jobs.", numUnlocked);
        });
    });
});

// Job definition
agenda.define(
        'My Job Name, { lockLimit: MAX_ATTEMPTS, concurrency: MAX_ATTEMPTS, lockLifetime: 10000 },
        async (job, done) => {
            const param1 = job.attrs.data.pram1;
            try {
                    Do something with param1 ...
                    done();
            }
            catch (err) {
                done(new Error(err.message));
            }
        }
    );

// Job start
agenda.now('My Job Name', {
                    param1: value1
                });

Tasks that were in the "running" status before the server was shut down are stuck in the "queued" status after the server is started.

[farhan711]

@GeorgiPopov1988 Did you find any solution? I have same issue where after restarting the previous jobs doesn't resume.
Even after trying the above-mentioned approach, I am facing same issue, job is queued but not picked up.

[GeorgiPopov1988]

I still don't have a solution for that. I'm sorry!

[Marcvander]

Hey, thanks for all the support everyone. I'm also experiencing something similar, though it happens without the server restarting. Often, the jobs just stay locked without the server restarting. Here is one of the document:

{ "_id" : ObjectId("5ed7bc1707ec2a71aa37a9fc"), "name" : "Courier Tracking App", "type" : "single", "data" : null, "lastModifiedBy" : null, "nextRunAt" : ISODate("2020-09-30T12:01:27.439+02:00"), "priority" : 0, "repeatInterval" : "1 minute", "repeatTimezone" : null, "lockedAt" : ISODate("2020-09-30T17:21:36.607+02:00"), "lastRunAt" : ISODate("2020-09-30T12:00:27.439+02:00"), "lastFinishedAt" : ISODate("2020-09-30T12:00:45.092+02:00"), "failCount" : 20, "failReason" : "no primary server available", "failedAt" : ISODate("2020-09-24T06:12:55.999+02:00") }

As we can see, the lastRun started at 12:00:27, was finished at 12:00:45, and is scheduled 1 minute after it started, at 12:01:27. So far so good. But for some reasons the lockedAt value is 17:21:36. How come could it be locked 5 hours later, without having run in the previous 5 hours? I tried to set lockedAt to null, but when doing so, it gets immediately a new lockedAt value equal to the current timestamp.

I'm setting agenda as follow:
On one file:

`const agenda = new Agenda({ db: { address: process.env.MONGODB_COMMO } });

agenda.processEvery('5 seconds');
agenda.start();

async function graceful() {
await agenda.stop();
process.exit(0);
}

process.on('SIGTERM', graceful);
process.on('SIGINT', graceful);

export default agenda;`

And on another file this:

`import { default as agenda } from "./agenda";

const TRIGGER_ID = "Courier Tracking App";

agenda.define(TRIGGER_ID, async (job, done) => {
//my logic here
done();
});

agenda.on('ready', () => {
agenda.every("1 minute", TRIGGER_ID);
});

export default {}`

[farhan711]

Hey, thanks for all the support everyone. I'm also experiencing something similar, though it happens without the server restarting. Often, the jobs just stay locked without the server restarting. Here is one of the document:

{ "_id" : ObjectId("5ed7bc1707ec2a71aa37a9fc"), "name" : "Courier Tracking App", "type" : "single", "data" : null, "lastModifiedBy" : null, "nextRunAt" : ISODate("2020-09-30T12:01:27.439+02:00"), "priority" : 0, "repeatInterval" : "1 minute", "repeatTimezone" : null, "lockedAt" : ISODate("2020-09-30T17:21:36.607+02:00"), "lastRunAt" : ISODate("2020-09-30T12:00:27.439+02:00"), "lastFinishedAt" : ISODate("2020-09-30T12:00:45.092+02:00"), "failCount" : 20, "failReason" : "no primary server available", "failedAt" : ISODate("2020-09-24T06:12:55.999+02:00") }

As we can see, the lastRun started at 12:00:27, was finished at 12:00:45, and is scheduled 1 minute after it started, at 12:01:27. So far so good. But for some reasons the lockedAt value is 17:21:36. How come could it be locked 5 hours later, without having run in the previous 5 hours? I tried to set lockedAt to null, but when doing so, it gets immediately a new lockedAt value equal to the current timestamp.

I'm setting agenda as follow:
On one file:

`const agenda = new Agenda({ db: { address: process.env.MONGODB_COMMO } });

agenda.processEvery('5 seconds');
agenda.start();

async function graceful() {
await agenda.stop();
process.exit(0);
}

process.on('SIGTERM', graceful);
process.on('SIGINT', graceful);

export default agenda;`

And on another file this:

`import { default as agenda } from "./agenda";

const TRIGGER_ID = "Courier Tracking App";

agenda.define(TRIGGER_ID, async (job, done) => {
//my logic here
done();
});

agenda.on('ready', () => {
agenda.every("1 minute", TRIGGER_ID);
});

export default {}`

Hello @Marcvander Please refer to this comment at https://github.com/agenda/agenda/issues/996#issuecomment-667503606

[Marcvander]

Hello @Marcvander Please refer to this comment at #996 (comment)

Thanks @farhan711, I'll give it a try and write back here once tested 👍

[Marcvander]

Hello @Marcvander Please refer to this comment at #996 (comment)

Thanks @farhan711, I'll give it a try and write back here once tested 👍

We've had it now in production for 4 days, and so far it seems it solved the problem. No jobs are being locked anymore, they all run without problem. Seems like this is a good fix, thanks @farhan711 !