Kue: Jobs Getting Stuck in Active State

[davisford]

Hi @behrad I am still experiencing jobs being stuck in the active state, similar to issue #391

I am using [email protected] and redis 2.8.17

I have tried to carefully setup all recommended steps to avoid this. Here are some code snippets. If I'm doing something wrong I would love to hear about it. I've just torn out the bits of source that deal with kue/jobs here and redacted stuff that wasn't relevant.

_producer_ -- separate node process

var jobs = kue.createQueue({ disableSearch: true, redis: config.redis });

var dying = false;
process.on('uncaughtException', function (err) {
   log.error('uncaught exception', err.stack);
   die();
   dying = true;
});

process.on('SIGTERM', function () {
  log.error('SIGTERM');
  die();
  dying = true;
});

function die() {
  if (!dying) {
    jobs.shutdown(function (err) {
      if (err) { log.error('Kue DID NOT shutdown gracefully', err); }
      else { log.info('Kue DID shutdown gracefully'); }
      process.exit(1);
  }
}

// createJob( ) is called on an interval as needed

function createJob(data) {
   var job = jobs.create('deliver:sub', { title: 'Deliver Subscription', sub: data })
    .removeOnComplete(true)
    .attempts(5)
    .backoff({type: 'exponential'})
    .save(function (err) {
      if (err) { return log.error('Error creating deliver:sub', err); }
      log.info(util.format('job[%s] created', job.id));
  });

  job.on('complete', function() {
    /* logic here */
  });

  job.on('failed', function () {
    log.error(util.format('job[%s] failed ', job.id));
  });
}

_consumer_ -- separate node process

var jobs = kue.createQueue({ disableSearch: true, redis: config.redis });

try {
  kue.app.listen(3001);
} catch (err) {
  log.error('Error: could not start kue express server', err);
}

// called in master b/c we use job retries
jobs.promote();

jobs.watchStuckJobs();

var dying = false;
process.on('uncaughtException', function (err) {
  log.error('uncaught exception', err.stack);
  die();
  dying = true;
});

process.on('SIGTERM', function () {
  log.error('SIGTERM');
  die();
  dying = true;
});

function die() {
  if (!dying) {
    jobs.shutdown(function (err) {
      if (err) { log.error('Kue DID NOT shutdown gracefully ', err); }
      else { log.info('Kue DID shutdown gracefully'); }
      process.exit(1);
    });
  }
}

jobs.process('deliver:sub', 5, function (job, done) {
  var domain = require('domain').create();

  domain.on('error', function (err) {
    log.error('domain error for deliver:sub', err);
    done(err);
  });

  domain.run(function () {
    process.nextTick(function () {
      /* logic here that ends with either */
      done();
      /* OR */
      done(err);
    });
  });

[behrad]

please first read my replies to your other topics and try to fix/find properly what's point of your problem. Then I am here to answer to your higher resolution questions.

[davisford]

hi @behrad, i read your comments here as well as scoured through numerous other issues/comments trying to find if there's something i'm doing wrong.

the suggestion to trap uncaught exceptions and use domains is one i have implemented. i pasted the snippet above to hopefully get a second set of eyes and see if there is something i'm not doing correctly.

can you take a quick look when you have a chance, and let me know how jobs can get stuck in the active state if the snippet above were deployed (and what else i can do to prevent it)?

[davisford]

I thought I should add, even thought I am trapping on uncaughtException, SIGTERM, and any uncaught domain error, when the jobs run and do get stuck, there is no evidence that an error was thrown. I log things fairly dilligently, esp. for error scenarios, and we monitor the app very closely using logentries with alerts set up to ping via email, hipchat, etc.

This situation happened again today where about 6 jobs were stuck in the active state. I am able to convert them to inactive in the kue ui, and force them to run again, but I am trying to figure out why they ever get stuck in the active state in the first place.

All nodes are in EC2, same VPC, so network connectivity should be quite good. I don't see any other errors indicative of network loss or disconnects.

[davisford]

Could it be related to a backpressure situation? For example, if the consumer is not able to consume the jobs fast enough and process them, then new jobs end up waiting in the active state.

I have them configured to allow 5 concurrent...e.g. jobs.process('deliver:sub', 5, function (job, done) {

The actual job itself may take several seconds to complete.

[behrad]

ould it be related to a backpressure situation?

No

[davisford]

Hi @behrad, I have looked at all the other threads and applied any and all suggestions. If you think I have missed something, I'd love to hear about it.

Do you have any other suggestions?

[behrad]

busy days... Please BUZZ me a few days later I ll try to help more

[behrad]

the only thing comes to my mind now, is it is related to a poor connection problem, or a unseen, swallowed error in your code...
please try to bind to error events:

kue.on( 'error', ... )

[timcosta]

I am also seeing this issue. when viewing our logs, I can see 32 "stuck" jobs from the past few days, some with no attempts, others where they were attempted twice and then got stuck as active. We are not registering any errors, and all recommendations are in place for error catching.

[behrad]

which version are you on!?
and make sure to read https://github.com/Automattic/kue#error-handling

[timcosta]

Our package.json has "kue": "^0.8.11". We have implemented the uncaughtException catching in our application.

[behrad]

99% percent of a job being stuck in ACTIVE state is user applications miss to call done So first you should try to trace whats happened to your stuck job! and node.js process and worker :)

[davisford]

For the record, this is still a problem for me too. AFAICT I implemented all the suggested error handling with domains, process.on('uncaughtException'), etc. I'm seeing about 2-3 jobs fail out of maybe 50 per day. I just haven't had time to get back to this issue, but it is on my todo list soon. I am also using the latest kue release from npm. If there's any other info I can provide, I'll be happy to do so.

The code I posted above is a stripped down / simplified version of what I have in production.

I intend to instrument kue with additional logging so I can try to gauge better what might be going on.

[behrad]

is your redis connection unstable?
are you calling watchStuckJobs?

[timcosta]

can watchStuckJobs be called on all instances, or just one? There's no documentation on that, and I know that promote can only be called on one instance.

[behrad]

it should be called on just one instance, like promote

[davisford]

I am calling watchStuckJobs on one node (master) only, and the redis connection is stable. All nodes are in the same VPC in amazon, low latency, etc. Plus I have log alerts ping me if, for example redis throws an error in the logs, or the node goes down or fails a status check.

[behrad]

I am calling watchStuckJobs on one node (master) only, and the redis connection is stable. All nodes are in the same VPC in amazon, low latency, etc. Plus I have log alerts ping me if, for example redis throws an error in the logs, or the node goes down or fails a status check

then it shouldn't be your app missing to call done, may be you have some errors in queue.on('error') or may be you are forgetting to call done in your logic...

I'll try to setup your provided code ASAP however I don't think it generates this

[dweinstein]

watchStuckJobs does not look for stuck active jobs it AFAICT looks for stuck inactive jobs. It's because some job (that was promoted to active from queued) in its processing stage isn't calling done() before a process exit and it's not being put into the inactive status. There are many reasons why this could happen.

Either we 1) make sure that active jobs that haven't updated "soon enough" are returned to inactive state, or 2) we similarly watchForStuckActiveJobs(..) -- this is the approach I took below.

I have been playing around with something like this to watch jobs that have been in the active state too long and not updated recently enough:

// this code can be cleaned up quite a bit... and will probably have to be written into a redis script instead
// the interval we want to check the active job list for stuck jobs:
var interval = 5000;
setInterval(function() {

  // first check the active job list (hopefully this is relatively small and cheap)
  // if this takes longer than a single "interval" then we should consider using
  // setTimeouts
  queue.active(function (err, ids) {

    // for each id we're going to see how long ago the job was last "updated"
    async.map(ids, function(id, cb) {
      // we get the job info from redis
      kue.Job.get(id, function(err, job) {
        if (err) { throw err; } // let's think about what makes sense here

        // we compare the updated_at to current time.
        var lastUpdate = +Date.now() - job.updated_at;
        if (lastUpdate > 2000) {
          console.log('job ' + job.id + 'hasnt been updated in' + lastUpdate);
          rescheduleJob(job, cb);  // either reschedule (re-attempt?) or remove the job.
        } else {
          cb(null);
        }

      });
    });
  });
}, interval);

I'm working on a good test case for you @behrad that will hopefully show you why this is happening :-) then maybe I'll send that with a patch for you to look over...

Also obviously doing this as a redis script I guess is preferred.

[behrad]

Thats right for watchStuckJobs, it is looking for inactive->active failures that is Kue responsibility. But I've not made up a watchStuckActiveJobs since it is more the app responsibility... however at kue side, it should make sure no active jobs are stuck, which is being implemented by TTL feature

I'd be more than thankful for your suggestions

[davisford]

So, I've managed to reduce the frequency of this by limiting the number of jobs that can run down to one at a time. In essence, I have removed all concurrency. Jobs can be put into Redis/Kue as fast as they want - I am not rate limiting producers or anything, but job consumers only execute one at a time.

This has improved the stuck jobs for me, but last night, 62 jobs were stuck in the active state this morning. No errors reported. Note that if I lose connectivity to Redis for any reason, the error logs spit out a firehose into our private HipChat window. I never see this.

What I did see this morning were 62 jobs waiting in the active queue, and 1 listed in the failed queue.

I could try to toggle the active jobs to inactive, but it just moved them. It did not trigger them to be consumed and run.

So, I go and restart the web server which is currently responsible for consuming the jobs, and voila...all the jobs stuck in active start draining and complete.

This is the problem I see: It is like the consumer side just gets stuck and no longer pulls from the queue -- even though this web server is active and healthy serving up normal http request just fine -- nothing unusual in system stats (memory, cpu, etc. totally normal). It's like the kue consumer side of kue just gets in a state where it won't consume jobs anymore, but if I restart the app it fixes it.

Why is this?

[behrad]

It is like the consumer side just gets stuck and no longer pulls from the queue -- even though this web server is active and healthy serving up normal http request just fine --

When by each stuck active job, you lose one worker concurrency, then done callback is not called by application. I still think you are missing to call done in some situations/execution paths, however you can add debug console logs to kue source to find out whats exactly happening.
Ensure for each job you are calling done by trace logs...

[davisford]

Thanks for the response @behrad -- that is my next step, enable trace debug on kue lib and see what is really happening.

One follow-up question: Let's say for argument's sake that something strange is happening here like the job executor somehow dies, and i don't trap the error, and the done callback is never called (I've reviewed this code several times and I don't see it, but let's just say that is the case).

Why would one failure cause all the rest to get stuck in the active queue, when the consumer is perfectly healthy and ready to accept active jobs?

I run pm2 on all my node processes which restarts them immediately in the event of failure.

There is something different about this scenario that I don't think is being handled properly. I need to give a bit more background info.

I have the following node processes running on various physical hardware; actual kue jobs are named in fixedtype:font below, and each bold-face list item represents a running node process. There are three processes involved in this flow, which is as follows:

1. subscription-process: queries database for different subscriptions that need to be executed at a certain time of day, when that time of day arrives, it puts a message in kue via a job named deliver:subscription
2. webapp-process: consumes deliver:subscription job and fires off a request to print-process via a job named create:pdf
3. print-process: consumes create:pdf, generates a PDF, uploads to S3 and calls done() on job sending back S3 url
4. webapp-process: consumes create:pdf response, emails link to user, calls done() on deliver:subscription

So, it is really a two-hop workflow. I have other one-hop work-flows in this same system and they never get stuck. It seems to be related to the double-hop. Not sure if that means anything to you, but I feel it is warranted to disclose. I will trace debug the lib and see if I can determine what is really going on.

[behrad]

Why would one failure cause all the rest to get stuck in the active queue,

this is not a true argument. for each job stuck in active queue, it's done has been missed to be called. this is something randomly happening in your case it seems.

when the consumer is perfectly healthy and ready to accept active jobs?

how can prove each worker is perfectly healthy and ready to accept active jobs? each stuck jobs represents a worker waiting for done to be called, until that worker process is restarted.

you can chain jobs and create flows... don't see it related to this. If you can create and post a code which reproduces those active stuck jobs, I sure would debug it

[deckchairhq]

I'm also experiencing this, after anything between 1 job or 2 hours of processing jobs, the queue will just stop consuming, often leaving jobs sat in an active state.

done() is being called, domains wrap the process and the job, and all the jobs complete perfectly and cleanly.. it just doesn't continue processing.

I currently have 10 EC2 machines running 8 cluster processes each, sitting idle.

If I iterate over and set all Active jobs to inactive, this will put my stuck jobs back on the queue (1000+ items sitting in the inactive queue anyway), but this still won't kick off the workers.

I'm using a hosted Redis solution (openredis.com). I didn't experience this issue with Agenda or Bull, so i'm content with the job code (and have tried many, many variations anyway).

If I turn on queue.watchStuckJobs(); and then run:

queue.failed(function (err, ids) {
    ids.forEach(function (id) {
        Kue.Job.get(id, function (err, job) {
            if (job.error().indexOf("Error getting image file") >= 0) {
                job.remove();
            }
        });
    });
});

With ~2800 jobs in my failed queue, it crashes my hosted Redis and I receive an email e.g.:

We received notifications that your Redis instance could not be reached at around 8:09 am PST. We were able to resolve the issue by restarting the Redis instance, resulting in a downtime of about 5 seconds. We believe the downtime started due to some Lua scripts being done your server.

Without queue.watchStuckJobs(); things _seem_ to be fine (this is a production system I can't keep attempting to crash it to be totally sure).

My jobs have a TTL set to 7mins but nothing happens, even after hours. I even have a timeout inside my job, that will kill it's wrapping domain at 6 mins, though this is irrelevant because the job cleanly finishes, and calls done().

@behrad Really sorry, I _know_ you need to see code / a test case. I can't post my company code and as this is affecting a production system at the moment it's more likely i'm going to have to jump to a more basic library to give things moving again and buy us time to work this out. I just wanted to add an extensive +1.

Restarting my app will get things moving again for upto 2 hours.

[dweinstein]

@thinkgareth curious why did you move away from Bull? One of the reasons I wanted to switch to it is because when I tested similar work flows with it there were no stuck jobs.

[soveran]

I think what is causing problems is the call to KEYS in watchStuckJobs, click here to see that line in context.

Here's the problematic part of the Lua script:

var script =
'local msg = redis.call( "keys", "' + prefix + ':jobs:*:inactive" )\n\

The KEYS command shouldn't be run in a production environment. For more information, check the warning in the command's documentation.

[deckchairhq]

I waited until all workers were idle last night, then moved all active jobs to inactive and restarted the cluster, it worked through the night and did not idle. In the AM I shut down a few of the servers, 5 minutes later, everything was idle again.

@soveran Thanks! that addresses the crash part of my comment.

@dweinstein Bull has more features and a better API now than it did when we used it, will be taking another look, but will still have to implement certain Kue features e.g. TTL / Exponential backoff, User interface (the Bull UI project wasn't very useful last time i checked).

[behrad]

1. watchStuckJobs looks for inactive stuck jobs, not active stuck ones.
2. Kue is currently fragile in poor networks or server crashes, from 1.0 I will refactor it to full lua scripts (same way Bull selected)
3. @thinkgareth I expect you to give more details about that 5mins, see anything in your redis log? kue app? network?

[behrad]

Here's the problematic part of the Lua script:

var script =
'local msg = redis.call( "keys", "' + prefix + ':jobs:*:inactive" )\n\
The KEYS command shouldn't be run in a production environment. For more information, check the warning in the command's documentation.

I think we should fix this, thank you @soveran

[davisford]

All my issues seemed to have gone away after migrating to bee-queue

[alelavoie]

I had the same problem for a while : when the server is stopped or crashes while a job is being processed, the job stays forever in active state after the server restarts and subsequent jobs are never processed.

I tried, upon server initialization, to put all crashed active jobs in inactive state like described in Programmatic Job Management but it didn't work very well for me. It unblocked the queue but the crashed jobs that were put in inactive state were never processed, even when I changed their "attempts" option to 10.

Here is how I fixed it :

Upon initialization, I check for any active jobs using kue.active() or kue.Job.rangeByState(). For every active job found, I create a new job with the same data and I call job.complete() on the old one. Note that you need to do this upon server initialization and before queue.process() is called.

Here is the code I use to do so :

 kue.Job.rangeByState( 'active', 0, 1000, 'asc', function( err, jobs ) {
      jobs.forEach(function(job) {
         job.complete();
         queue.create(job.type, job.data).save(); 
      })
 });

Note that this code is not async so, if you call queue.process() soon after, you'll need to make it async in order to be sure that all active jobs are processed before running the queue.

[behrad]

Upon initialization, I check for any active jobs using kue.active() or kue.Job.rangeByState(). For every active job found, I create a new job with the same data and I call job.complete() on the old one. Note that you need to do this upon server initialization and before queue.process() is called.

I'm afraid this won't work for a clustered environment. You can't be sure if that active job is stalled or has a worker actively working on it.

I'm proposing you to test Kue 1.0.0-alpha which is available in v1 branch of repo. I'm testing that to release it. And you will have non of current problems with that :)

[behrad]

I just need to add an upgrade note ASAP

[alelavoie]

I'm afraid this won't work for a clustered environment. You can't be sure if that active job is stalled or has a worker actively working on it.

It still works for us since we process kue jobs on a specific cluster only but you're right, it's not a good solution.

I'll be sure to check out 1.0.0-alpha. Very good news !

[drunkhacker]

There was also same issue here. My business critical logic (sending SMS) uses Kue, and one job is stuck in active state. Anyway, it's good to hear the release of 1.0.0.

[phongtattuan]

I also have the same issue 😭
Some jobs are stuck in active state
Is there a way to force those job run ?

[behrad]

Is there a way to force those job run ?

You can handle them programmitcally according to docs, however you'd better switch & test v1 branch.