Bull: How to retry a job until it's done?

[tonywei92]

Description

How to retry until its done?

Minimal, Working Test code to reproduce the issue.

>> Jobs.js
const Sync = require('./Sync');
const Queue = require('bull');
const job = new Queue('data sync');

job.process(function(job, done){
    console.log('job start');
    Sync.dataSync(job.payload, function(response){
        if(response.status === 200){
            done(response.data);
        }
        else{
            console.log('error');
            throw new Error('some unexpected error');
        }
    })
});

module.exports = {
    job
};


----------------------------------
>> Sync.js
const axios =require('axios');

async function dataSync(payload, callback){
    return await axios.post('https://www.google.com/queue',payload).then(response=>{
        if(callback){
            callback(response);
        }
        return response;
    }).catch(error =>{
       if(callback){
           callback(error.response);
       }
       return error.response;
    });
}

module.exports = {
    dataSync
};

---------------------------------------
>> index.js

const Jobs = require('./Jobs');

Jobs.job.add({
   payload: 'test'
});

Bull version

"bull": "^3.4.4"

Additional information

run with "node index.js"
The result:

error
(node:23105) UnhandledPromiseRejectionWarning: Error: some unexpected error
    at /home/vagrant/sites/queueTest/Jobs.js:13:19
    at axios.post.then.catch.error (/home/vagrant/sites/queueTest/Sync.js:11:12)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
(node:23105) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:23105) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

of course it's error, but it's never retry.
please help!

[manast]

if you do not use "done" and instead use promises all the way it will work. Remember that you can return promises in your process function instead of using "done", but do not specify that argument in that case.

[tonywei92]

Hello @manast , Can you show me the working code, please?
I refactor Jobs.js as follow:

const Sync = require('./Sync');
const Queue = require('bull');
const job = new Queue('data sync');

job.process(function(job){
    console.log('job start');
    Sync.dataSync(job.payload, function(response){
        console.log('status', response.status);
        if(response.status === 200){
            return Promise.resolve(response.data);
        }
        else{
            console.log('error');
            return Promise.reject('some unexpected error');
        }
    })
});

module.exports = {
    job
};

still not retrying

[manast]

You need to return a promise from dataSync as well:

const Sync = require('./Sync');
const Queue = require('bull');
const job = new Queue('data sync');

job.process(function(job){
    console.log('job start');
    return Sync.dataSync(job.payload);
});

module.exports = {
    job
};

>> Sync.js
const axios =require('axios');

async function dataSync(payload){
    return await axios.post('https://www.google.com/queue', payload);
}

module.exports = {
    dataSync
};

[KonstantinKudelko]

@manast If I right understand, I don't need to handle errors of the Promise to retry job?

For example for this case should my job retries?

    const testQueue = new Queue('test', 'redis://127.0.0.1:6379');

    testQueue.process(async (job) => {
      console.log('job start');
      return Promise.reject('reject');
    });

    await testQueue.add({payload: 'request'});

Or for example with done()
I expected that job should retry all the time, nope?

    const testQueue = new Queue('test', 'redis://127.0.0.1:6379');

    testQueue.process(async (job, done) => {
      console.log('job start');
    });

    await testQueue.add({payload: 'request'});

[manast]

The second example is wrong because you are defining a "done" function, however it is never called, so the job will never complete (or fail).

[KonstantinKudelko]

The second example is wrong because you are defining a "done" function, however it is never called, so the job will never complete (or fail).

Yep, but first one, I see like my job invokes only one time

For example, I need to solve this case:
My service should make a request until gets response with status 200 until this job should retry the request. When request gets error, job should retry

[bentesha]

You will have to specify number of attempts and backoff strategy when adding a job for bull to retry a failed job.

Below job will be attempted 5 times in 5secs intervals before failing completely

testQueue.add({ payload: 'request' }, { attempts: 5, backoff: 5000 });

Check docs for other options to meet your specific needs