Bull: Jobs sometimes start, sometimes don't

[otaviobps]

Description

I've implemented a Redis and a microservice to run in 1h intervals, using cron. Sometimes the jobs start, sometimes don't, and I'm not really sure why this is happening.

Minimal, Working Test code to reproduce the issue.

(An easy to reproduce test case will dramatically decrease the resolution time.)

I have a server.js file that starts my backend rest api, a queue.js file that process the Queue and a rates.js file that adds the jobs to the Queue.:

server.js:

import express from "express";
import cors from "cors";
import morgan from "morgan";
import BullBoard from "bull-board";
import Queue from "./lib/Queue.js";
import routes from "./routes.js";
const app = express();

app.use(cors());
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
// use morgan if in development
if (process.env.ENVIRONMENT === "development") app.use(morgan("dev"));

// starts the Bull Dashboard
BullBoard.setQueues(Queue.queues.map(queue => queue.bull));

app.use("/api", routes);
app.use("/admin/queues", BullBoard.UI);

app.listen(3333);

queue.js:

import Queue from "./lib/Queue.js";

Queue.process();

rates.js:

import rateJobs from "./jobs/exchangeRates/addToQueue.js";
import Queue from "./lib/Queue.js";

// clean queues
Queue.empty("SetArgentinaRate");
Queue.empty("SetBtcPrice");
Queue.empty("SetBrazilRate");
Queue.empty("SetIndiaRate");
Queue.empty("SetSouthAfricaRate");
Queue.empty("SetVenezuelaRate");

// add job to check rates
Object.keys(rateJobs).forEach(job => job);

lib/Queue.js imports all jobs to a single object:

import Queue from "bull";
import redisConfig from "../config/redis.js";

import * as jobs from "../jobs/index.js";

const queues = Object.values(jobs).map(job => ({
    bull: new Queue(job.key, { redis: redisConfig }),
    name: job.key,
    handle: job.handle,
    options: job.options
}));

export default {
    queues,
    add(name, data) {
        const queue = this.queues.find(queue => queue.name === name);

        return queue.bull.add(data, queue.options);
    },
    empty(name) {
        const queue = this.queues.find(queue => queue.name === name);

        return queue.bull.empty();
    },
    process() {
        return this.queues.forEach(queue => {
            queue.bull.process(queue.handle);
        });
    }
};

One of the jobs is, i.e., to set an exchange rate:

import Currencies from "../../models/currencies.js";
import rp from "request-promise";

export default {
    key: "SetArgentinaRate",
    options: {
        repeat: { cron: "0 * * * *" },
        backoff: { type: "exponential", delay: 30000 },
        attempts: 10
    },
    async handle() {
        // Promise all to get btc price in dest country, spread and btc price
        const [btcArs, { spread }, btc] = await Promise.all([
            rp.get({
                uri:
                    "https://localbitcoins.com/bitcoinaverage/ticker-all-currencies/",
                json: true
            }),
            Currencies.findOne({ currency: "ARS" }).select("spread"),
            Currencies.findOne({ currency: "BTC" }).select("exchange_rate")
        ]);

        // calculate exchange rate
        const exchange_rate = (
            (btcArs.ARS.avg_1h / btc.exchange_rate) *
            (1 - spread)
        ).toFixed(2);

        // set exchange rate to database
        await Currencies.findOneAndUpdate(
            { currency: "ARS" },
            {
                exchange_rate,
                lastUpdate: Date.now(),
                $push: {
                    history: {
                        date: Date.now(),
                        rate: exchange_rate
                    }
                }
            }
        );
    }
};

Here is a codesandbox example: https://codesandbox.io/s/youthful-kapitsa-x5vyz It's not fully functional because there is no redis instance running, but it's the same as I'm implementing.

When the jobs run, they execute with no errors.

I use npm-run-all to start the 3 microservices in parallel:

"prod:server": "node --require dotenv/config src/server.js",
"prod:queue": "node --require dotenv/config src/queue.js",
"prod:rates": "node --require dotenv/config src/rates.js",
"start": "npm-run-all -p prod:*",

Bull version

Running bull 3.12.1 and node 13.7.0

[otaviobps]

Even after running the server and redis for awhile, if I try to force to add the job manually again, it won't add/process the job

[manast]

I am sure the problem is in your code. You should start with the simplest possible case and then build up until you find the issue.

[otaviobps]

Hi @manast, did you have time to open the codesandbox? I'm really not sure why it's sometimes working and sometimes don't. The job's code is all there. Can I somehow log the jobs to see exactly why sometimes is failing to start?

[manast]

Listen, I do not have time to debug others people code. The only way I can solve an eventual bug in bull is if you can isolate the problem so that I can easily reproduce it. As mentioned, most likely the problem is in your code.

[juanvillegas]

I have the same issue. I solved it by adding a unique job id to each repeatable job using Date.now().
When you boot your server, just:

• delete all repeatable jobs
• create new repeatable jobs.
  The unique jobId makes sure the job starts.

[haxtordotid]

Thanks that solved my problem, i create unique job name and it's running again like normal

[otaviobps]

@juanvillegas where did you put the id? Inside the options? I can't find it in the documentation. I have a unique job name for each job, but can't find where to put the id. Thanks!

[juanvillegas]

Hey @otaviobps , refer to the docs here: https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queueadd

I can't go through your code, but it would be something like this:

queue.add('sendReminders', {}, { jobId: 'a_unique_job_id' })

Note however that if it's a repeatable job then you should make sure to remove other instances, as this will effectively create a new repeatable job if the Id is unique.

Just for reference, in our team we solved this doing a clean for all jobs in the queues, then call the empty method for the queue, and then fetch all repeatable jobs using queue.getRepeatableJobs() and delete them one by one using queue.removeRepeatableByKey(job.key). Doing all this effectively removes the repeatable jobs, so there is no need to use the unique id.

[otaviobps]

Hey @juanvillegas thanks for your tip. So doing this way you're not using the options: { repeat: { cron: * * * * * } }? As I understood you are dealing with it manually, is it right?

I'm really not sure why my jobs sometimes start and sometimes don't. I debugged all my code and can't find a mistake.

Basically this is my job:

import Currencies from "../../models/currencies.js";
import rp from "request-promise";

export default {
    key: "SetBtcPrice",
    options: {
        repeat: { cron: "55 * * * *" },
        backoff: { type: "exponential", delay: 30000 },
        attempts: 50
    },
    async handle() {
        // get btc price
        const price = await rp.get({
            uri: "https://www.bitstamp.net/api/v2/ticker/btceur/",
            json: true
        });

        // set btc price to database
        await Currencies.findOneAndUpdate(
            { currency: "BTC" },
            {
                exchange_rate: price.last,
                lastUpdate: Date.now(),
                $push: {
                    history: {
                        date: Date.now(),
                        rate: price.last
                    }
                }
            }
        );
    }
};

When I start my service I call the Queue.process(job.handle) and then I add the job to the Queue with Queue.add()

The strangest part is that if I restart the script a couple of times, all process are added to the queue and process normally. But many times when I start the script no process is added or just some are. I really can't find a mistake in my code. I'm starting to think that the cron is the problem.

[juanvillegas]

@otaviobps Remember the key is not a unique identifier, but rather it's a category. Also, when you define a cron, the Job is categorized as a repeatable job, which is handled in a different way from a regular job.

Try setting a unique key through the options object and see if that works.

[juanvillegas]

Another tip while you test is to open up a redis client and flushall before your tests begin. Otherwise, if you make changes to the code but have old redis data, you may not see accurate results.

[otaviobps]

Yeah thank you very much @juanvillegas I could make it work! I'm passing a uuid as a jobId and everything seems to be working now