Bull: Guidance on structuring within an existing Node.js app

[pruhstal]

Hi there,

I'm migrating a previous project over from using Agenda.js to Bull. The main reason is to consolidate into redis by removing the additional mongo dependency, along with the extra features that Bull provides.

The way I structured tasks in Agenda was the following:

/tasks/index.js

import Agenda from 'agenda';
import * as email from './email';
const a = new Agenda({db: { address: process.env.MONGO_URL }});

email.newMember(a);

a.start();

export default a;

/tasks/email.js

export const newMember = (agenda) => {
  agenda.define('new member', (job, done) => {
    // Do some work
    if (err) return done(err);
    done();
  });
};

Then calling the task from say: /controllers/users.js

import tasks from '../tasks/index';

// Do some work (e.g. create a user)
tasks.now('new member', {
  username: user.username,
  email: user.email
});

• In bull.js, where should I define my queues? Where I use them or a single place like I had them above in /tasks/index.js?

• Should I have a queue for every different task or every different task type?
  e.g. this:

  const newMemberEmailQueue = new Queue('new member email', 'redis://127.0.0.1:6379');
  const updatePasswordEmailQueue = new Queue('update password email', 'redis://127.0.0.1:6379');
  const articlePublishQueue = new Queue('article publish', 'redis://127.0.0.1:6379');

vs. this:

  const emailQueue = new Queue('email queue', 'redis://127.0.0.1:6379');
  const articleQueue = new Queue('article queue', 'redis://127.0.0.1:6379');

I took a look at the examples, but wanted to be sure I was following best practices as the Agenda api is a bit different.

Any advice/info would be greatly appreciated.

EDIT: one thing I think is helpful that would be nice (and I'm willing to contribute to this once I know the proper way to structure things) is having a section in the docs for an example project structure like: https://github.com/agenda/agenda#example-project-structure.

[pruhstal]

Here's an example of what I've come up with so far:

Everything goes into /worker

This is my /worker/worker.js:

import Queue from 'bull';
import redis from 'redis';
import config from '../config';
import { run as emailRun } from './tasks/send-email';

const client = redis.createClient({
  host: config.redis.host,
  port: config.redis.port,
  enable_offline_queue: true
});

const emailQueue = new Queue('email notifications');
emailQueue.process(emailRun);
emailQueue.on('completed', (job, result) => {
  job.remove();
});

export { emailQueue };

All tasks live in /worker/tasks.

Here's my /worker/tasks/send-email.js:

export const run = (job, done) => {
  // Send an email in here with some library
  done();
};

Then within my package.json I have an additional command I run:

    "worker": "./node_modules/.bin/babel-node worker/worker.js",

@manast thoughts on this structure so far? Any issues I might run into with this setup?

[manast]

I have not so much feedback to give about this. There are endless possibilities on how to structure your project depending on so much factors, there really isn't anything right or wrong.