Bull: Preventing creating the duplicated queues

[Tin-Nguyen]

Hi all,

Is there any pending improvement or work around solution to prevent creating the duplicated queues when doing the code below multiple times (same name)?

const newQueue = new Queue('same-name')

Currently, my code invokes the code above multiple times, even though I just set the queue proceeding only 1 job at a time, but I see a lots of jobs running at the same time instead of only one. So, I guess because of initializing multiple times a queue with the same name, Bull still create multiple queues, instead of one, is it working by design or a bug?

[Tin-Nguyen]

I actually figured out a solution by writing some code in my abstraction class for Bull queue. When my code invokes to create a new queue, I will check the data in Redis with keys bull:<queue-name>*, if we find out any data exist there, it means the queue already exists, then we just re-initialize the queue, instead of creating new one. But I don't know Bull provides a function to re-initialize the existing queue or not, I'm looking for a work-around solution on that.

[manast]

That is not needed. Bull does not create anything in redis when you instantiate a queue. You can call new Queue('same-name') as much as you want and it will not affect its use.

[Tin-Nguyen]

That's great, let me check my code again.

[Tin-Nguyen]

hey @manast , the duplication issue happens on my code when I run multiple times the code below:

const newQueue = new Queue('same-name')
newQueue.process((job, done) => {})

[Tin-Nguyen]

It seems if we run process function multiple times we will make the queue proceed multiple jobs at the same time even we set the concurrency is one. I think we can make an update at https://github.com/OptimalBits/bull/blob/master/lib/queue.js#L807, if we have any running jobs, we should not execute processJobs function or if the queue is already running, we should not call the function.

Another issue, if I call const newQueue = new Queue('same-name') multiple times, it creates a lots of Redis connections.

[manast]

If you define several process functions in new queue instances, yes indeed they will process in parallel. What would be the point on defining more than one process function if you still want to process one job at a time?

[Tin-Nguyen]

Now I understand how Bull queue works, so I fixed the issue from my side. Thanks @manast

To me, since we're having concurrency configuration already to make the queue process multiple jobs in parallel, so that I don't think we should allow the job is processed in parallel when we define several process functions in a queue. It's better if we override the current process function when we invokes it again.

[manast]

The concurrency parameter is for scaling "vertically", where defining multiple process methods in different machines allows you to scale "horizontally".

[Tin-Nguyen]

@manast can we run multiple queues (10+) on the same server has 1 CPU core?

[samhoooo]

I am facing similar problem too.
I have multiple nodeJS servers, would like to share a same queue (for example, the queue is named as 'same-name').

In my code, I initialize a queue object like this:

const queue = new Queue('same-name');
queue.process("sharedTask", 1, sharedTask);

But then I realized that my queue isn't shared between my nodeJS servers, each server run it's own queue.

Is there any way for Bull to share a queue among servers? Thanks.

[manast]

If you are using the same redis server for all the queues then they will be shared, that is the whole point of using redis as backend.

[chrisfsampaio]

@manast I did some investigation (version 3.7.0) on this:

You can call new Queue('same-name') as much as you want and it will not affect its use.

While that is correct as in it will not create anything new in Redis, it's easy to think you can call new Queue('same-name') as much as needed each time you need to add a job to it. Which, in turn, will create a Redis connection that seems to be held open as long as the node process is running.

For example, the setup I had was:
1) An HTTP endpoint to create a job /add-job
2) Each call to this endpoint would get the queue by calling new Queue('same-name') and add a job to it.
3) A new Redis connection is open every time. And it's not closed even after the function is finished. (lsof | wc -l returns an increasing number every time a job is added, and the number is never reduced.)
4) After a few thousand calls to this endpoint, the node process would crash with the error "too many files open". As the operating system can only handle a maximum number of open socket connections. 💣

The solution was to have a single instance of the queue, created outside of the function that creates jobs.
I'd be happy to create a PR documenting this if the solution sounds appropriate to you. I saw that you already have a section about "Reusing Redis connections", but I'm not sure that is clear enough to inform that a Redis connection is created and held open every time a queue is instantiated in JS.

Here's a test case if you want to quickly reproduce this locally:

setInterval(() => {
        const queue = new Queue('test-queue');
        queue.add({ foo: 'bar' });
}, 1000);

Then get the PID (e.g. 1234) of your node process and run sudo lsof -p 1234 | wc -l a few times. You'll see the number only goes up.

[manast]

Normally, as with any other resource, not just Bull queues, you should free them when you are done using them, in our case you need to call to close. Reusing the same queue instance as you propose is also a valid way, but I am not sure is this is something that needs to be stated as something special with this library or just programming with node in general?

[chrisfsampaio]

Thanks for the response! I guess you're right, I don't have too much experience with Node so it didn't cross my mind I had to do that. If it's not a commonly reported problem, having a documentation section for it is probably too much.
In any case, thanks for the great work! It's powering up a very important part of our app.