Bull: Memory leak when adding many jobs

[weeco]

In production I always got this error message when I was adding a lot jobs to the Bull queue:

(node:22380) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 global:completed listeners added. Use emitter.setMaxListeners() to increase limit
(node:22380) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 global:failed listeners added. Use emitter.setMaxListeners() to increase limit

I was iterating with Promise.map() using a concurrency of 100 to await the values from Bull jobs. The above error message does appear sometimes (I think only when the redis keyspace has been deleted before) in the given scenario (see code snippet below).

Here you can see the TypeScript Code snippet: https://ghostbin.com/paste/hkjvt . If you want to get the full project instead of just that code snippet let me know.

[manast]

yes, that is not a memory leak. The EventEmitter is warning that you are creating many listeners and that may be a memory leak. The listeners are inside job.finished(), since you are calling that method after adding every job. This is normal, and if you want to avoid the warning just increase the number using emitter.setMaxListeners()

[weeco]

Follow up question @manast , shouldn't we use setMaxListeners() on the emitter in lib/queue.js ? It can be set on emitter instances which should avoid this warning message for others too.

Like this: https://i.imgur.com/OjCsJKT.png

[weeco]

@manast Is there a reason we shouldn't set unlimited max listeners on bull's event emitter as shown in above screenshot?

[Atinux]

I believe a better solution will be to emit a specific event when a job is finished like global:completed:${jobId} and global:failed:${jobId} this way, only 1 event listener is created when finished is called, what do you think @manast?

BTW, great job on Bull, I use it for my project and it's impressive :raised_hands:

[manast]

@Atinux thanks for the kind words. The problem that @weeco has it is because he needs to call the method Job##finished() many times concurrently, and every such call requires two listeners: https://github.com/OptimalBits/bull/blob/master/lib/job.js#L344
So we will need to redesign how this method works so that we do not need to increase the amount of listeners per call, but I have not thought about a good solution for it just yet.

[johnjensenish]

A workaround is here https://github.com/OptimalBits/bull/issues/503#issuecomment-338212399