Bull: MaxListenersExceededWarning when creating a few queues reusing Redis connection

[coox]

Description

Creating a few queues reusing Redis connections, using just the sample code provided in the patterns documentation, triggers a MaxListenersExceededWarning:

• for error event listeners after 5 queues
• for ready event listeners after 10 queues

One of the culprits seems to come from the code in this section:
https://github.com/OptimalBits/bull/blob/ee88d25/lib/queue.js#L146
where the error event handler is unconditionally added to the queue (regardless of whether redisClientGetter returns a fresh or a re-used instance), and is never removed.

Minimal, Working Test code to reproduce the issue.

import Bull from 'bull';
import Redis from 'ioredis';

// Reusing Redis Connections, code borrowed from patterns documentation:
// https://github.com/OptimalBits/bull/blob/master/PATTERNS.md#reusing-redis-connections

const sharedRedisClientConnection = new Redis(process.env.REDIS_URL);
const sharedRedisSubscriberConnection = new Redis(process.env.REDIS_URL);

console.log('initial listener count', {
  error: sharedRedisClientConnection.listenerCount('error'),
  ready: sharedRedisClientConnection.listenerCount('ready'),
});

const options = {
  createClient: type => {
    switch (type) {
      case 'client':
        return sharedRedisClientConnection;
      case 'subscriber':
        return sharedRedisSubscriberConnection;
      default:
        return new Redis(process.env.REDIS_URL);
    }
  },
};

// Creating a few queues with shared Redis connections
// triggers a MaxListenersExceededWarning:
// - for "error" listeners after 5 queues
// - for "ready" listeners after 10 queues

const queues = [];

for (let index = 1; index <= 10; index++) {
  console.log(`creating queue #${index}…`);
  queues.push(new Bull(index, options));

  console.log('updated listener count', {
    error: sharedRedisClientConnection.listenerCount('error'),
    ready: sharedRedisClientConnection.listenerCount('ready'),
  });
}

Output:

initial listener count { error: 0, ready: 0 }
creating queue #1…
updated listener count { error: 2, ready: 1 }
creating queue #2…
updated listener count { error: 4, ready: 2 }
creating queue #3…
updated listener count { error: 6, ready: 3 }
creating queue #4…
updated listener count { error: 8, ready: 4 }
creating queue #5…
updated listener count { error: 10, ready: 5 }
creating queue #6…
updated listener count { error: 12, ready: 6 }
creating queue #7…
updated listener count { error: 14, ready: 7 }
creating queue #8…
updated listener count { error: 16, ready: 8 }
creating queue #9…
updated listener count { error: 18, ready: 9 }
creating queue #10…
updated listener count { error: 20, ready: 10 }
(node:45293) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit
(node:45293) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 ready listeners added. Use emitter.setMaxListeners() to increase limit

Bull version

Could reproduce with 3.6.0.

[cnaccio]

Having the same issue. Is this actually a problem we should be concerned with? We're experiencing some other issues, and I'm starting to wonder if this is related.

_P.S. We're creating thousands of queues at times, and so this could be more of an issue for us._

[manast]

this is just the way nodejs works https://nodejs.org/api/events.html#events_eventemitter_defaultmaxlisteners

[coox]

@manast —

I agree with you that the warning itself is “just the way nodejs works”, and that answers @cnaccio’s question.

However, do you consider Bull’s behavior of unconditionally stacking up more and more event listeners to a shared connection to “work as designed”?

I would argue that this could be considered a leak of event listeners, and kind of partially ruins the benefit of sharing connections between queues. This is why I opened the issue in the first place (sorry if that was not clear).

Do you agree with this? Am I missing something here? I would appreciate you considering re-opening this issue to track discussion, and potential resolution, of this behavior.

[loris]

@manast quick question, related to the issue raised above since it looks the warnings are only triggered when "sharing" redis connections: what are the pros & cons (throughput, memory usage, etc) of sharing connections, considering our redis server do not have any restrictions on connection count.
Thanks!

[manast]

@loris performance wise it is probably better to not reuse connections since I expect better parallelism. Of course every connection also consumes more memory, but being within reasonable limits it should not be any problem.