Node: `MaxListenersExceededWarning` with `Stream.pipeline()`

[ehmicky]

• Version: v12.4.0
• Platform: Linux my-laptop 5.0.0-19-generic #20-Ubuntu SMP Wed Jun 19 17:04:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem: stream

index.js:

const { Readable, Writable, pipeline } = require('stream')
const readable = new Readable()
for (let i = 0; i <= 5; ++i) {
    pipeline(readable, new Writable(), () => {})
}

$ node index.js
(node:19205) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Readable]. Use emitter.setMaxListeners() to increase limit
(node:19205) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 end listeners added to [Readable]. Use emitter.setMaxListeners() to increase limit

This happens even when pipeline() is called after readable has ended and been destroyed. Once a stream has ended, its pipeline() callback should be called then its listeners should be cleaned up. However this is not the case at the moment: eos() returns a cleanup function but this function is never called.

[Himself65]

C:\Users\Himself65\Desktop\github\test>nvm use 12.16.1
Now using node v12.16.1 (64-bit)

C:\Users\Himself65\Desktop\github\test>node bundle.js
(node:20800) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Readable]. Use emitter.setMaxListeners() to incr
ease limit
(node:20800) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 end listeners added to [Readable]. Use emitter.setMaxListeners() to increa
se limit

C:\Users\Himself65\Desktop\github\test>nvm use 13.10.1
Now using node v13.10.1 (64-bit)

C:\Users\Himself65\Desktop\github\test>node bundle.js
(node:21880) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 end listeners added to [Readable]. Use emitter.setMaxListeners() to increa
se limit

[ronag]

I believe this was fixed in d016b9d70897b7702e7862252d768ecdde89bc48 which is labeled as semver-major and as it looks now won't make it into 12.x.

[ehmicky]

Thanks for the update!

Trying with 13.10.1 (which includes d016b9d) and getting the same result: there is still one MaxListenersExceededWarning. Shouldn't there be none?

[ronag]

Shouldn't there be none?

I don't think so. Remember there are nextTicks involved here. You are registering lots of new listeners in the same tick before they are removed (next tick).

If you add a process.nextTick in between it should be better.

[ronag]

Hm, maybe you are right, it doesn't unregister the event listeners in this case.

[ronag]

@ehmicky: The docs say:

stream.pipeline() leaves dangling event listeners on the streams after the callback has been invoked. In the case of reuse of streams after failure, this can cause event listener leaks and swallowed errors.

[ronag]

This can partly be fixed. However, it still needs to leave an 'error' listener on the streams (which kind of defeats any other fix) in order to not break lots of other stuff.

[ehmicky]

Thanks. If I understand you correctly, this is intended and this issue can be closed?

[ronag]

Thanks. If I understand you correctly, this is intended and this issue can be closed?

Yes, it's an expected behavior.