Grpc-node: MaxListenersExceededWarning: Possible EventEmitter memory leak detected. error

[apatora]

Problem description

I am using the package @google-cloud/bigtable which relies on this one and when I ran npm install today it updated @grpc/grpc-js from 0.5.2 to 0.5.3. I am getting the following error :

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit
    at _addListener (events.js:256:17)
    at TLSSocket.addListener (events.js:272:10)
    at TLSSocket.Readable.on (_stream_readable.js:826:35)
    at Http2CallStream.attachHttp2Stream (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call-stream.js:275:35)
    at Http2SubChannel.startCallStream (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/subchannel.js:110:20)
    at finalMetadata.then.metadataValue (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/channel.js:205:28)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Reproduction steps

Run npm i @google-cloud/bigtable in a project. Then try to execute for example :

const Bigtable = require('@google-cloud/bigtable')

for (let j = 0; j < 100; j++) {
    Bigtable.instance('INSTANCE').table('TABLE').insert([{ key: 'key', data: { data: { some: 'data' } } }]
}

Environment

I am running the code in the following docker image : node:10.16.0

[murgatroid99]

I think this is the result of #1021. I'll need to figure out a better way to do that.

[ddehghan]

also happens on node:12.10.0-alpine

 MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [TLSSocket]. Use emitter.setMaxListeners() to increase limit
    at _addListener (events.js:267:17)
    at TLSSocket.addListener (events.js:283:10)
    at TLSSocket.Readable.on (_stream_readable.js:863:35)
    at Http2CallStream.attachHttp2Stream (/var/www/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:275:35)
    at Http2SubChannel.startCallStream (/var/www/app/node_modules/@grpc/grpc-js/build/src/subchannel.js:110:20)
    at /var/www/app/node_modules/@grpc/grpc-js/build/src/channel.js:205:28

[ddehghan]

for now i rolled back to ver "0.5.2" and it fixed the problem.

[yatingoswami]

Also facing the same issue on 12.4.0-stretch
Getting pod restarts every 4 hours.

Cant rollback version as its part of google-gax which is nested inside @google-cloud/pubsub.
Is there a proper solution for this

[apatora]

@yatingoswami You can solve the issue for now by putting :

"resolutions": {
  "@grpc/grpc-js": "0.5.2"
}

in your package.json file. Then you execute npx npm-force-resolutions and npm i.

It seems that the issue is affecting a lot of packages. Maybe it would be better to rollback #1021 until it is fully resolved.

[nicolasnoble]

How is a warning impeding production usage exactly?

[yatingoswami]

"resolutions": { "@grpc/grpc-js": "0.5.2" }
This solved it for me.
Thanks

[merlinnot]

How is a warning impeding production usage exactly?

There's a massive memory leak. This warning is accurate. I observed consumption of ~1GB/h/instance across 200+ Google Cloud Functions (1k+ instances), causing so many OOMs that the retry mechanism gave up.

This is a severe issue and I highly recommend downgrading to 0.5.2, which does not have this leak.

[murgatroid99]

In the past we have had various reports of this specific warning in other code paths, and in general it has not been indicative of memory leaks, but simply that there are more than 11 calls open simultaneously in a single channel. Because the warning was the only thing reported here, I assumed that that was the only user-facing symptom. Now that I know that there is a memory leak, I can investigate that and deploy a fix.

[merlinnot]

Thanks, I thought it's being worked on and was just patiently waiting :)

It should be very easy to reproduce, but just in case you can't, I can provide a repro (although there are some code samples in this thread already).

[murgatroid99]

grpc-js version 0.5.4 is now out with the fix for this.

[jrthib]

@murgatroid99 looks like the build is failing

[murgatroid99]

If you're referring to the version bump build, that's just because I merged it quickly.

[merlinnot]

I can confirm that there's no memory leak anymore.

[kirillgroshkov]

We still have these errors in production (screenshot from today attached). We're using Datastore node library.

yarn list @grpc/grpc-js                                                            01:37 Dur
yarn list v1.17.3
warning Filtering by arguments is deprecated. Please use the pattern option instead.
└─ @grpc/[email protected]
✨  Done in 0.74s.

[murgatroid99]

What is the call stack for those errors? I removed the line of code that caused the original errors; it doesn't exist in version 0.5.4.

[jakeleventhal]

I am on 5.2 as per recommended and am seeing a MASSIVE memory leak from something different that may be related. https://github.com/googleapis/nodejs-firestore/issues/768

[apatora]

I tried the last version of this module and got the following error after a while :

Error: No connection established
    at Http2CallStream.call.on (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
    at Http2CallStream.emit (events.js:203:15)
    at Http2CallStream.EventEmitter.emit (domain.js:448:20)
    at process.nextTick (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

[murgatroid99]

@apatora that looks like an entirely separate error. I suggest filing a separate issue with more details.

[nekdev]

I tried the last version of this module and got the following error after a while :

Error: No connection established
    at Http2CallStream.call.on (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
    at Http2CallStream.emit (events.js:203:15)
    at Http2CallStream.EventEmitter.emit (domain.js:448:20)
    at process.nextTick (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

@apatora I'm facing the same error in Google Cloud Functions. Did you find any workaround?

[apatora]

@nekdev You should check https://github.com/grpc/grpc-node/issues/1064. For now the only way to circumvent this error is to force (in package.json) :

"resolutions": {
    "@grpc/grpc-js": "0.5.2",
    "google-gax": "1.6.2"
}

I am closing this issue as it seems the MaxListenersExceededWarning is gone in the newest versions of grpc-js.

[nekdev]

All problems solved after updating to:

    "firebase-admin": "^8.6.0",
    "firebase-functions": "^3.2.0",