Rocket.chat: MaxListenersExceededWarning after upgrade to 3.6.3

Created on 30 Sep 2020 · 26Comments · Source: RocketChat/Rocket.Chat

Description:

Since upgrade to 3.6.3 I see in logs a lot of (node:[some number]) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 wakeup listeners added to [Connection]. Use emitter.setMaxListeners() to increase limit
What is about and how to fix that?

Steps to reproduce:

Upgrade to 3.6.3
Get errors in logs

Expected behavior:

Clean of errors and warnings logs

Actual behavior:

All seem to work fine, but I'm concern about that warnings

Server Setup Information:

Version of Rocket.Chat Server: 3.6.3
Operating System: CentOS7
Deployment Method: docker
Number of Running Instances: 25
DB Replicaset Oplog: Enabled
NodeJS Version: 12.16.1
MongoDB Version: 4.0.17

Client Setup Information

Desktop App or Browser Version: 2.17.7
Operating System: Windows 10

Additional context

We have a lot of instances - 25 and maybe it's related to problem

Relevant logs:

(node:9) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 wakeup listeners added to [Connection]. Use emitter.setMaxListeners() to increase limit

Source

ankar84

👍3

Most helpful comment

FYI This issue is still occurring on 3.9.0.

(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 hgx2wuCiBea846AfB listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit

Can confirm the behavior in 3.9.1

kaiiiiiiiii on 7 Dec 2020

👍7

All 26 comments

Happens to me too.
It causes very high CPU and disk I/O to the Digitalocen Droplet bring it to unstable state, forcing manual reboot.
MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added to [process]. Use emitter.setMaxListeners() to increase limit
Rocket chat server: 3.6.0
Ubuntu Ubuntu 20.04.1 LTS
Deployment method: rocketchatctl
NodeJS Version: 12.18.3 - x64
MongoDB Version: 4.0.20

jorueda on 8 Oct 2020

Hello,
I can confirm the issue as well in version 3.7.0

Rocket.Chat Version: 3.7.0
NodeJS Version: 12.18.4 - x64
MongoDB Version: 4.0.20
MongoDB Engine: mmapv1
Deployment Method: docker
ReplicaSet OpLog: Enabled
Commit Hash: 2c82cd9cee
Commit Branch: HEAD

At some point the Chat crashes and restarts as Node runs out of memory.
Please find attached the crash log
CrashLog.txt

claures on 15 Oct 2020

Hello @claures , the crash log where did you obtain it from? What is the name of the file in the Linux machine? Regards.

jorueda on 15 Oct 2020

Hello,
I extracted that log from the Docker with compose. So I can't tell you where the chat itself saves the Logs.

claures on 15 Oct 2020

Hello, it seems that release 3.7.1 solves this problem.
https://github.com/RocketChat/Rocket.Chat/releases/tag/3.7.1

jorueda on 20 Oct 2020

Hello, it seems that release 3.7.1 solves this problem.
https://github.com/RocketChat/Rocket.Chat/releases/tag/3.7.1

Yeap, I'm about to upgrade to 3.7.1

ankar84 on 20 Oct 2020

Hello, I updated to 3.7.1 and used USE_NATIVE_OPLOG=true variable.
Unfortunately I still see the message in the logs and the instance crashes on a regular basis with the same error.

claures on 21 Oct 2020

Hello, it seems that release 3.7.1 solves this problem.
https://github.com/RocketChat/Rocket.Chat/releases/tag/3.7.1

No, I see this events all the time at 3.7.1 version now...
We don't use USE_NATIVE_OPLOG=true variable.
@sampaiodiego Diego, can you take a look at this?

ankar84 on 1 Nov 2020

you guys don't need to worry about this message.. it is just Node trying to tell us something could be wrong.. but in this case it is not.. we know there is a memory leak in rocket.chat but this specific message is not about it.

recently we've manually increased this limit to stop showing the message as you can see here:
https://github.com/RocketChat/Rocket.Chat/blob/02f6e2b2a252249ba040733ee6f5bc2639020fdf/app/models/server/models/_oplogHandle.ts#L116

sampaiodiego on 3 Nov 2020

👍3

Showing up again in 3.8.0

zenjabba on 15 Nov 2020

Also having this showing up in 3.8.0. Have not seen before.

stenrose on 17 Nov 2020

Same here in 3.8.0. Instances (3) seem to work fine though

onigoetz on 17 Nov 2020

Seeing it in 3.8.0 also, having issues with status being wrong after a while.

sschueller on 20 Nov 2020

I'm having it on 3.8.1 too, following the auto update on SNAP from this week-end.

Nono-m0le on 23 Nov 2020

👍4

Seems that this is fixed in 3.8.0 version:

Captura de pantalla de 2020-11-23 09-21-24

https://github.com/RocketChat/Rocket.Chat/releases/tag/3.8.0

jorueda on 23 Nov 2020

Happening here with 3.8.1

bellegarde-c on 25 Nov 2020

Attached are the logs of the events causing high CPU, Load and Disk I/O. All three happen at the same time. What could be causing this?

logs_rocketchat_highcpu.txt

jorueda on 27 Nov 2020

you guys don't need to worry about this message.. it is just Node trying to tell us something could be wrong.. but in this case it is not.. we know there is a memory leak in rocket.chat but this specific message is not about it.

recently we've manually increased this limit to stop showing the message as you can see here:
https://github.com/RocketChat/Rocket.Chat/blob/02f6e2b2a252249ba040733ee6f5bc2639020fdf/app/models/server/models/_oplogHandle.ts#L116

Wouldn't be interesting to set this value through ENV?

arthurd2 on 27 Nov 2020

However, the server shows high CPU, high load and high Disk I/O. See image attached.
Captura de pantalla de 2020-11-27 08-12-12

jorueda on 27 Nov 2020

FYI This issue is still occurring on 3.9.0.

(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 hgx2wuCiBea846AfB listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit

Bengrunt on 4 Dec 2020

FYI This issue is still occurring on 3.9.0.

(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 hgx2wuCiBea846AfB listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit

Can confirm the behavior in 3.9.1

kaiiiiiiiii on 7 Dec 2020

👍7

Same problem

OloBo-MSK on 11 Dec 2020

Because of this error, files larger than 1 MB are no longer sent, server version 3.9.1

DeadMozay on 14 Dec 2020

I've also been seeing these, even after upgrading to 3.9.1. I think I'm getting at least one for every active user. FWIW, using the dockerized image that rocket.chat provides via a k8s cluster.

odin $ kubectl logs -n rchat rocketchat-deploy-xxxxxxxx-xxxxx rocketchat  | grep "memory leak" | wc -l
354
odin $ kubectl logs -n rchat rocketchat-deploy-xxxxxxxx-xxxxx rocketchat  | head -30
[...]
➔ +----------------------------------------------+
➔ |                SERVER RUNNING                |
➔ +----------------------------------------------+
➔ |                                              |
➔ |  Rocket.Chat Version: 3.9.1                  |
➔ |       NodeJS Version: 12.18.4 - x64          |
➔ |      MongoDB Version: 4.0.20                 |
➔ |       MongoDB Engine: wiredTiger             |
➔ |             Platform: linux                  |
➔ |         Process Port: 3000                   |
➔ |             Site URL: https://xxxx.xxxx.xxx  |
➔ |     ReplicaSet OpLog: Enabled                |
➔ |          Commit Hash: 798241b9e5             |
➔ |        Commit Branch: HEAD                   |
➔ |                                              |
➔ +----------------------------------------------+
(node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 GENERAL listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 xxxxxxxxxxxx listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
...

eengstrom on 19 Dec 2020

Same issue with rocketchat/rocket.chat:3.9.3 image

(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 n5vsL3FeS3yPeG7nL listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 FFeypBBKGtjwiBxnN listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 dexMjXPx6eA7DEMcc listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 GKfmt6wGFzX6kbJgJ listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit

Vinutha-Garimella on 23 Dec 2020

3.10 docker image problem persists
➔ +----------------------------------------------+
➔ | SERVER RUNNING |
➔ +----------------------------------------------+
➔ | |
➔ | Rocket.Chat Version: 3.10.0 |
➔ | NodeJS Version: 12.18.4 - x64 |
➔ | MongoDB Version: 4.0.19 |
➔ | MongoDB Engine: mmapv1 |
➔ | Platform: linux |
➔ | Process Port: 3000 |
➔ | Site URL: https://rocketchat |
➔ | ReplicaSet OpLog: Enabled |
➔ | Commit Hash: 3a13cead22 |
➔ | Commit Branch: HEAD |
➔ | |
➔ +----------------------------------------------+
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 iASzq3KWMXXbMzXYe listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 adwC2t9GKwzYfYr9n listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 JE5kzqxvLyNNYdgN4 listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 8NDhzCdKpLLA7sZja listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 H8bESRNZzziNDWuXH listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 iwFgRBh5FTTAA3qfc listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 rpBqdg3kSDeaTeBq3 listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 XkPxPBChuaRFf4S2J listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 cw4z2xhFRNHxMa3Zu listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 vKyEhrqPjAufFoFMM listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 9JcaYrBZugLJA2GFB listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 QwPYXoeMfJABjETQa listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 N77NuPBWdxJM2dLgP listeners added to [Stream]. Use emitter.setMaxListeners() to increase limit
(node:8) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 dvee7bEKybYt4WNGH listeners added to [Stream]. Use emitter.setMaxListeners()