Rocket.chat: High CPU usage

[LiraNuna]

Rocket.Chat version: 0.42.0

When running Rocket.Chat on Ubuntu 14.04 and Apache with mod_proxy, CPU is always at a 100% as soon as a user is connected and stays at 100% even if the (single) user disconnects and no other users stay on the server.

Would be happy to help debug this if possible.

[lucasvanhalst]

You probably haven't enabled mongo's oplog, try enabling it and see if it helps

[LiraNuna]

@lucasvanhalst I'm not sure I understand what this means and googling the subject gets me into setting up replication with mongo. Does Rocket.Chat require replication?

[pasmon]

I think we have this same issue, difference being that we are running with CentOS 7.2 and Nginx.

Node version 6.5.0.

[pasmon]

Note that I used Node version 6.5.0 since I had issues with "fibers" module, it required v8 version 5.1 for some reason.

I wonder if its the Meteor which is causing the CPU spiking, they had issues in v1.4 but they already stated that it was fixed in later versions.

[pasmon]

Not sure if this is of any relevance, but sometimes we have this in our logs:

Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: Exception while invoking method 'updateMessage' TypeError: Cannot read property '_id' of undefined
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at packages/rocketchat:lib/server/methods/filterATAllTag.js:7:39
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at packages/rocketchat_lib/lib/callbacks.coffee:73:21
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at Array.reduce (native)
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at Object.RocketChat.callbacks.run (packages/rocketchat_lib/lib/callbacks.coffee:68:103)
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at Object.RocketChat.updateMessage (packages/rocketchat:lib/server/functions/updateMessage.js:18:33)
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at updateMessage (packages/rocketchat_lib/server/methods/updateMessage.coffee:30:14)
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at methodsMap.(anonymous function) (packages/rocketchat:lib/server/lib/debug.js:17:26)
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1711:12)
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at packages/ddp-server/livedata_server.js:711:19
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at withValue (packages/meteor/dynamics_nodejs.js:56:1)
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at packages/ddp-server/livedata_server.js:709:40
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at withValue (packages/meteor/dynamics_nodejs.js:56:1)
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at packages/ddp-server/livedata_server.js:707:46
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at Session.method (packages/ddp-server/livedata_server.js:681:23)
Oct 12 10:16:48 i-bdbb4cb7 rocketchat[15455]: at packages/ddp-server/livedata_server.js:551:43

[LiraNuna]

I do not have these exceptions, so it's probably irrelevant?
What's the status of this? I tried debugging this but couldn't get very far.

[LiraNuna]

I "solved" this issue by switching to using Rocket.Chat via docker. I'm not sure what the difference is.

[jsumners]

~sigh~ I upgraded to 0.45.0 from 0.42.0 and noticed this problem crop up. However, I had also upgraded to Node 6.9.1 at the same time. Downgrading to Node 4.6.1 has alleviated this issue. It's quite ridiculous that the current LTS release can't be used.

[pasmon]

I finally got the Docker version of our Rocket.Chat working with all our Hubot plugins and scripts, and true enough, there is no high CPU usage. Still not using oplog, as @lucasvanhalst suggested.

[bazarnew]

@engelgabriel

[bazarnew]

Hi, I'm also Mshklv one can say what's the problem?

[bazarnew]

[bazarnew]

@Sing-Li

[bazarnew]

@sampaiodiego

[Sing-Li]

For those who experienced the 100% + CPU, can you please let us know if:

1. One or more of your disk volumes are mounted using nfs
2. You have had, in the past, both Node 6 and Node 4.x installed

TIA

[jsumners]

@Sing-Li:

1. No.
2. Yes, via https://github.com/tj/n

[sebbacon]

1. No
2. Can't remember. Using wheezy-backports now, so possibly.

[pasmon]

1. No
2. Yes, first Node 4.5 is installed but as we couldn't get Rocket.Chat working with that Node 6.5 was installed with Node version management software (https://github.com/tj/n)

[LiraNuna]

1. No
2. I never had both installed but I did upgrade from 4.x to 6.x before installing rocket.chat

[jsumners]

I should point out that my install of 4.x and 6.x were not concurrent, as far as Rocket.Chat is concerned.

It is clear to me that using 6.x is the problem, and it shouldn't be.

[bazarnew]

The problem is the version 6 node js
My problem was solved by switching to version 4

[bazarnew]

@LiraNuna
@Sing-Li
@pasmon

[apachler]

Is it enough to switch to 4.5 with "n 4.5"?
Because i am facing the same problem even if switched to 4.5 with n ...

Maybe this is related to the message:
(node:61374) fs: re-evaluating native module sources is not supported. If you are using the graceful-fs module, please update it to a more recent version.

[jsumners]

@apachler use 4.6.2 (4.6.1 if that doesn't work). The graceful-fs warning is not related.

[Sing-Li]

There might be some problem with some intermediate scripts that has first
line like #!/usr/bin/env node

And it may or may not jive well with env-mangling node managers like n.
We've seen at least two cases where problems were caused by multiple
versions of node being used.

[jsumners]

1. n 6.9.1
2. Start RC
3. Watch the CPU peg out
4. Stop RC
5. n 4.6.2
6. Start RC
7. CPU is fine

The point of n is that you _don't_ have multiple versions being used. You have the single version you choose to run at the time. Thus, env node is only going to report a single version -- whichever one happens to be activated.

I blame it on the Meteor.js foundation -- https://github.com/meteor/meteor/issues/4314

[apachler]

starting RC with "n use 4.6.2 main.js 2>&1 >$LOG &" works like a charm ...

[engelgabriel]

Don't use Node 6

Don't use Node 6 until this pull request: https://github.com/meteor/meteor/pull/6923

Meteor with Node 6 is not ready right now and there is still work to be done. Per the Roadmap.md and the subject of this PR, it is a work in progress. The comment above by @benjamn about the Travis CI suite working is only part of the tests. As you can see by the CircleCI tests on this PR (a separate suite of tests entirely), they failed spectacularly. Not to say that this means there is substantial work, but there are areas that need attention.

If you'd like to try using Meteor with Node 6 there are instructions in the initial post of this PR but I really wouldn't expect things to work flawlessly. Your help testing would be appreciated, but if you choose to try it out, please embark on that adventure with a willingness to help report/debug/solve the problems you encounter.

[jsumners]

Saying "don't use the current LTS Node" is not a solution to the problem.

[rodrigok]

@jsumners The version 4 is LTS too as you can see here

https://nodesource.com/blog/essential-steps-long-term-support-for-nodejs/

[jsumners]

Yes, but it isn't the "current" one in my opinion. v4 is merely there for transition. But the point is that the product should work on _any_ LTS.

[rodrigok]

Both are current LTS. We hope Meteor will add support for v6 bafore the end of v4 LTS.

For now you should use version 4 to run rocket.chat. If you need to run other applications with other node version you can use PM2 or Docker to run multiple NodeJS versions.

That is the current solution to the problem.

[LiraNuna]

Since this is biting many people, is it possible to add this fact to the installation guide?

[engelgabriel]

@LiraNuna can you please suggest the changes with a PR to https://github.com/RocketChat/Rocket.Chat.Docs/tree/master/3.%20Installation/4.%20Manual%20Installation/Ubuntu ?

[engelgabriel]

PLEASE NOTE THAT NODEJS 6 is NOT currently supported by our framework

[jsumners]

@engelgabriel it should be. Node 6 is the current LTS.

This is why people shouldn't use Meteor.

[graywolf336]

As others on this thread have said, Node.js currently has two active LTS versions. v4 is still active with Maintenance ending on 2018-04-01. The other active one is v6 which has a Maintenance ending on 2019-04-18. So, Rocket.Chat is currently on an active LTS Node.js version and there should be no issue with that since it is still going to be maintained for over another year.

_Source_

Also, it has been mentioned in this thread that we do indeed rely on Meteor and that is the support architecture of Rocket.Chat. As such, we are going to be using whatever version of Node.js is supported by Meteor. Whether people should use Meteor or not as a result of them not always using the "latest and greatest" version of Node.js is a matter of opinion. Rocket.Chat is built on Meteor and as a result it would take far too long for us to not use it anymore that it would set back the project far too long.

I am going to lock this thread and will update it whenever someone contacts me with good information to add (I don't like doing this, but I'd rather the discussion not go in circles). We will be working on getting the documentation updated (we're always open to pull requests if you're open to helping us) and there is also the pull request #5692 which will cause Rocket.Chat not to start when running on a version which is no supported.