Rocket.chat: mongodb at 100% cpu for 30 secs

[sinteur]

Rocket.Chat Version: 0.55
Running Instances:
DB Replicaset OpLog: 2 mongo's in a failover
Node Version:

had an chat that was running slow every now and then until it was almost unworkable. Noticed mongo cpu was at 100% for longer and loner periods. Found these in the mongo log:

2017-04-19T22:17:15.504+0200 [conn79] command meteor.$cmd command: update { update: "users", updates: [ { q: { _id: "966ncuZr7keAvtr8N" }, u: { $addToSet: { services.resume.loginTokens: { hashedToken: "RBe0ZcN5hwjGrRlsyTmZQee6uowx1BmtwIAJ2kU/1sE=" } }, $set: { _updatedAt: new Date(1492633002405) } }, upsert: false, multi: false } ], ordered: true } ntoreturn:1 keyUpdates:0 numYields:0 reslen:95 32304ms
2017-04-19T22:18:23.334+0200 [conn94] update meteor.users query: { _id: "966ncuZr7keAvtr8N" } update: { $addToSet: { services.resume.loginTokens: { hashedToken: "m2IORoA2NtXsKaJ+Dt495WOcilJMPo1EfM4xr4Sj3OE=" } }, $set: { _updatedAt: new Date(1492633069890) } } nscanned:1 nscannedObjects:1 nMatched:1 nModified:1 keyUpdates:2 numYields:0 locks(micros) w:32682040 32681ms
2017-04-19T22:18:23.335+0200 [conn94] command meteor.$cmd command: update { update: "users", updates: [ { q: { _id: "966ncuZr7keAvtr8N" }, u: { $addToSet: { services.resume.loginTokens: { hashedToken: "m2IORoA2NtXsKaJ+Dt495WOcilJMPo1EfM4xr4Sj3OE=" } }, $set: { _updatedAt: new Date(1492633069890) } }, upsert: false, multi: false } ], ordered: true } ntoreturn:1 keyUpdates:0 numYields:0 reslen:95 32681ms

Exported the user collection to inspect it by hand. Found one user with over 16k of hashedTokens in it.

Deleted that user, added it again, added it to channels and problem gone.

I have NO idea how it is possible for a user to accumulated that many hashedTokens in the user collection, but it sure is messing stuff up when it happens.

[sinteur]

current suspicion: each login via the API leaves a token behind

[sinteur]

ok confirmed, each API login adds a hashedToken that's never removed. One of our script writers had a cron job that logged on each minute instead of re-using the received hashedToken

[bbrauns]

I have several scripts doing continous logins too. Do the login tokens not expire over time ?

[sinteur]

On 21 apr. 2017, at 10:59, bbrauns [e-mail] wrote:

I have several scripts doing continous logins too. Do the login tokens not expire over time ?

apparently not.

[chrbo]

We can also confirm that there is no handling which removes created tokens. Can anybody clarify when they can be removed?

Does the tokens expire at any time? Are all of the tokens usable or perhaps only the last?

What could be the best option to remove "old" tokens centralized?

[bbrauns]

I was able to clear the tokens in our test installation with the following cmd in the mongodb shell:
rs0:PRIMARY> db.users.update({_id: "xyz"}, { $set: { "services.resume.loginTokens": []}})
but I agree with you all that this is not a good solution :smiley:

[localguru]

The should be a scheduler to clean up tokens from users, which are offline for a longer timer.

[Munzy]

This is affecting me as well. I used an API call to login each time, as I wasn't easily able to store the login hash for extended periods.

[galrotem1993]

I had the same problem.

in my forked repository i added a cron job that every X time (every 1 hour for example) for every user i delete all the token except for the last 50 tokens

[sinteur]

Please share the script you wrote for that cron job, @galrotem1993

[thapakazi]

there there...
not sure if setting this Accounts_LoginExpiration in setting helps
https://github.com/RocketChat/Rocket.Chat/blob/21c4efa08a2854de8d2cccae3018e4bad58bbb85/server/lib/accounts.js#L1-L3

checking if that helps auto-expiry of those un-necessary login Hashed Tokens...

[barrydegraaff]

@bbrauns running rs0:PRIMARY> db.users.update({_id: "xyz"}, { $set: { "services.resume.loginTokens": []}}) will also log that user out from the app on their phone, if they use that.

I am also in a situation where I do not want to store the tokens for all my users. So having the db clutter with tokens is less than ideal.

Googling services.resume.loginTokens show a number of similar issues in apps that are build on meteor.

[fcoppolani]

I believe it is linked also to https://github.com/RocketChat/Rocket.Chat/issues/11144

[MarcosSpessatto]

According to this comment and this another one the tokens are removed. I've tested and the tokens are removed. Also I created a PR that limits the resume tokens to 50, when the limit is exceeded, older tokens are removed.