Rocket.chat: Deleting a user doesn't delete rocketchat_subscription's for DM's

[12test12]

Rocket.Chat Version: 0.54.2
Running Instances: 1
DB Replicaset OpLog: Disabled
Node Version: 4.8.1

I had the same problem like #1149.

I followed kalebwalton solution step to solve the problem.

use mydb
db.users.find(
{"username":"kwalton"}
) <-- just to make sure the user existed
db.users.remove(
{"username":"kwalton"}
) <-- deletes the user
Then I re-logged in to Rocket.Chat and it worked. But I ran into another issue where I couldn't direct message someone, so I went back into mongo and:
use mydb
db.rocketchat_subscription.find({"name":"kwalton"}) <-- copied the rid value for the person and my conversation
db.rocketchat_subscription.remove({"name":"kwalton"}) <-- deletes my subscriptions
db.rocketchat_room.find({"_id":""}) <-- just to make sure the room exists
db.rocketchat_room.remove({"_id":""}) <-- remove the room
Also found any other rooms that matched the other person and I:
db.rocketchat_room.find({"usernames" : [ "theotherperson", "kwalton" ]})
db.rocketchat_room.remove({"usernames" : [ "theotherperson", "kwalton" ]}) <-- removed all rooms between us, appeared to be a dupe in there causing trouble

But now I have an other problem.
Now it says "No user with username was found!" when trying to access the direct message.

[willihelm]

need this to be fixed

[willihelm]

Fixed this by deleting the subscriptions of the effected users in mongoDB. After that I had to restart the server and the users had to relogin.

[Reamer]

How looks your mongodb-Query?

[willihelm]

db.getCollection('rocketchat_subscription').find({ "name": "username1", $and: [ { "u.username": "username2" } ] })
db.getCollection('rocketchat_subscription').find({ "name": "username2", $and: [ { "u.username": "username1" } ] })

[x-drum]

Bug is still here, i just observed a couple users with this issue, it's really time consuming for the server admin

[ellisgl]

0.56.0: Issue is there. Discovered it when trying to DM.

[ellisgl]

So I was playing with on my local VM and it seems that it works fine there, but there is a difference between my local VM and the bigger test bed, is that mine complains about how I don't have my MongoDB on ReplicaSet mode.

Error: Rocket.Chat requires oplog tailing when running in multiple instances
Please make sure your MongoDB is on ReplicaSet mode and MONGO_OPLOG_URL environment variable is defined correctly on the application server

Wonder if having it in ReplicaSet mode could be causing it not to work?

Well the first post said he didn't have ReplicaSet on.. So that that's probably not it. I did see a bunch of stuff about caching around the queries.

[cturcich-catasys]

Same issue with deleted/recreated user.

[fmuecke]

Problem still exists with v0.57. :(

The provided solution is just a nasty workaround - shouldn't the problem still be fixed?
I am not sure how to apply this at all. Is there some other way that does not involve messing with the datastore?

Exception while invoking method 'createDirectMessage' MongoError: 
E11000 duplicate key error collection: parties.rocketchat_subscription index: u._id_1_name_1_t_1_code_1 dup key:
 { : "RWmMv7vkye3Kb7Ao8", : "fmuecke", : "d", : null }   
at Object.Future.wait (/snap/rocketchat-server/829/node_modules/fibers/future.js:449:15)

[kiamgrebgnur]

Hi,

I removed the user account, every subscription and then restarted everything.
Same problem. I can't dm the user. Anything else I can do?

I'm using 0.57.2

[cturcich-catasys]

Any progress on this?

[jhonatanmorais]

Same problem here :\

Any progress on this?

[nitinbhadauria]

Same Here, below error on 0.58.0 on Docker with Oplog enabled:

Exception while invoking method 'createDirectMessage' MongoError: E11000 duplicate key error collection: rocketchat.rocketchat_subscription index: u._id_1_name_1_t_1_code_1 dup key: { : "GbFCZzXFtPbp9JEA8", : "XXXXX", : "d", : null }
``` at Object.Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:449:15)
at [object Object]. (packages/meteor.js:213:24)
at [object Object].MongoConnection.(anonymous function) [as update] (/app/bundle/programs/server/packages/mongo.js:835:49)
at Object. (packages/matb33_collection-hooks.js:504:27)
at Object.collection.(anonymous function) [as update] (packages/matb33_collection-hooks.js:146:21)
at [object Object].update (/app/bundle/programs/server/packages/mongo.js:3917:31)
at ModelsBaseDb.update (/app/bundle/programs/server/packages/rocketchat_lib.js:10914:32)
at ModelsBaseDb.upsert (/app/bundle/programs/server/packages/rocketchat_lib.js:10989:16)
at ModelSubscriptions.upsert (/app/bundle/programs/server/packages/rocketchat_lib.js:4486:36)
at [object Object].Meteor.methods.createDirectMessage (/app/bundle/programs/server/app/app.js:6585:35)
at [object Object].methodMap.(anonymous function) (packages/rocketchat_monitoring.js:2731:30)
at [object Object].methodsMap.(anonymous function) (/app/bundle/programs/server/packages/rocketchat_lib.js:1110:26)
at maybeAuditArgumentChecks (/app/bundle/programs/server/packages/ddp-server.js:1858:12)
at /app/bundle/programs/server/packages/ddp-server.js:904:20
at [object Object]._.extend.withValue (packages/meteor.js:1126:17)
at /app/bundle/programs/server/packages/ddp-server.js:903:47
at [object Object]._.extend.withValue (packages/meteor.js:1126:17)
at /app/bundle/programs/server/packages/ddp-server.js:902:46
at [object Object]._.extend.protocol_handlers.method (/app/bundle/programs/server/packages/ddp-server.js:875:21)
at /app/bundle/programs/server/packages/ddp-server.js:754:85

- - - - -

at Function.MongoError.create (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:31:11)
at toError (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:114:22)
at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:1049:67
at /app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:455:18
at nextTickCallbackWith0Args (node.js:489:9)
at process._tickDomainCallback (node.js:459:13)```

[bsubasic]

Same problem here, a user has been deleted and then created again using custom Oauth (IdentityServer) login implementation.
Direct chat to this username throws "username not found".

[34mI20170920-10:01:21.730(0) Exception while invoking method 'createDirectMessage' MongoError: E11000 duplicate key error index: rocketchat.rocketchat_subscription.$u._id_1_name_1_t_1_code_1 dup key: { : "W9TGtB4KCYxzfj3hR", : "Malou.Runnemo", : "d", : null } at Object.Future.wait (/opt/Rocket.Chat/programs/server/node_modules/fibers/future.js:449:15) at [object Object].<anonymous> (packages/meteor.js:213:24) at [object Object].MongoConnection.(anonymous function) [as update] (/opt/Rocket.Chat/programs/server/packages/mongo.js:836:49) at Object.<anonymous> (packages/matb33_collection-hooks.js:504:27) at Object.collection.(anonymous function) [as update] (packages/matb33_collection-hooks.js:146:21) at [object Object].update (/opt/Rocket.Chat/programs/server/packages/mongo.js:3899:31) at ModelsBaseDb.update (/opt/Rocket.Chat/programs/server/packages/rocketchat_lib.js:10647:32) at ModelsBaseDb.upsert (/opt/Rocket.Chat/programs/server/packages/rocketchat_lib.js:10722:16) at ModelSubscriptions.upsert (/opt/Rocket.Chat/programs/server/packages/rocketchat_lib.js:4283:36) at [object Object].Meteor.methods.createDirectMessage (/opt/Rocket.Chat/programs/server/app/app.js:6506:35) at [object Object].methodMap.(anonymous function) (packages/rocketchat_monitoring.js:2731:30) at [object Object].methodsMap.(anonymous function) (/opt/Rocket.Chat/programs/server/packages/rocketchat_lib.js:1028:26) at maybeAuditArgumentChecks (/opt/Rocket.Chat/programs/server/packages/ddp-server.js:1823:12) at /opt/Rocket.Chat/programs/server/packages/ddp-server.js:905:20 at [object Object]._.extend.withValue (packages/meteor.js:1122:17) at /opt/Rocket.Chat/programs/server/packages/ddp-server.js:904:41 at [object Object]._.extend.withValue (packages/meteor.js:1122:17) at /opt/Rocket.Chat/programs/server/packages/ddp-server.js:903:46 at [object Object]._.extend.protocol_handlers.method (/opt/Rocket.Chat/programs/server/packages/ddp-server.js:876:21) at /opt/Rocket.Chat/programs/server/packages/ddp-server.js:755:85 - - - - - at Function.MongoError.create (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:31:11) at toError (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:114:22) at /opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:1049:67 at /opt/Rocket.Chat/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:455:18 at nextTickCallbackWith0Args (node.js:420:9) at process._tickDomainCallback (node.js:390:13)

[metalcated]

I am seeing the same issue as well. Thanks.

[SeverinDK]

Is anything gonna be done to solve this? I've had a conversation that I'm unable to delete/hide, for months now, because before the user was deleted, he sent me one message. That means that I am now stuck with "steve.houston [1]" in my list of Direct Messages. There is no way for me, through the UI, to hide that room, because the "hide room" button is where the [1] currently is. It also means I ALWAYS have a [1] on my favicon on the tabs.

[cello86]

Present the issue on version 0.59.3. We have recreated a user and we noticed this issue.

[maxdwit]

@rodrigok Maybe a cleanup/verify database script would be helpfull?

[graywolf336]

@rodrigok bringing this to your attention as there seems to be a continued issue and I've seen talk in the #dev channel on open.rocket.chat about this issue.

[himpich]

FYI: you can manually delete the affected subscription entries from the mongodb, e.g. (no guarantees):

start the mongo shell (e.g. enter your docker-mongo container and execute mongo)

• use the rocketchat db
  use rocketchat
• find entries for the given user:
  db.rocketchat_subscription.find( { "name" : "USERNAME_OF_THE_DELETED_USER" } )
• if you are satisfied with your findings, delete them:
  db.rocketchat_subscription.deleteMany( { "name" : "USERNAME_OF_THE_DELETED_USER" } )

Edit: Unfortunatly it does not work with rocketchat clients, they seem to access another dead database entry :(
Edit2: completly deleting the rocketchat+ client directory (with all local databases, etc) makes the RC+ client work again with formerly broken users

Hopefully the deletion of subscriptions will be fixed within rocketchat, as well as a 'sanitize database' function added :)

[fkoehler]

If you have applied the deletion of users above you can find the default directories which need to be deleted for the clients here:

• Linux: ~/.config/Rocket.Chat+
• macOS: ~/Library/Application Support/Rocket.Chat+

[rodrigok]

Folks, I wasn't able to reproduce the problem, could you let me know the steps?

Are you doing this via our web interface?

Are you using which version?

Thanks

[himpich]

I deleted (ldap) users via the admin ui. Versions ranged up to 'latest version 3 month ago'. I don't know if the error persists afterwards.

[metalcated]

I deleted a non-ldap user (because resetting 2fa as an admin for a user
does not exist and should) and he registered back with the same name which
now caused folks who try to DM him unable to do so saying user does not
exist.

On Wed, Jan 17, 2018, 7:36 AM himpich [e-mail] wrote:

I deleted (ldap) users via the admin ui. Versions ranged up to 'latest
version 3 month ago'. I don't know if the error persists afterwards.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/RocketChat/Rocket.Chat/issues/6535#issuecomment-358291420,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUYCtPt-pm0GPWJMvd8u4Y59kfNOe7oks5tLelcgaJpZM4MuW0E
.

[rodrigok]

@metalcated Can you let me know the steps and version you are using? I tested this 10 times here and all the times I got the correct behavior deleting the subscriptions and rooms and been able to recreate the user and open DMs with the same users.

[metalcated]

Did you enable 2fa for the user? And also make sure to create a backlog of
message exchange between yourself and the test user. As for version, it's
been since 5x.x to current. I am not 100% certain on what the version was
at the time, but I do the issue still persists for this one user.

On Wed, Jan 17, 2018, 7:49 AM Rodrigo Nascimento [e-mail]
wrote:

@metalcated https://github.com/metalcated Can you let me know the steps
and version you are using? I tested this 10 times here and all the times I
got the correct behavior deleting the subscriptions and rooms and been able
to recreate the user and open DMs with the same users.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/RocketChat/Rocket.Chat/issues/6535#issuecomment-358294345,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUYCiE2X1EctHgWgV-97VkmiUtOMQWtks5tLexvgaJpZM4MuW0E
.

[rodrigok]

Ok @metalcated I found the problem, thanks.

PS: It's not related to the 2fa

[metalcated]

Excellent. Thanks!

On Wed, Jan 17, 2018, 8:27 AM Rodrigo Nascimento [e-mail]
wrote:

Ok @metalcated https://github.com/metalcated I found the problem,
thanks.

PS: It's not related to the 2fa

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/RocketChat/Rocket.Chat/issues/6535#issuecomment-358303780,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUYCqQce9yvD7to0EITuXiEcItdyN0sks5tLfVHgaJpZM4MuW0E
.

[maxdwit]

@rodrigok Will this fix existing defective users records?

[TwizzyDizzy]

@rocket-cat open

@rodrigok if you can, please do reply to @maxdwit:

Will this fix existing defective users records?

I think this question is rather important, because if this was not the case one would have to manually fix this and we should provide an "official" solution.

Cheers
Thomas

[maxdwit]

@TwizzyDizzy @rodrigok

Testet with 0.61 I still have a broken User that can't be direct chatted to if you had a previous direct convo open before the user was deleted.

[willihelm]

Have you tried to restart the server and then log out and back in with your user? This is what you have to do if you remove the subscriptions manually.

[maxdwit]

Well, the user logged in / out several times over several versions.
Just verified with 0.62.1
User is still broken.
Would be helpfull to now how I can remove the User contrains @rodrigok or have a script that fixes the user ID.

[TomaszDom]

Any progress on this? I'm in the same boat.

[ArchimedesFM]

The problem still occurs on 0.62.2
The user schmidt.katha was deleted 3 month ago.

I20180329-14:46:16.932(2) Empty Room for Subscription { _id: 'riw2zx3fZBDPDyXBE', rid: 'rv2SY6R78kQK5p9PftBtnrt5pk4MQnKnA3', ts: 2017-05-22T09:20:24.288Z, ls: 2017-05-22T09:23:14.485Z, open: false, _updatedAt: 2018-03-29T12:17:40.868Z, name: 'schmidt.katha', t: 'd', alert: false, unread: 0, u: { _id: 'tBtnrt5pk4MQnKnA3'

[dbluxo]

@rodrigok same here with version 0.63.0

As @maxdwit mentioned:
"I still have a broken User that can't be direct chatted to if you had a previous direct convo open before the user was deleted."

We also have a couple of users that have leaved our project (and were deleted) and after coming back to our project (we added those users again) now can't be direct chatted to from users that had a previous direct conversation open to those deleted useres. Error message:

"No user with username "username" was found!"

[jhonatanmorais]

I still have this problem. I've just updated to 0.63.1.

Meteor ➔ method createDirectMessage -> userId: MSZDNCva67DFWRm7h , arguments: { '0': 'jean' } Exception while invoking method 'createDirectMessage' MongoError: E11000 duplicate key error index: rocketchat.rocketchat_subscription.$u._id_1_name_1_t_1_code_1 dup key: { : "MSZDNCva67DFWRm7h", : "jean", : "d", : null } at Function.MongoError.create (/opt/rocketchat/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/error.js:31:11) at toError (/opt/rocketchat/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/utils.js:139:22) at /opt/rocketchat/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/collection.js:1059:67 at /opt/rocketchat/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb-core/lib/connection/pool.js:469:18 at _combinedTickCallback (internal/process/next_tick.js:131:7) at process._tickDomainCallback (internal/process/next_tick.js:218:9)

[kaji-bikash]

Same observation as @jhonatanmorais , @dbluxo and @ArchimedesFM

Need to find solution for this. Our logs are filled with tens of thousands of these. Team, let us know if you need more verbose logs or scenario clarifications.

20180419-15:11:02.958(0) Empty Room for Subscription { _id: 'fmYDMu4d9PKnmHwvL', rid: 'FpyuB2N6rFZgnmDdEQcTZKa7PYArC6yQ9G', ts: 2017-05-02T09:33:49.404Z, ls: 2017-08-29T06:41:31.894Z, open: true, _updatedAt: 2017-10-31T18:17:24.591Z, name: 'REDACTED', t: 'd', alert: false, unread: 0, u: { _id: 'QcTZKa7PYArC6yQ9G', username: 'REDACTED' }, archived: true, meta: { revision: 0, created: 1523799229453, version: 0 }, '': 11103, _room: undefined, _user: { _id: 'QcTZKa7PYArC6yQ9G', createdAt: 2016-09-16T06:00:14.471Z, services: { password: [Object], resume: [Object] }, emails: [ [Object] ], type: 'user', status: 'away', active: true, roles: [ 'user' ], _updatedAt: 2018-04-19T15:10:59.626Z, username: 'victor_migabo', utcOffset: 3, name: 'REDACTED', importIds: [ 'U0Y3EL8B1' ], lastLogin: 2018-04-19T15:10:59.605Z, statusConnection: 'online', statusDefault: 'away', avatarOrigin: 'gravatar', language: 'en', settings: { preferences: [Object] }, meta: { revision: 263, created: 1523799227334, version: 0, updated: 1524150659631 }, '': 74 }, fname: undefined }

[mrsimpson]

For those in control of the database, see here

@geekgonecrazy You seem to have had a similar issue earlier.
In order to improve performance, I implemented it with aggregate, see here.
Feel free to copy to a new migration...

[geekgonecrazy]

Yes this seems to be becoming more and more of an issue.

@rocketchat/core what do you guys think? This seems to be an issue for a lot of people still. I had to help someone fix just today. I think we've fixed new cases of this happening... But I think we have a lot of broken still that needs cleaned up

[TwizzyDizzy]

Hi Aaron

Yes this seems to be becoming more and more of an issue.

Nope, this has been an issue all that time... it's just that nobody cared ;)

I think we've fixed new cases of this happening.

Is there proof? My feeling is the same. Though I think there may still be cases that cause this in the LDAP/User sync or import contexts...

But I think we have a lot of broken still that needs cleaned up

Most definately. I think it might make sense to go about it this way:

• create an official solution in terms of "this is how you fix this problem"
• close all issues that involve this error with a reference to that solution
• see what comes back
• look out for future cases of this (just in case it hasn't been fixed in every context)

What do you think?

Cheers
Thomas

[sampaiodiego]

what do you guys think is a good solution for this?

• delete all subscriptions of the removed user? doing this other people will not be able to see old messages with that user.. so, we don't even need to keep messages from that subscription's room
• allowing users to still see messages of the removed user but showing somewhere the user was removed?

1st solution is the easiest from a development perspective but I can see how bad it is from an UX perspective. it would also needs a migration to clean up the database.

2nd solution might be hard to implement (but not that much) and is very user friendly IMO, no migration is needed since it is mostly front-end work.

[kaiiiiiiiii]

2nd solution might be hard to implement (but not that much) and is very user friendly IMO, no migration is needed since it is mostly front-end work.

Wasn’t that functionality already implemented with PR #9947?

Keep (don't remove any message)

[sampaiodiego]

@kaiiiiiiiii partially.. that PR will remove deleted user's messages, but this issue will still happen since the subscription record will not be deleted.

[mrsimpson]

@sampaiodiego I didn't get what you objected with your first idea:

delete all subscriptions of the removed user? doing this other people will not be able to see old messages with that user.. so, we don't even need to keep messages from that subscription's room

From what I understood, there are corrupt subscriptions: the user subscribed does not exist anymore, e. g. since he's been renamed. Removing them will not result in any loss of data, since the subscription was anyway not applicable for the renamed user. And messages are not deleted either when purging the subscriptions from the database. It's not about deleting the user, is it?

I already implemented a "migration" in our fork. It solved our issue without negative side-effects (at least not that I've noticed)

https://github.com/assistify/Rocket.Chat/blob/f7dd9a6a5bbb82da0306e5334dff451ebaf1ba9c/packages/assistify-help-request/server/migrations.js#L7-L31

[maxdwit]

Just a quick note here. If you really are in need to have a user repaired, based on the commands provided by himpich above.
This is what I did to fix ONE deleted User to talk to ANOTHER existing User:

Soo. What I did to fix the subscription between two users. One of them was deleted. The other had problems to subscribe.
Get a mongo 3.2 shell client and connect it to your mongo db.

From above:
Replace parties with rocketchat or whatever database you use. I have snapd, so I go with parties:
use parties
Find the User that has been deleted
db.rocketchat_subscription.find( { "name" : "THEDELETEDUSERNAME" } )
Now look carefully for "THE_OTHER_USERNAME" the one who has issues opening any direct talk to THEDELETEDUSERNAME
Now find the subscription where both are in the same line:
In my case I had both Users in the _id uZCHxENps5EtG5SYR
To verify if I would only hit one Record I ran find again:
db.rocketchat_subscription.find( { "_id" : "uZCHxENps5EtG5SYR" } )
Now I kicked it:
db.rocketchat_subscription.deleteMany( { "_id" : "uZCHxENps5EtG5SYR" } )

Relogged both users.
Invite works again.

Lucky I had only one User that was defective. I like @sampaiodiego approach to fix this for everyone.

[engelgabriel]

delete all subscriptions of the removed user? doing this other people will not be able to see old messages with that user.. so, we don't even need to keep messages from that subscription's room

@sampaiodiego are you sure about this?

@Hudell can we implement a DB migration like https://github.com/assistify/Rocket.Chat/blob/f7dd9a6a5bbb82da0306e5334dff451ebaf1ba9c/packages/assistify-help-request/server/migrations.js#L7-L31 ?

[sampaiodiego]

@engelgabriel yes! if we decide to delete all removed user's subscriptions, then we can remove also its messages and room object..

[mrsimpson]

@sampaiodiego I don’t know whether this is relevant, but we had the issue after renaming a user.
I was not 100% confident that all models are affected in the same way as subscriptions.
Also, stale messages of a removed user could beactually desired (not in the standard, but in a custom fork).
In General, I consider all these cleanup activities some kind of data curation. If there was an option in the admin-Ui with the various activities to be performed (like deleting stale messages), this might be an approach. But you are the product owner, so also adding it as a migration and adding a bold breaking in the release notes might do.

[Hudell]

I created the migration (#10700) and also changed the user deletion method because there were a few subscriptions and empty rooms being left behind.

[sampaiodiego]

we have changed again the process of deleting a user again for 3.2.0 release, so I'll be closing this as it not be a problem anymore.