Rocket.chat: Users stuck online (green) since upgrade 0.65.1 -> 0.660, not being sent notifications

[antgel]

Description:

As per my post in the support forum - please visit the link for more information.

Steps to reproduce:

1. Upgrade to 0.66.0 from 0.65.1

Expected behavior:

Users are marked as away for a certain time.

Actual behavior:

Online users are permanently online (green), never going to away (orange).
It also appears that a few users are stuck online (green), it's reasonable to assume the issues are connected.

This is actually pretty serious, because these users don't get push / email notifications that they have new messages.

Server Setup Information:

• Version of Rocket.Chat Server: 0.66.0
• Operating System: Ubuntu Linux 16.04
• Deployment Method: docker
• Number of Running Instances: 1
• DB Replicaset Oplog: disabled
• NodeJS Version: 8.9.3
• MongoDB Version: whatever is in the docker image

Additional context

@geekgonecrazy answered me in the thread, but as far as I know no issue was opened, so I'm opening this.

Relevant logs:

Don't see any relevant logs,

[geekgonecrazy]

Can you check and see what your users that are stuck as online have in My Account->Preferences under user presence

[antgel]

Hi @geekgonecrazy !

It's set to False / 300 both in Admin's Default, and in my user's personal settings. Nobody has ever touched it. Could it be that a migration flipped it from True to False? If I change it to True in the Admin Default, will that fix it for all the users?

[christianreiss]

I am having the exact same issue.

[707eric]

Same issue here as well.

Was it previously enabled by default?

[707eric]

@antgel If you enable in the Admin area it will change it globally.

[geekgonecrazy]

Did you guys have any issues with any of the migrations while updating?

@RocketChat/core anyone know why this might have happened? If a migration did fail is there a fix they could manually run to fix this?

I know if you want everyone to have it you likely could just run a query to set everyones back to true.

[christianreiss]

Not that I know of. But if I set "Always notify mobile" to true everything works again (of course this annoys the sh*t out of users). So it must be something within RC and the Online state (detection).

[sampaiodiego]

removing all records from usersSessions collection might do the trick. it's safe shutting rocket.chat down and removing them.

[christianreiss]

I stopped the rocket, did a
use rocketchat db.usersSessions.deleteMany({})
and deleted 8 records; re-started the rocket again.

Edit: Removed not working comment.

[christianreiss]

Actually dropping these records solved the issue for me. Thanks!

[sampaiodiego]

@antgel please let me know it that works for you as well. thx

[707eric]

I think the issue is resolved for us once it was enabled in the admin area. Any specific reason why it defaulted to off?

[sampaiodiego]

@707eric it was changed to disable by default as a mistake when the admin option was created.

[antgel]

Seems okay for me as well, after deleting user sessions and resetting the admin setting. @sampaiodiego Maybe it's worth resetting the admin to Enable in a future migration? Surely most users prefer this to be on...

[stickboy06]

Can you make it so updates stop changing settings that an administrator sets? This has been causing me headaches for the last 6 months where I have to go through all our settings we set for the company after an update because the update changed something from True to False or removed the text that was entered. Nothing is documented to state that the settings changed either. I have been searching for a month looking at possible reasons push notifications stopped working, only to find out something I set was changed during an upgrade. Please leave configurations the way they were before an update.

[sampaiodiego]

@stickboy06 I'm sorry if something caused trouble to you. I can guarantee we do not change setting values across updates, what we do is change the default value for them, that will not change your current value if you're updating.

in very few cases, we change the value of a setting if it's still with the default value, but still not that usual.

[rodrigok]

@stickboy06 can you provide us the list of settings you think was changed and the version you was upgrading to? Then we can investigate that.
We may have a bug if that is happening, as @sampaiodiego sayid we do not changed settings that don't have the default value set.

[stickboy06]

One of the times I upgraded to version 0.63.2. This changed the Push settings for Enable Gateway from True to False and then it cleared the Gateway text box. It also changed the Accounts settings in Default User Preferences for Enable Auto Away from True to False.
Upgrading to version 0.69.1 broke our Layout in the Custom CSS, but I'm not sure if it was changed because some of the text I put in that box gets hidden. I couldn't see if anything that was hidden was changed or not. Our sidebar footer image was not aligned properly, so I had to add something similar to the following:
.root {
--sidebar-footer-height: 120px;
}
After saving and exiting Administration and then going back into Administration, that root section was not displayed in the box anymore in Custom CSS. So I'm not sure if we had anything in there or not, but the update broke the sidebar footer what worked before the update.

I took over RocketChat administration when the person who used to do it left our company. I know he told me a couple of times that settings he made had been changed on other updates as well. One of the things that he told me changed was our File Upload settings. It changed the maximum size and also the Accepted Media Types behavior changed. We ended up having to remove our list because it didn't work properly, so now we allow ALL media types. I don't know what the other settings where.

[jwilczek]

I've seeing this same issue with Rocket Chat Version 2.2.0 and desktop Version 2.16.2

[sampaiodiego]

@jwilczek if that happens you can restart all rocket.chat processes to free those stuck online users.. in the meantime, we're investigating what might cause this

[stickboy06]

We have a script that restarts all of our RocketChat services every night because of memory leaks and other weird things where the services break or slow down so much that the users have issues. Restarting every night has made our cluster of RC servers more stable and usable across every version we update to.