Rocket.chat: Email notifications to @all not working

[antgel]

Description:

See below.

Server Setup Information:

• Version of Rocket.Chat Server: 0.63.0
• Operating System: Ubuntu 16.04 LTS
• Deployment Method(snap/docker/tar/etc): docker
• Number of Running Instances: 1
• DB Replicaset Oplog: Disabled
• Node Version: 8.11.1
• mongoDB Version: Whatever's in the docker image

Steps to Reproduce:

1. Be offline or away in a channel.
2. Wait for somebody to ping @all in said channel
3. Wait for an email notification that never arrives.

Expected behavior:

We expect all users in a channel who have email notifications configured, to receive said notifications when they are offline or away and somebody mentions @all in the channel.

Actual behavior:

Some users don't receive said emails, and haven't done since we upgraded to 0.60.1.
All users do receive email notifications for DMs and personal (not all) @mentions.

I don't know how to debug this, but if you tell me where to poke around in Mongo, I can find more information. Also, email sending doesn't appear to be in email logs, which makes things difficult to debug.

Relevant logs:

If anyone from the RC team works on this (and we hope somebody will), ping me and I'll put the logs somewhere suitable, or better still, you're welcome to SSH in.

Cc @schurder @shazzaPM

[TwizzyDizzy]

Hi @antgel

so you are currently on 0.60.1? If so, I would kindly ask you to upgrade to 0.63.1 (or maybe wait for 0.63.2 which is coming soon) and try to reproduce your error there. There are also a lot of new settings surrounding the topic of notifications.

As stated in the documentation, we do not support versions other than current stable. Please also adhere to the issue template. Thank you! :)

Cheers
Thomas

[antgel]

@TwizzyDizzy No, we're on 0.63.0 which is pretty close to your 0.63.x suggestion, the error has been happening since 0.60.1.

[Hudell]

How many users do you have on the channels where this problem is happening?

On the administration center, under "General" and "Notifications", there's a setting called "Max Room Members Before Disabling All Message Notifications".

I just noticed that this setting is being used differently for desktop notifications and email notifications.
This setting is supposed to disable notifications for all messages once a channel grows to have too many users, but on the email notifications it is disabling @all mentions instead.

We currently have someone reorganizing our notification system aiming to improve the perfomance of it, so I'll ask him to fix this problem while he's at it.

[antgel]

@Hudell Thanks for getting back to us on this. On our system, that setting is set to 100, and we don't have (nearly) 100 RC users, so I'm not sure it's related the to issue (but correct me if I've missed something).

[Hudell]

Have those users verified their email address? As an admin you can check if they did and even mark them as verified if needed.

Rocket.Chat won't send emails if the address is not verified.

[antgel]

@Hudell I wish it was that. They're all verified (GSuite login only). As per my OP, it seemed to work fine up until a certain date. Anything I can check in Mongo?

[Hudell]

Other than that, the only things that can disable the emails are the user's preferences (both general and per channel). I'm checking the code to see if I can find anything else.

[sampaiodiego]

@antgel not sending emails on @all mentions is by design, it was this way since the beginning.

but looks like https://github.com/RocketChat/Rocket.Chat/pull/8059 added a logic to send email to @all if the room have less than Notifications_Max_Room_Members members.. is that a expected behavior? if so, I can add it here on my notification logics improvement PR, which is currently clearly not sending email to @all.

[antgel]

@sampaiodiego Hey, long time, hope all well. :) It's definitely expected behaviour, at least according to pretty much all of our users. Otherwise people who aren't online / frequent users, simply don't know what's going on, and the @all loses a lot of its power. What do you think?

[sampaiodiego]

@antgel yep, long time. =)

sending a huge amount of emails can be heavy on server and can lead to exhaust some SMTP quotas. but since the addition of Notifications_Max_Room_Members , I think the behavior was changed to send emails on rooms up to that limit.

I'll make sure then email notifications have the same behavior as other notifications, being sent to @all to rooms with less then Notifications_Max_Room_Members . 👍

[antgel]

@sampaiodiego Cool, although do see https://github.com/RocketChat/Rocket.Chat/issues/10434#issuecomment-385422065, so I'm not sure what the issue is, because as you say, they should be sent up to the limit. However, maybe the logic is buggy somehow. Is there, or will there be a way to debug this from the logs? When I look at the log, I feel like I'm flying blind. :)

[sampaiodiego]

the current state might be buggy. I'll not investigate the current implementation but I'll make sure that PR works as expected.

[antgel]

I'm reopening this, because even in 0.65.1, it still doesn't work, when people type @all in a channel, email notifications are not sent, and the logs don't seem to provide any visibility into this. Again, happy to provide anyone from RC team access to our server to help debug.

[sampaiodiego]

@antgel I have tested this on my local environment and it worked as expected.

rocket.chat will send emails to @all mentions, only to users with verified emails and if the room's member count is not greater then Max Room Members Before Disabling All Message Notifications
(Notifications_Max_Room_Members) setting - under General -> Notifications - which is 100 by default.

[Hudell]

The Notifications_Max_Room_Members setting should only affect servers where the default configuration is to send notification on all messages, it should not affect the @all notifications.

[antgel]

@sampaiodiego Thanks for getting back to me. As I said above, we don't have 100 users server-wide and that setting hasn't been changed. The email addresses are all verified (GSuite login only). Would you be willing to login and take a look, or let me know what to check? There's no related debug in the log as far as I can see (should there be?).

[sampaiodiego]

@antgel I can take a look on your server.

there is no additional logs to check.. I might consider adding some at higher log levels.

[antgel]

@sampaiodiego Cool, you have email to the address in your GitHub profile. :)

[sampaiodiego]

@antgel I think this was actually fixed by https://github.com/RocketChat/Rocket.Chat/pull/11024

I tested this on docker versions 0.65.1 and 0.65.2 and indeed I was not getting notifications for @all mentions.. but on 0.66.0 they were sent as expected.

[antgel]

Confirmed working in 0.66.0! Thanks @sampaiodiego and team!