Nodebb: Notification of new topic in category

[mattmyne]

It seems you can currently get notifications for:

• someone you follow posts a new topic
• someone replies to a watched topic

..but no option for a notification on a new topic (in a watched category).

Like many forums we have an announcement forum. Our users probably won't be checking the board very often, but would like the option of an e-mail when a new announcement topic is posted.

I've tried the nodebb-plugin-category-notifications which adds it's own "subscribe" option to categories. The main problem with this is it also sends an e-mail for any replies to topics too, which users don't want (they want to know new topics, but not have an inbox full of everyone's replies to it).

Can a "new topic in watched category" notification be added to the core notification system? It'll be less confusing for users (needing to use a separate "subscribe" button from the plugin) and better integrated (they can choose if they want a notification or e-mail, rather than the admin deciding for them via the plugin) and uses the existing "watched" method.

Following a comment on the forum (where I posted this question originally) by @barisusakli given the default case of all categories being watched by users, I agree this new notification could either be set to None by default (to prevent unexpected notifications for all new posts) or the default for categories changed to not watching.

The first option would be a quick fix, but I'd suggest the second since that's effectively what we're expecting. If you're watching a category, you'd actually expect to be notified of new topics in it (replies to topics are already handled). Since this isn't the case currently, then it's perhaps more logical to make the default category setting as not-watched.

[julianlam]

Time for Ignoring, Not Watching, Watching, and Following? :thinking:

[mattmyne]

I envisioned watching staying as it is (watching something generates notifications), but adding a "new topic in category" notification type.
I think adding "following" as well would be confusing for users alongside "watching".

[alfazaz]

I follow this because I think this would be a great improvement of nodebb (that I need).

Like mattmyne, "following" seems confusing (when I imagine it in its french translation). It's to close in its meaning with "watching".

"Subscribed/not subscribed" ? (don't know if it is really greater...).

[julianlam]

I also do agree that adding a new "Following" state would cause unnecessary confusion.

@barisusakli when you say that you wanted to make this an option, would this be in ACP or UCP?

For backwards compatibility, we could add an upgrade script to manually toggle this option off for pre-existing users, but default on.

The only remaining blocker is a quick discussion on whether it is okay to have a watched category be:

1. Appearing in /unread
2. Appearing as a notification

Or neither, so you can't do one or the other, only both.

[barisusakli]

It would be both in ACP(for the default) and the UCP(so the user can change it). Like the other notification types.

Default should be off so everyone doesn't get spammed by new topic notifications on every category.

I think it's ok to have it appear in both, appearing in unread only helps if you are visiting the forum regularly while the notification can email you for watched categories. If you are regularly visiting the forum and have the notification turned on then you would see some redundant data, same topic in /unread and in notifications.

[mattmyne]

The only remaining blocker is a quick discussion on whether it is okay to have a watched category be:

1. Appearing in /unread
2. Appearing as a notification

Or _neither_, so you can't do one or the other, only both.

I'd vote for watched categories still appearing in /unread the same way replies to watched topics are.

If there's a separate "new topic in watched category" notification option, then 2. can be decided by the user's notification settings. So both.

[julianlam]

:+1: sounds good to me!

[alfazaz]

Like mattmyne, I vote for both too...

[barisusakli]

Looks like this isn't possible right now, well at least not in a scalable way.

The current default setting for categories is watching meaning all users are assumed to watch all categories. Now if a new topic is posted in a category then we would have to go through all users to figure out which ones to send the notification to.

The only way I see this working is to remove the default settings for categories and hard coding it to Not Watching which is what topics are doing. All new topics are considered Not watching until an explicit user action to watch or ignore it. Obviously this would cause other headaches, when an existing nodebb upgrades to latest users would have to explicitly watch categories they are interested in.

[mattmyne]

I thought we'd agreed the default category settings after the update would be not-watching, so current installations wouldn't spam their users with new topic alerts. In addition, the default setting for "notify on new topics in watched category" could also be "None".

I'm new to this software so have not looked into its internal workings. However, would the code not operate in a similar way to how nodebb checks for users that need notifying when a new reply to a topic is posted? Presumably this is not too much computational effort (I would hope the topics have an internal list of users watching them, rather than a loop being run through all users checking if they want notifications for every board reply, but either way - I guess it's working ok).

Can you clarify what the blocking issue is?

[barisusakli]

For topics this is not an issue because there is no admin/user setting to change the default behaviour from Not watching. Topics are always considered Not watching and when a user watches a topic they are added to a list of uids that are watching that topic. So when a new reply is made we get that list.

Categories work a bit differently because of the default settings. Let's say you have the default set to Watching which is the case for most nodebb installations out in the wild. Now when we are checking to see if you are watching this category we check if you have explicitly watched it and if not we check the default settings. In code it looks like this

    Categories.getWatchState = async function (cids, uid) {
        const keys = cids.map(cid => 'cid:' + cid + ':uid:watch:state');
        const [userSettings, states] = await Promise.all([
            user.getSettings(uid),
            db.sortedSetsScore(keys, uid),
        ]);
        return states.map(state => state || Categories.watchStates[userSettings.categoryWatchState]);
    };

So when it comes to get the uids who are watching a category we can't just read them from the list 'cid:' + cid + ':uid:watch:state' because if the default is Watching they might not be in the list but still considered watching the category.

We can't just change the default to Not watching as that would only work for people who keep using Not watching but if they change to Watching or Ignoring that would not work.

[mattmyne]

Thanks for the further explanation. If I understand correctly, uids are currently only added to the categories if the watched setting is changed from the default (makes sense). So if the default is changed, the behaviour for the users previously on the old default may be unexpectedly changed too?

I expect this could have been compensated for using a one-time (during nodebb upgrade) regeneration of the watching uids per category by iterating through the global users and rebuilding the list according to the user's settings and old default logic.

However, let's just keep it simple then. Don't change the default category watch state - keep everything as it is. Just add the new notification option for "notify on new topic in watched category" and set its default as None. This was one of the alternatives you'd originally suggested.

That'll keep everything the same for current boards but just give users this new notification option if they wish. They might then prefer to change their default category watch state after enabling the option (and maybe unwatch some categories afterwards), but that's up to them. Adding a "reset all categories to default watch state" button in the settings would be a nice time saver, but not essential.

[barisusakli]

uids are currently only added to the categories if the watched setting is changed from the default (makes sense).

Yes this is correct. Right now there are no uids in the list. So when a new topic is posted nobody would get a notification even when they enable the notification setting. They would have to first change their category watch state explicitly (even if it shows Watching) so they get added to the list.

I expect this could have been compensated for using a one-time (during nodebb upgrade) regeneration of the watching uids per category by iterating through the global users and rebuilding the list according to the user's settings and old default logic.

We can't do this because we ended up with issues where the forum had 400+ categories and 2 million users and they wanted everyone to ignore all categories resulting in millions of entries just to mark all users as ignoring all categories. That's why the default options were added.

It is an annoying problem. Personally I would be OK with removing the default category watch settings an assume all categories are Not Watched.

[mattmyne]

It is an annoying problem. Personally I would be OK with removing the default category watch settings an assume all categories are Not Watched.

Let's do that then!

Maybe add a button in user settings to "set all categories as watched" for those that really want /unread to fill with new topics from all categories without manually setting them all. It'll save having to do a large automatic iteration for everyone during the upgrade.

[PostMidnight]

Just looked for something similar today, what's the odds for that ;)

Please consider having separate, per-category default watch state flag. Such flag may be set by category create or update. Subsequently, isIgnoring may be modified to first look for the db's per-uid ignore field (as implemented today), and if it does not exist, revert to the said default category flag.

[mattmyne]

So can I confirm there's no intention of implementing this anytime soon, because you can't?

[barisusakli]

Pretty much, implementing this would mean removing the ability to set defaults which has its own problems. Some forums want to set all users to ignore or watch by default. That wouldn't be possible if we hard code it to not watching and require explicit action from users.

In the meantime I can probably improve https://github.com/NodeBB/nodebb-plugin-category-notifications so it gives an option to only receive notifications for topics.

[alfazaz]

This feature is really missing from nodebb. The nodebb-plugin-category-notifications can be a solution but, to the end, I think it should be a core functionality.

I understand the difficulty and the time necessary to implement this in core (but it could a really great improvement for nodebb).

[barisusakli]

There is no difficulty in terms of implementing and wouldn't take long to implement. The problem is it can't be done in a performant way without removing the default watch state feature.

[alfazaz]

Yes, sorry... I was thinking about things to do to change these things (create an upgrade path accounting the change of the watch state...).

[mattmyne]

I agree this should really be a core feature, given we already have notifications for similar things. It'll be a shame to need to include a separate plugin for this and the added confusion with having a "subscribe" button.

I appreciate it's not a technical but a compatibility issue, though the end result is the same at the moment! Surely there's a route through this though, with potential ideas such as:

1. handle database updates given old defaults during update so users wouldn't notice (high computational load during upgrade for large boards though).
2. put the new notification option in, but set as none by default. No change in behaviour for existing users, users who want to use new notification might choose to change their category watch status themselves afterwards.
3. remove default watch state, assume all categories not watched and allow users to opt-in if they want to watch it. Possibly add a "set all as watched" button in users' settings for users that genuinely want all categories to be watched. Would change behaviour for current users, in that their unread folder would now not fill with new topics in all categories (though recent would still work as before) unless they re-watched the categories they wanted manually. Maybe that's a fair change going forwards anyway - do the majority of users really want to watch all categories by default?

[barisusakli]

Out of those ideas only 3 works.

1. Can't do this, on a forum with 400+ categories and 2 million users we would have to create 800million records in the upgrade script, if their default is watch all categories

2. This also leads to weird issues if the category default is set to watching. Category default is Watching so users see Watching when they go to a category, then they enable the notification in their settings but they won't get any notifications because they haven't explicitly watched that category.

[mattmyne]

To try and continue progress;

1. Sure, though there may be acceptable ways of achieving this (managing the load) even with this extreme example. But it seems the preference is a fix in code rather than database update.
2. Again, a fix in code to treat the default state the same as an explicitly-set state would make sense? Arguably that's good practise for a default state to be treated the same way as an identical explicit state as the user can't easily tell which condition is happening. The code that checks for users directly registered as watching against a category should then perform the same check for all users given default settings.

Still, rather than try and work on 1 or 2, you say 3 works so could we shift to see how 3 can be implemented?

[barisusakli]

1. This isn't an extreme example, the reason why the current system was built is to address this issue. The issue is with storing that much data, for example we have a client with ~4.5million users and ~300 categories. And storing that many category watch states was consuming 100gb+ of ram on redis.

2. The code that checks for users directly registered as watching against a category should then perform the same check for all users given default settings.

Like I said this is doable but not performant, we just can't go through 4 million users whenever a new topic is made to figure out which users to send the notification to.

I still want to build this and change the default state to Not watching.