Smokedetector: What do we need in a central blacklist tracker?

[Undo1]

As brought up in https://github.com/Charcoal-SE/metasmoke/issues/257, there's a desire to move blacklists out of GitHub and into some other tool. Whether that tool is metasmoke, another app, or some other alien-tech-powered solution, we'll want to know what it should do. Here's my first take:

• Store blacklist entries categorized by type ([watch|blacklist] [website, keyword,username], etc.)
• Have an easy-to-use way to add entries
  
  
  
  • From a web UI, instances, and/or metasmoke
  
  
• Have reasonably good uptime
• (?) See what changed since the last update?

Thoughts? Am I missing the mark? What else do we need?

[BrockA]

Looks good to start.
Maybe also search/sort/graph by hits , and/or false positives, per entry?

[j-f1]

It would be useful to have an independent web UI so blacklisting can still go on if MS is down. Maybe a page on https://charcoal-se.org?

[AWegnerGitHub]

I've been looking at the links you provided here and thinking about this today.

I think the blacklist would need

• type: blacklist-keyword, blacklist-username, blacklist-websites, watchlist
• text: the pattern to blacklist.
• created_at: Just a simple timestamp of when it was added. I think we're just going to have to populate the list with the date when this is implemented initially
• modified_by: Who to blame when a pattern fails; last person to touch this entry
• updated_at: Obvious

From a front end, I think it could really be as simple as - select type, entry pattern. The rest can be automatically supplied by metasmoke.

---

Since I've been thinking about this, I also think we need a table for notifications, since we lose those between instances right now. The layout of that table would be the same as we store it now, which I think it user_id, site, chat_server?

---

For your last bullet, the simplest method I came up with was maintaining a third table of last updated timestamps for each of the other tables. Every time one of the other tables is added/updated/deleted from, this table has the key associated with that table updated to the current time.

list                timestamp
blacklists      2017-10-05 00:44:27
notifications 2017-10-04 12:15:00

Alternatively, we could do it based on table/key so the blacklist entry above could be broken down to something like

list                type          timestamp
blacklists      keyword   2017-10-05 00:44:27
blacklists      user          2017-10-01 15:13:11
blacklists      website     2017-10-04 23:57:51
notifications 2017-10-04 12:15:00

Each instance of Smokey maintains a local value of when it last updated it's cache. When it's time to check (ie. brought up from standby, initial start up, forced update command, etc), it fires off a check against that table. If it's cache is newer than the value in the table, it doesn't do anything. If it isn't, it pulls down the new list for only the affected table and then update's it's local cache (and record of when it pulled the data).

---

Assuming we go this route and it's outside of MS (the MS front end can just query the same way), this allows us to build in smokey commands that will continue to function even if MS doesn't. It also allows us to continue to update blacklists if Smokey goes down and it will pull them automatically when it recovers.

[AWegnerGitHub]

One thing to consider if we do go the serverless framework route: we need two sets of credentials. We need a read-only that smokey uses by default so anyone can pull updates. We also need a set that can write , update and delete so that authorized instances can update the data.

We'll have to talk about how that information is stored, but I'd guess Smokey runners would need that

[Undo1]

MS could sync access tokens to something Serverless has access to - wouldn't need to have another set at all. Allow read-only access through unauthed requests; write access with a valid access token.

[quartata]

I'm pretty against the idea of having just a central server storing the blacklist, simply because if it goes down (unlike MS going down) our scanning capabilities are kinda hamstringed. I would much prefer a system in which MS propagates blacklist changes from one instance to all the other instances on standby via ActionCable, in addition to potentially keeping a copy for itself for new instances or a pretty UI like suggested. (plus MS storing blacklist requests waiting on approval)

[AWegnerGitHub]

@quartata No one is suggesting that Smokey query the remote blacklist for every check (at least I hope not...we'd chew through bandwidth/API calls). We are suggesting a decoupled service on AWS - a pretty reliably network - that serves as the source of record for these entries. Each instance of Smokey would keep a local cache of these records and all scanning is done against the local cache.

Existing commands would be easily modified to pull the records from AWS instead of performing the whole write to git, commit, push, merge, build, pull routine that goes on now. Instead, a command would write to AWS and we either decide to write directly to the cache or repull the entire thing and rebuild the cache. Other instances would rebuild their cache at start up/taking off standby.

Depending on MS to push changes over a web socket - an area where we have constant problems with reliable connectivity - seems like the wrong way to distribute updates. There is no reason to have the standby instances constantly updating. It adds in another problem area we have to worry about. Did the active and standby instances all update? Instead, pulling updates only when coming out of standby is more reliable. The new instance is requesting the data, instead of hoping the web socket didn't die.

Regarding the UI and approval of blacklists, I think all of that can be built easily, regardless of where we ultimately end up storing the blacklist. It requires one more column on any table - approved (maybe a second with approved_by and third approved_at - and an MS front end that only allows users with X permission to click the "I approve" button.

[Undo1]

@quartata What Andy said - this would be leagues more reliable than any other solution I can think of, once we get it running. Here's how I'd envision it looking:

1. Blacklist item is added by any means - MS interface, !!/blacklist, whatever.
2. Lambda tells MS to broadcast a WS update saying "hey, some blacklists changed. Go look over there." (not necessary for chat-issued commands already going through instances, but useful for blacklist entries from outside of the Smokey instance)
3. If the instances get the websocket update, awesome. They update their local copy of the blacklists.
4. If they don't, we'll get the update eventually anyway - they'll update on reboot, with a dedicated !!/update-blacklist command, on a schedule, or whatever.

Imagine our current system, with blacklists stored locally, and a repository-free way to update them. That's what we're talking about.

EC2 has a 99.95% uptime SLA (about four hours a year); Lambda doesn't have an explicit one but I'd expect it to be higher due to the nature of the service.

Anyway, I think we want compatible things - (1) a distributed system that isn't wholly dependent on one thing to work, albeit with reduced functionality, (2) simplicity that doesn't require people to open firewalls and such, and (3) a hope to never need that distributed part of the system.

Also, (4) cheap.

[angussidney]

Not to mention, if these services go down it means that EC2 has also probably gone down, which means that most of our Smokey instances will be offline anyway.

[Glorfindel83]

I'd like to be able exclude (or include) certain Stack Exchange sites when blacklisting websites / keywords, like this and this.

[SulphurDioxide]

Any instance which is in standby won't automatically get the blacklist updates anyway until it is next pulled in the current system. It would be the same in this case and you could just run !!/update-blacklist when a new instance takes over.

Would it be possible to periodically get the standby instances to request a new blacklist - say once a day - so that when they are called up, they're not woefully out of date?

[angussidney]

Re @Glorfindel83's idea: this would require an all field as well as a sites field, which would work similar to how the current Smokey works. However, it would require some relatively large modifications on the Smokey side relating to how our blacklist functions work, so maybe we can leave this until we have everything working properly.

[angussidney]

Remember, we also want to move the rest of the data when we do this. This includes:

Notifications (as mentioned by Andy):

• user_id - chat user ID
• chat_server - chat.SE, chat.meta.SE or chat.SO
• room_id - the room which the user wants their notifications to be posted in
• site - SE site to be notified about
• date_added - might be useful in some cases

Privileges - this would allow the syncing of privileges between MS and Smokey automatically, making it less complicated and easier to maintain. However, I'm not sure what we'd do about people who have Smokey privileges but not a MS account, which is required to manage MS privileges. Finally, do we have a way to programmatically find chat IDs?

• user_name - for easy recognition by humans
• network_id - Main user network ID
• chat_se_id - Chat user on chat.SE
• chat_so_id - Chat user on chat.SO
• chat_meta_id - Chat user on chat.meta.SE
• date_added - may come in handy from time to time?

Blacklisted and whitelisted users - due to the high frequency with which we switch instances, this rule isn't as useful as it should be.

• user_id - Main user ID on the site
• site - SE site
• date_added - when was this user first blacklisted/whitelisted?
• added_for - a link to the offending post or MS record
• reason - could be useful for later reference

Ignored posts - same story here

• post_id
• site
• date_added
• added_by - user ID? Name? MS feedback ID?
• reason - could be useful for later reference

[j-f1]

We should also move the other lists (bad keyword etc) into this blacklist structure.

[quartata]

@Undo1 OK, that sounds much better than what I thought you were envisioning. (That's basically the kind of thing I was trying to describe)

[AWegnerGitHub]

Privileges - this would allow the syncing of privileges between MS and Smokey automatically, making it less complicated and easier to maintain. However, I'm not sure what we'd do about people who have Smokey privileges but not a MS account, which is required to manage MS privileges. Finally, do we have a way to programmatically find chat IDs?

Do we really need separate permission groups? We could merge everything into one set of permissions. It'd make management of permissions easier. I think that's a discussion for another thread though.

As for finding chat IDs...the answer is 'kind of'.

Example using me. First find their network ID. (63984 for me)

• chat.stackexchange.com ID: Visit http://chat.stackexchange.com/accounts/[networkid], for me this redirects to https://chat.stackexchange.com/users/66258/andy
• chat.stackoverflow.com ID: Visit http://chat.stackoverflow.com/accounts/[networkid], for me this redirects to https://chat.stackoverflow.com/users/189134/andy
• chat.meta.stackexchange ID: Not sure. Following the pattern above redirects my network id to another user.

So, we can get two of the three IDs. We'll miss out on the Tavern one though.

[j-f1]

chat.meta.stackexchange ID: Not sure. Following the pattern above redirects my network id to another user.

I get redirected to https://chat.meta.stackexchange.com/users/186281/andy, which seems to be correct.

[quartata]

Not sure what the context of this is, but ChatExchange User objects have an id field. You can also get SmokeDetector's ID from the client with <client>._br.user_id.

[ArtOfCode-]

We could sync permissions, aye. If metasmoke sync'd its permissions list to this new system, then Smokey can use those as the permissions for itself as well.

Also norepro on the chat.meta ID problem.

[angussidney]

Note that once we start syncing permisssions, we need a separate authentication method for admins to make sure that someone with a Smokey key can't edit permissions. Maybe only allow the MS key to change user permissions, then admins can change it through the existing interface.

[Undo1]

Nice thing about this: Metasmoke could edit the DDB database directly for things like permissions - no fancy auth needed.

[spacemonaut]

There's something I'd like this new tracker to have capacity to do:

• track number of matches against different patterns
• items in watchlist with a certain threshold of matches get brought up to propose moving them to a blacklist
• items in watchlist which haven't seen anything in a long time possibly get suggested for removal from the watchlist
• anything in a list which doesn't get matched for a long time, or has lots of false positives, gets brought up for removal

[Undo1]

@doppeldeveloper Everything there sounds like something we could handle in metasmoke (or another utility); and we definitely should do it. The goal here is to have a high-reliability way to sync data between instances - once we have that, it's easy to query it from other tools.

Metasmoke has the hit data needed to pull off those items; as a non-relational DB this wouldn't.

[tripleee]

There are large complex internal regexes which ideally I'd like to refactor so we can track which parts of them are matching what; but again,that'a different thread really (namely, #1133).

For the record, I have most of the proposed watchlist management functionality done, I think - Halflife has some automation to help decide when to promote a watch to blacklist, and the other way around was then easy to build, but only exists as a batch command-line tool prototype yet.

[magisch]

Ideally we'd have integration with Metasmoke to provide detailed stats on individual reasons / entries.

Aditionally, I concur with @tripleee that we should be seperating some of the larger internal regexes into their own reasons so we can more accurately track their efficacy and maintain them easier (and cull old / irrelevant entries more easily).