Smokedetector: Is it worth hooking Smokey into Redunda?

[ArtOfCode-]

Redunda is a status-monitoring project, built mainly by Undo, Nobody, and myself, for SOBotics. It does a similar sort of thing to what metasmoke's status page does, but because it's designed for the purpose, it does it better.

Is it worth hooking Smokey into that system as well as metasmoke? We do need to keep metasmoke for a couple of other things it does that are unlikely to make it into Redunda, but one more HTTP request isn't going to take up a whole lot of resources. My logic here is that using a system designed for status monitoring is generally going to get better results than using a system designed for status-monitoring-and-twenty-other-things.

(also, as @Undo1 has said, it'll force us to get Redunda right, which benefits more than just us.)

[AWegnerGitHub]

As long as it is optional, like integration with MS, I think it's a good idea. Presumably, the changes to Smokey would be minor, and it will allow the monitoring aspect of metasmoke to be removed.

[ArtOfCode-]

Optional can be done fairly easily, I imagine, though getting a key for Redunda is significantly easier than getting one for metasmoke.

[angussidney]

I'm really not sure if this is the right thing to do.

• Apart from the fact that a few (or all?) of the devs of Redunda are part of Charcoal, Redunda and SOBotics don't have anything to do with Smokey and Charcoal. All the other bots post to the SOBotics room and share the same 'rules and regulations', but Smokey does neither. It doesn't make much sense.
• Access control. Currently, we have a big long list on MS which is used to control people's access to things (only people with code privs can cause a bot to failover). However, if we transition over to Redunda, it means that there is yet another privilege list which we have to maintain, and everyone will have to create accounts over there, etc.
• Will metasmoke still have failover functionality? If it does, what happens when Redunda and MS each choose a separate standby instance to put back into action? We get two active instances for smokey? That isn't good.
• Using MS keys means that we can easily signify that 'This instance is a certified official instance of Smokey that has full MS integration'. If we switch over to Redunda, anyone can create their own instance with status monitoring, which may incorrectly imply that it is official
• What benefit is there to switching over to Redunda? From my early impressions, it doesn't yet have all of the features that MS has, and there isn't anything that stands out as particularly useful which MS can't do. Not to mention that it will be another project that we will have to put in the 'Project links' message.

[ArtOfCode-]

@angussidney:

• There sort of are no rules or regulations for Redunda, so there are no issues there.
• Yes, maintenance of lists and accounts is a thing we would have to do. However, account creation on Redunda is two clicks, since it's all done through SE OAuth.
• We could disable failover stuff on metasmoke. We need to keep pinging it, but disabling failovers there wouldn't be hard and wouldn't affect the other things we need to keep pinging for.
• Redunda is set up so that only people with collaborator access to a bot can create an instance of it. We can restrict collaborator access to MS code admins. Smokey's account credentials are still separate from Redunda, so even people with collaborator access can't create an "official" Smokey (i.e. running on the Smokey account) without someone giving them creds manually.
• The main benefit I was seeing is that it's a service designed for the purpose of status monitoring, as opposed to metasmoke which has status monitoring to some extent, but it's not the primary focus. There's also benefit for other bots derived from Smokey using Redunda. And, since it's technically an SOBotics project, it doesn't have to go in our Project Links.

[ArtOfCode-]

Oh - and the other benefit it would get us is the ability to share data between instances. We'd have to write the functionality to add this to Smokey, but @NobodyNada is working on a feature that lets bots store arbitrary data on Redunda. We could use this to share notification lists, user blacklists and whitelists, etc.

[AWegnerGitHub]

That seems like a misplaced feature for both Redunda and each of the bots.

If bot instances need to share state data, they should be holding that information in their own databases that they control, not depending on a third party to store it. From a Redunda point of view, that goes outside of the scope of "monitoring" to include "arbitrary data storage". It introduces a potential point of failure that the bots need to account for - what happens if Redunda database is inaccessible, but the bots are still running?

Additionally, if that's the reasoning for Smokey to use it, I'm going to change my previous tentative support for this to "Don't think this is good". It'd remove the "optional" aspect and it'd force a dependency on a third system to ensure our entire solution is working (SmokeDetector, MetaSmoke, Redunda)

[ArtOfCode-]

@AWegnerGitHub The problem with that is the same problem we've had with Smokey - sure, we could hold that data in a database, but how do we share that database between instances? Barring zipping the data up and emailing it to the next host when hosts change (which is inefficient and time-consuming), we still have to have a server somewhere to upload data to and download it from.

SOBotics' bots have had the same issue: if they need to share data, they need a server. Any server is going to have the same issues as Redunda does - what if it's inaccessible? Wherever the server is, bots are going to need to code for protection against it being inaccessible. So, we figured instead of making everyone set up their own databases and servers and have to make it part of a bot architecture they have to maintain, we might as well centralise it to Redunda, which is an EC2 server that Undo runs. It's not often those just die on us, and it means that bot maintainers can maintain bots, instead of keeping their own servers up as well.

[AWegnerGitHub]

@ArtOfCode- At last check, we have a server that MetaSmoke runs on. It has a database. Why can we not utilize that existing infrastructure, instead of depending on another service?

[ArtOfCode-]

@AWegnerGitHub We could, there's nothing stopping us. Historically, we haven't done it because of downtime concerns. However, whether we use metasmoke or Redunda, someone needs to develop that functionality - and if we're doing that, why not put it in Redunda where multiple bots can benefit from it?

[AWegnerGitHub]

Let Redunda build that functionality, I'm arguing that should not be a reason for SmokeDetector to use it though. To me, that doesn't provide enough of an incentive to complicate our solution.

[ArtOfCode-]

What are the complications though, @AWegnerGitHub? I drafted the code yesterday; it's two lines in the config, a few in globalvars, and one new method in metasmoke.py. That gives us one extra HTTP request per 45 seconds (could be a touch longer, too).

[AWegnerGitHub]

If we are using it for monitoring, that seems OK.

The complication, though, is that now a single instance of SmokeDetector depends on the following:

• SmokeDetector itself
• MetaSmoke
• Redunda (fairly minor at this point, as it's just doing monitoring; this becomes a major component though, if we do database storage)

---

I am not concerned about the complications when adding Redunda as a monitoring solution. I'm more concerned about when it's a shared database between Smokey instances and all kinds of other bots. Then it becomes more than a single HTTP request.

[AWegnerGitHub]

All of that said, if the code is written and the dependency on MS for monitoring has been removed, I don't see a reason not to switch.

[j-f1]

@AWegnerGitHub The content saved to Redunda would not be accessible by other bots, only Smokey instances.

[Undo1]

After sleeping on it a couple times, I think I'm on the "not yet" side. Andy makes good points, and metasmoke has some statistics gathering logic that it doesn't make sense to include in Redunda.

[angussidney]

I'm still not getting it. @ArtOfCode-: your main argument is that Redunda would be better than MS for status monitoring because that's what it's designed for. But I still don't see how it is better than MS. They both monitor the status. They both allow a certain selection of people to trigger a failover. They both automatically cause a bot to failover when it goes offline.

Unless there is some sort of non-insignificant benefit to using Redunda over MS, I don't think changing is worth the fragmentation it would cause. MS should remain as our primary method of status monitoring.

Note that I am not against setting up Smokey to work with Redunda, I am just against removing any functionality from MS.

[ArtOfCode-]

So (as I said in chat just now) I'm not proposing removing functionality from metasmoke, nor am I proposing we stop pinging metasmoke entirely. I'm suggesting that we ping Redunda in addition to pinging metasmoke; that gets us the benefits of Redunda without losing anything we get from metasmoke.

Would particularly like to hear current opinions from @angussidney, @AWegnerGitHub, and @Undo1.

[Undo1]

I don't see a reason not to send pings to Redundant, but we probably shouldn't let it handle failover. Metasmoke should stay the sole source for that.

And we shouldn't be dependent on it at all.

[AWegnerGitHub]

Tabled for now. We can reopen this if/when a new discussion about using redunda for monitoring or database management comes up.