Smokedetector: Smoke Detector instances can't coordinate without MS

[spacemonaut]

What problem has occurred? What issues has it caused?

MS has been down this morning. All online Smoke Detector instances (iBug, ArtOfCode, teward/Solar Flare, teward/Lunar Eclipse) keep claiming they're receiving failover signals and starting up. In the past four hours of MS being down, about 26 standby requests have been made in Charcoal HQ (see transcript here and the next time period, ctrl+f for !!/standby).

What would you like to happen/not happen?

Metasmoke instances should coordinate between themselves over chat somehow:

• which ones are online
• which one is the primary instance
• whether the primary instance is healthy and responsive

We should never have two instances of metasmoke running and responding simultaneously, even when MS is down.

We should never have more than one instance running actively (i.e. out of standby). All other instances beyond the primary should be in standby.

We should never have multiple instances repeatedly claiming to have received a failover signal and leaving standby while another instance is already primary and active. This should be maintained even when MS is down.

When no instances are present and healthy and online asserting primacy, other instances should attempt to assert primacy. If multiple assert primacy simultaneously there should be a means to allow them to decide who "wins": for example, if they all post to Charcoal HQ asserting primacy, the one with the earlier chat message ID wins and becomes the primary, others do not leave standby because they recognise they had a later chat message ID.

[ArtOfCode-]

I have a neat little websocket server around somewhere, and plenty of machine space to host it. I'd suggest using that - all instances, including standbys, connect to it, and essentially use it as a relay between themselves to co-ordinate when MS is down.

[spacemonaut]

@ArtOfCode- I think it will be useful to have a last ditch protocol of "use chat" as well beyond that in case even that server is down. Even if everything is down, Chat is still up. (If Chat is not up, Smokey instances have nothing to do anyway.)

[ArtOfCode-]

Eeeeeeh. Using chat for this kind of communication feels icky.

[Undo1]

Does sound interesting, though - we already know how to talk and listen to chat, could be neat to make a new room where Smokey talks to itself. Would need to be low volume, but it'd completely remove us from the hosting question for this

[stale[bot]]

This issue has been closed because it has had no recent activity. If this is still important, please add another comment and find someone with write permissions to reopen the issue. Thank you for your contributions.