Shields: Migrate GitHub tokens to cloud Redis

[paulmelnikow]

As discussed in #1848, let's migrate the Shields runtime data store from the server file system to cloud Redis.

This centralized / hub-and-spoke architecture is simpler and easier-to-reason about than today's peer-to-peer setup, which requires each each server to know about all of the other servers. Bringing a server online requires reconfiguring all of them.

Switching to cloud Redis also makes it possible to move to a PaaS.

A reason to do this _now_ is that we may need to add token rotation to support Gitlab (https://github.com/badges/shields/issues/541#issuecomment-484264221). It would reduce the complexity of supporting Gitlab if we make the migration first.

The code is already in place for this (see #1939, #1906, and #1848), so the only thing left is to operationalize this:

• [x] Choose a cloud provider
• [x] Provision it
• [x] Configure backups
• [x] Load the tokens into the database
• [x] Test the config locally (#3705)
• [x] Push config to a canary server (s0)
• [x] Push config to the remaining servers

[paulmelnikow]

Two options from https://github.com/badges/shields/issues/1848#issuecomment-410557894:

• Compose, which I've used before, has Redis with persistence starting at $18.50/month
• RedisLabs has it for $7/month, or if we want it highly available, $9/month

We could ask for sponsorship, though this isn't a lot of money and we do have it in our budget.

[calebcartwright]

I'd be good with either of those. I think Heroku offers Redis-as-a-service too

[paulmelnikow]

Interesting! Heroku did agree to give Shields a credit, which we could use for our production hosting, though since Redis is a shareable add-on, we probably could use it for this as well.

One concern is there doesn't seem to be an offline way to access the backup:

https://devcenter.heroku.com/articles/heroku-redis#persistence

[calebcartwright]

I think that's only applicable to the hobby (free) tier though.

I haven't looked at any of these providers in detail, but I believe any would be a solid choice. I've not used Compose myself before but i've heard a lot of good things, and you know RedisLabs will do Redis well 😄 If we end up using Heroku for production hosting then there's probably something to be said for using their Redis offering as well

[paulmelnikow]

It does say:

The hobby tier for Heroku Redis does not have any persistence.

Though it also says:

Heroku Redis does not expose a method or interface for downloading the snapshots that are taken. If a backup is desired, using the fork option via the CLI or external replication is a way to snapshot the Redis instance in its current state.

I'm fine with Compose or RedisLabs. I guess for half the price ($9 instead of $18.50), may as well try Compose? If we do move to Heroku someday maybe that would be a good time to migrate.

I sent @espadrine an email asking if he objects to moving forward with this.

[calebcartwright]

Though it also says:...

Oops! I see what you meant now

I guess for half the price ($9 instead of $18.50), may as well try Compose?

SGTM

[chris48s]

$9/month is a good price.

Just as another data point, AWS is slightly difficult to compare because they bill by the hour and there are different tiers and it depends what region you're running it in and... stuff, but I'm currently running a t2.micro AWS ElastiCache redis instance and it costs about $13/month.

IMO, this is wise use of the budget because as it helps the move to a hosting model that will allow us to reduce the project's bus factor for deployment.

[calebcartwright]

helps the move to a hosting model that will allow us to reduce the project's bus factor for deployment.

I am +1000 on anything that helps us with that 😄

[paulmelnikow]

Thought I'd revive this!

Back in April Thaddée responded offline. He mentioned three things.

(1) He was concerned about latency and suggested having one in Canada and the other in France. Since the goal is to synchronize data between servers having two of them won't work.

I'm not super-concerned about latency, since the initial tokens are loaded asynchronously after the server starts up, and after that asynchronously (and outside of any request) when something changes. As far as I can reason, it's not a problem if this is a little slow.

https://github.com/badges/shields/blob/aa135d370fadea22f0f642466a3d17c52ffb00c9/core/server/server.js#L327
https://github.com/badges/shields/blob/aa135d370fadea22f0f642466a3d17c52ffb00c9/services/github/github-constellation.js#L53-L79

(2) He also mentioned:

It's easy to misconfigure Redis to send data in plaintext over the wire. We need to be careful to ensure the connection is encrypted, and to ensure that all ports open on Redis, including admin access, are secure.

👍

It looks like Compose supports SSL and makes it possible to turn off non-SSL connections, so I will give that a shot.

(3) Finally, he mentioned jsonsync, a library created to synchronize content like this using CRDT.

jsonsync isn't actively maintained and I'd prefer to move forward with the Redis solution for reasons mentioned here.

[paulmelnikow]

With the fix from #3707, this is now live and well on s0!

[paulmelnikow]

Live on all three!