Bedrock: Move Bedrock to SQLite in production

[pmac]

We think we can remove the external database server dependency from bedrock. Bedrock causes no data to change while it is running. This means that it can use a read-only database. SQLite is quite good and fast even in multi-process environments when it is used read-only. So here's the plan:

1. Setup the bedrock docker containers to start 2 processes (which they're already doing for l10n file updates)
   a. 1 process will be the WSGI process serving the HTTP requests
   b. 1 process will be a cron-like process that runs management commands on a schedule
2. Update and ship a sqlite database file in the bedrock production images during deployment

When these containers run they will both use the local sqlite database to serve data-driven pages (e.g. release-notes, security advisories, and product-details), and they will keep these data up-to-date using a single cron process.

[pmac]

I have an example of this working now. It is deployed to my demo. I have it pointed to my fork of the release notes data and I have tested updating the data while the website is running and it is refreshed appropriately.

The changes are only currently in my demo branch, but I should have a PR shortly.

[pmac]

/cc @jgmize @bensternthal @metadave @glogiotatidis

[glogiotatidis]

I like the idea. How do you plan to monitor the update of the multiple sqlite databases? Using a DMS trigger for each?

[pmac]

@glogiotatidis I've been thinking about that. I've developed a view on the site that will report how long it's been since the last update for all of the update tasks. Each task also has a max time (which is 5 min longer than the task's schedule). If any task is late that view will return 500 and indicate which task is late. We'll then have "k8s liveness probe" check that URL every so often and alert us if they're having problems. This seems to be a good way to go. I don't think the way DMS currently works that it would be manageable to have that many snitches.

I'm happy to go into more detail about how the cron tasks are setup and how the view monitors them. I'm very pleased with how it works. You can also just check out the PR.

[glogiotatidis]

Yep, I was thinking the DMS is sub-optimal for this one. I like the liveness probe implementation 👍

[pmac]

New PR for this is #5334.

[pmac]

Do the following to do the initial deployment of The SQLitening:

DEV

• [x] Turn off the clock process type in all but one cluster (leave on in oregon-b)
• [x] ensure the S3 credentials are set in the deis config for the dev apps in all clusters
• [x] merge sqlitening branch to master
• [x] ensure www-dev is working as expected

OREGON-B

• [x] merge master to oregon-b-prod branch which will deploy and test stage and prod in oregon-b
• [x] turn on clock process on stage and prod in oregon-b
• [x] verify all is working in oregon-b stage and prod
• [x] add oregon-b to stage and prod traffic policies

PROD

• [x] Turn off the clock process type in all but one cluster (leave on in oregon-b)
• [x] ensure the S3 credentials are set in the deis config for the stage and prod apps in all clusters
• [x] merge master to prod and tag release
• [x] ensure prod is working as expected
• [x] celebrate 🎉

[jgmize]

This is now in production 🎉