Habitat: Data migrator from Redis-backed Builder to Postgres-backed Builder

[reset]

After all services are backed by PostgreSQL we need to create a migrator to move all data stored in Redis into PostgreSQL. We need to target just the dataset that is currently available on live, it isn't necessary to migrate any other environment's data.

[nellshamrell]

Notes from initial conversation

• need to migrate data from redis store to postgres

Possible implementation

• create new crate for migration
• should output single binary that takes a source and destination
  
  
  
  • source = redis datastore
  
  
  • dest = postgres DB instance
  
  
• named schema in postgres
  
  
  
  • copy of "normal schema"
  
  
  • we preshard data 128 times
  
  
  • bucket of DB migrations
  
  
• make sure to check version of habitat running in prod, will need to translate that version's protobufs into current version's protobufs.

[nellshamrell]

Looks like app.habitat.sh is currently running 0.11.0-dev/20161101174906

[reset]

@nellshamrell I changed the way we did versioning of our servers to using a sequenced version of our git commit hash sometime after the release which is running in production. I did that so we would know exactly which commit is where the service was built from. This is super useful for services since you want to deploy them as often as possible.

Unfortunately this means that there's no way to identify 100% which commit was used to build what's in production. I would maybe recommend using this tag to work off of

[mwrock]

one thing to keep ion mind is that the redis store will not have target info and will also lack the key contents for private and public keys. The missing targets is trivial, everything will be linux but the tool should inject that target string. We will have to iterate the actual key files on the source and dump the keys into postgres.

[nellshamrell]

is there an easy way to find all the redis data that needs to be migrated to postgres in the current production installation of Habitat? I imagine everything involving origins as well as the depot needs to be migrated, but is there more I should be looking for?

[adamhjk]

You need origins, sessions/users, packages, at the very least.

[nellshamrell]

Just did a skim of the builder-components in 0.12.0

Here are the components with a dependency on Redis:
builder-dbcache
builder-depot
builder-jobsrv
builder-protocol
builder-sessionsrv
builder-vault

[adamhjk]

We won't care about dbcache when we are done; it's all boilerplate implementation for Redis backed data structures.

[nellshamrell]

Just went through the data store files of all the builder components which depend on redis. Here are db things I see as needing to be transferred to Postgres. Are there any that are not actually needed, or does it seem like anything is missing?

builder-depot

• packages
• channels
• origin_keys

builder-jobsrv

• jobs

builder-sessionsrv

• accounts
• feature_flags
• sessions
• github_accounts

builder-vault

• origins
• origin_secret_keys
• origin_invites
• projects

[nellshamrell]

My approach for creating and testing this:

1) Create test redis store identical to the one used in 0.12.0, populate with some sample data
2) Create test postgres store identical to the one used in the latest release
3) Verify that core functionality works with the redis store
4) Move data from redis store to postgres store (transforming protocols, etc. as necessary)
5) Verify the core functionality still works with the postgres store

[adamhjk]

I don't think you'll need to bring feature_flags over, as they are now configuration options.

[nellshamrell]

Upon some further investigation I'm realizing how extensive the differences between 0.12.0 and our current master are. I've altered my implementation plans slightly and would welcome any feedback!

1) Copy current prod redis db (will need access to our AWS account to do this)
2) Create test redis db from this copy
3) Create test postgres db based on the schema in master
4) Create migrator that transforms the data as needed from the redis store (including correcting protocols, etc.) and inserts into the postgres db
5) Manual(?) testing of all depot functions involving that data

(All of this will be done on copies of infrastructure, not on the prod infrastructure itself)

[mwrock]

I think in order for this to work, both the source and target depots will use the same disk storage because the depot "data" is really a combination of the metadaya in redis/pgres amd the depot filesystem where the actual harts reside. Is this a correct assumption? That we plan to migrate on the same instance? If not, it seems like targeting a filesystem as the source instead of redis would be more effective since the files have everything needed to recreate the db.

[nellshamrell]

Prototype: https://github.com/nellshamrell/redis-to-postgres-migrator

[nellshamrell]

(These milestones and tasks are by no means set in stone, this is solely to help organize my thoughts).

Milestone 1 for this: accounts able to be copied from redis store to postgres store

• [x] Set up new habitat_migrator project
• [x] Copy builder-sessionsrv datastore from Habitat 0.12.1 into a module in habitat_migrator project
• [x] Set up test redis store
• [x] Create functionality in main.rs to pull account entries from the redis db, decode them using the 0.12.1 builder-sessionsrv datastore code, and print out values for account attributes, etc.
• [x] Set up test postgres db
• [x] Copy builder-sessionsrv datastore code from master
• [x] Send account data from main.rs to master builder-sessionsrv datastore
• [x] master builder-sessionsrv should insert data with appropriate protobuf into the postgres db

Milestone 2: sessions and github_accounts able to be copied from redis store to postgres store

Milestone 3: origins, origin_secret_keys, origin_invites, and projects able to be copied from redis store to postgres store (NOTE - secret_keys may be stored in filesystem, those must be copied over too)

Milestone 4:
packages, channels, origin_keys able to be copied from redis store to postgres store

Milestone 5: jobs able to be copied from redis store to postgres store

[nellshamrell]

I notice that in master, accounts have attributes like is_admin, is_builder, and is_builder worker. I don't see these attributes in 12.1.0. When taking the data from Hab prod (which is running 12.1.0), am I good with setting these attributes to false by default?

[nellshamrell]

Current checklist of things to transfer

• builder-depot
• [x] packages

• [x] origin_keys

• builder-sessionsrv

• [x] accounts

• builder-vault

• [x] origins
• [x] origin_secret_keys
• [x] origin_invites

[nellshamrell]

Migration checklist

Copying data from prod

• [x] Copy dump of of current prod redis db by ssh'ing into it, entering redis cli, and running save
• [x] scp dump.rdb from current prod to local workstation
• [x] scp dump.rdb from local workstation to datastore node of "nell" environment
• [x] stop redis service on datastore node of nell environment, copy dump.rdb to appropriate place, start redis service again
• [x] Take snapshot of EBS attached to prod (/dev/sdf)
• [x] Spin up new EBS volume from snapshot
• [x] Attach new volume to "nell" datastore node
• [x] Mount new volume on "nell" datastore node

Performing Migration

• [x] set $KEY_ROOT env variable to /mnt/mounted_dir/svc/hab-builder-api/data/
• [x] Set $PGPASSWORD env variable to postgres password (you can find this in the config.tomls for builder-sessionsrv or builder-originsrv)
• [x] Run migrator on builder-datastore node

Getting data into acceptance/prod environment

• [ ] Spin up new volume from the EBS snapshot and attach to acceptance/prod env builder-datastore node
• [ ] Mount new volume
• [ ] Take copy of Postgres db off of the nell environment
• [ ] Load copy of Postgres db onto builder-datastore node in new acceptance/prod env