Mybinder.org-deploy: Automated push to deployment via GitHub PRs

[yuvipanda]

Deployments should be as automated as possible. This makes it easier for lots of people to do them quickly and without making arbitrary mistakes.

I've done a bunch of 'push to deploy' setups over the last few months, and here's what I think it should be like (assuming we've switched beta to mybinder.org at this point):

1. We have a 'staging' and 'prod' branch, and they each correspond to staging.mybinder.org and mybinder.org
2. When someone wants to make a change, they make a PR to the staging branch. Travis runs some tests to make sure things look ok.
3. Someone merges the PR. This triggers an actual deployment via travis, which passes or fails. It is the responsibility of the human who merged the PR to make sure the deployment succeeds, and if it doesn't, debug it + roll back if necessary.
4. If (3) went well, another PR is made merging staging into prod branch. Travis runs tests again.
5. The PR from (4) is merged.

This is the model we've successfully followed in https://github.com/berkeley-dsep-infra/datahub/ for a while now. It has given us the following advantages:

1. Much much less error prone than manual deploys, which might fail for any number of environmental reasons.
2. Most adversarial effects are caught in the staging deployment, so less downtime for the production one
3. Most config can live in common.yaml, with minimal staging / prod specific ones (such as hostnames and secrets) in their specific files. This prevents the biggest enemy of having useful staging environments - drift.
4. Leaves a very clear audit trail of both who merged what (in PRs) as well as output of each deployment (in travis).
5. No need to figure out-of-band secret key transfer mechanisms. Whoever is able to merge PRs in the repo has rights to do deployments.

It also does have some negatives:

1. Flip side of advantage 5 - anyone who compromises travis has compromised our infrastructure, including all of travis employees. Is this acceptable?
2. ??? (can't think of any!)

IMO the positives outweigh the negatives, and we should JDI. It will also free up good chunks of my time to focus on other issues :)

[yuvipanda]

/cc @minrk @willingc @choldgraf objections to this plan?

[choldgraf]

Agree - we'll need to document that workflow for the team but it sounds like a net positive to me for sure. If it works well for datahub then I think it's worth trying here.

[minrk]

I think this is a great plan. For details, I assume that 'staging' is actually the 'master' branch, and then we push to 'prod'? This might also be a case for trying out protected branches on this repo.

[willingc]

Big +1. No objections as this worked well for other Django projects that I've worked on (whether automated through travis or deployment scripts) when master -> staging -> prod.

A few items to think about:

• Do we use master, staging, prod or just master/staging, prod? I guess it depends if we wish to do some load testing on staging (or if staging serves a small sample of actual users) before release to prod?
• > It is the responsibility of the human who merged the PR to make sure the deployment succeeds, and if it doesn't, debug it + roll back if necessary.
  
  
  
  • I would expand to the responsibility of 1) the human who merged and 2) the author of the PR.
  
  
• I'm assuming that we will have a bot of some sort to do rollbacks easily and consistently.

[yuvipanda]

@minrk For Berkeley (and in general) I prefer calling the branches 'staging' and 'prod', and have them correspond throughout - to cluster names, namespaces, release names, domain names, etc. I'd like to do the same here. +1 for using protected branches - we've those too!

[yuvipanda]

@willingc I think we should just do staging and prod now, and not do any traffic redirection yet. +1 on having a specific 'you have deployment power, now here are your responsibilities' doc. For rolling back, we just revert the PR and merge that - I think that's sufficient for now.

[yuvipanda]

Some common tasks we should have a good documented workflow for:

1. Just changing the config items in this repository & pushing it out
2. Making a change in the http://github.com/jupyterhub/binderhub repository & then pushing that out without any config changes.
3. Making a change in the binderhub repo + related changes in this repo & pushing them out atomically.

A PR should be our atomic unit of deployment / rollback.

[yuvipanda]

A very big chunk of this is done! There's some preliminary documentation at https://github.com/jupyterhub/mybinder.org-deploy#deploying-a-change. Here's an example of deploying something to staging: https://github.com/jupyterhub/mybinder.org-deploy/pull/15. And then deploying that to beta: https://github.com/jupyterhub/mybinder.org-deploy/pull/16.

I'm unsure how to get rid of the merge commits, unfortunately. 'rebase and merge' on the github UI still changes the hash, so not sure what to do here.

I've opened https://github.com/jupyterhub/mybinder.org-deploy/issues/11 separately for writing up deployment guidelines & responsibilities.

[yuvipanda]

I've also opened up #10 which should wait until https://github.com/jupyterhub/binderhub/pull/135

[minrk]

I'm unsure how to get rid of the merge commits, unfortunately.

What would be the goal of discarding the merges? It seems like preserving them would show a trail of what's been deployed to prod vs staging, which would otherwise be discarded.

[yuvipanda]

@minrk I guess it's ok. I was only slightly annoyed at them because it means beta and staging branches will never actually be the same (since there will always be a merge commit on top in beta) but I guess that's ok.

[yuvipanda]

Now deployment won't mark as succeeded until:

1. All pods have finished creating (5min timeout)
2. An end to end test that tries to launch a known good github repo onto binder succeeds.

This should give us more confidence in our deployments!

I'm going to close this issue now, since we've separate issues for more specific things we wanna do.

\o/