Mybinder.org-deploy: separate GCE project for staging

[minrk]

In the move to put the beta on binder-prod, a separate binder-staging project was created. I can't find any reasoning behind this, and it makes management way more complicated because admins has to be added/removed twice, logs can't be combined, billing is separate, etc.

Shouldn't there be just one GCE project for Binder, with separate clusters for prod/staging?

[yuvipanda]

I separated them out both for access control + dev reasons. With a separate project, we can easily hand out credentials to staging to people for use with https://github.com/jupyterhub/binderhub/pull/168. AFAICT, there's no way to hand out credentials to just one cluster (IAM doesn't seem to let you restrict which clusters you can have access to). This also helps with billing limits. Plus if there's a security breach in staging, that doesn't affect prod. It also helps us separate the built images, which we can not without having these two be separate.

IMO, staging and prod should be this separate. The list of people who are admins on staging should be larger and looser than list of people on prod. It's ok to screw up the whole staging project. Billing is separate is a feature IMO.

If other folks feel really strongly about this I am happy to move it back to one project, but only if those are really strong feelings!

[yuvipanda]

I had thought I had mentioned separate projects as well in https://github.com/jupyterhub/mybinder.org-deploy/issues/83 - apparently I hadn't. That's my bad, there weren't enough specifics there.

[yuvipanda]

First comment I wrote after waking up, it reads a little harsher than I intended! I'm happy to merge it together if in practice how we use it doesn't match up to this ideal, but I'd love for us to give this full separation a shot.

Maybe I can add some more docs on how to manage switching between these? Once you have the credentials for both the clusters locally then for kubectl it doesn't matter if they're in the same project or not. kubectl config use-context should work...

[minrk]

It doesn't seem worth the admin overhead of two projects to me, and it doesn't seem to solve a problem that exists. When would we grant credentials to staging but not prod?

Billing is separate is a feature IMO.

In what way, when billing should always be handled the same for both, and any changes or updates to accounts must be always be done twice?

[willingc]

Just catching up from vacation. While I see benefits of full separation of prod from staging and from development, there are definite pros and cons in terms of complexity and managing the infrastructure alongside the development. We're scaling out things on a number of fronts, and the ability for team members to understand the process, step in as needed, and have credentials to respond in a service outage to prevent a bus factor of one are all important considerations.

Lots of exciting growth and changes are going on. While documentation will help, we should find a way to have more communication, collaborative discussion, and move the entire team in the same direction and leverage all of our strengths. The challenge is how to do this while maintaining the positive momentum without burning folks out or blindsiding them.

[yuvipanda]

It looks like @minrk has strong feelings about this too. I had assumed this would be fairly uncontroversial (and implicit in https://github.com/jupyterhub/mybinder.org-deploy/issues/83). Clearly I'm wrong this time too. While I still believe that full separation is important to have from the beginning (and much harder to tack on afterwards), I've also come to experience that @minrk and @willingc's ideas around complexity have generally born out better than mine :D So I'm happy to help consolidate this into one project (probably binder-prod), although we should be very careful around who we give access to in that case. We can not give people access to just one Kubernetes Cluster.

It would be great if someone else takes on this task of consolidation :) Am happy to review PRs.

@willingc In this case re: communication, my struggle was that a large number of decisions were made in quite real time when we moved the project from binder-testing (such as: two separate projects, removing any common secrets, nginx version upgrades, SSD sizes for nodes, autoscsaling limits, etc). They were mostly uncontroversial, and I'm not sure where the line for 'I am just going to do this' vs 'I am going to wait at least a 24h cycle' is for things like this. The time zone differences within the team don't help either when it comes to administrative things like this that involve lots of button pushing & command-line running (rather than PR reviewing & merging). I try to make sure that the rest of the team is aware of my actions beforehand whenever possible (that was my intent when I filed https://github.com/jupyterhub/mybinder.org-deploy/issues/83, although it could have used more details).

Personally, it's also sometimes demotivating to spend 2 days making sure the move happens properly and only see negative feedback on the stuff that I screwed up - this is a constant issue in doing operational tasks, communication & non-code things anyway (it is far easier to notice the screw up than all the things that went great), and that constantly leads to more burn out & feelings of being underappreciated...

anyway, tldr is I apologize if I blindsided folks - didn't think this was a controversial decision that would have required team wide discussion, long-distance-remote-team-stuff is hard and idk what to do but I think we'll get better over time.

[willingc]

Personally, it's also sometimes demotivating to spend 2 days making sure the move happens properly and only see negative feedback on the stuff that I screwed up - this is a constant issue in doing operational tasks, communication & non-code things anyway (it is far easier to notice the screw up than all the things that went great), and that constantly leads to more burn out & feelings of being underappreciated...

I'm not saying that you screwed up. I believe you that you acted on your best judgment while doing a big migration. As a team, we need better communication and/or better separation of development and operations. There are a lot of moving pieces and repos which starts us off with a high level of complexity.

I think most of the confusion and frustration stems from operations/deployment of binder/site reliability being mixed in with development work.

A couple of thoughts on what may help the team going forward:

• create an inventory of all the repos
  
  
  
  • related to mybinder deployment and development
  
  
  • related to binderhub development
  
  
  • related to jupyterhub development
  
  
• create an infrastructure team and infrastructure guide for mybinder deployment
• have a weekly team meeting with agenda sections for:
  
  
  
  • mybinder deployment and site reliability
  
  
  • mybinder development
  
  
  • binderhub development
  
  
  • z2jh development
  
  
  • jupyterhub development
  
  

Yes, remote team stuff is hard. I agree that we'll get better over time.

[choldgraf]

I agree with @willingc 's ideas re: team communication. I'm +1 on a weekly sync meeting.

re: this specific issue, I am also hoping it was a one-off challenge due to the last-minute nature of the switch. While we were at the Binder workshop in Davis, I got an email from Jeremy Freeman saying "HHMI is pissed at me for not discontinuing payments to Binder, we need to make this switch ASAP", which triggered 2 days of chaos + Yuvi working behind the scenes to make this switch so mybinder.org didn't go black.

Barring more "oh crap that's a big fire we gotta put it out now" moments, I totally agree that a more deliberative process is a good idea. As Yuvi mentions, I think we're all trying to figure out the right balance here.

[yuvipanda]

+1 on weekly sync meeting too! I think that'll definitely help!

[minrk]

@yuvipanda sorry for the only negative feedback. You did a great job making the transition seamless, and one of the awesome things about that was the fact that the deployment infrastructure is pretty insensitive to whether there are two projects or one, which is why I didn't notice it in the PR. That's pretty cool! I should have started with saying that the transition was smooth and very well executed, and I just had one question about one of the decisions along the way. I don't want my objection to seem stronger than it is. You didn't do anything wrong.

In the same way that I felt separating the projects was a premature optimization, I think rejoining them now that they are already working and separate would be as well. Since the two projects are already established and the tooling already works great, I suggest that we stick with the current two-project deployment and only consolidate to one if the separate accounts really do cause an issue down the line. If we don't have any issues, then there's nothing to fix!

As a data point, I did a couple of test deployments to for the https PRs, and I did these on the binder-staging project. I did feel a little more secure that I wouldn't mess anything up by deploying on staging instead of prod.

A weekly (or bi-weekly) sync would be great, as things are moving fast and we are a distributed team.

[minrk]

@yuvipanda by the way, I was just going through issues now and you did mention the two projects in the TODO Issue. I just missed it while I was catching up on things.

[willingc]

Closing this as resolved. Created Dropbox paper document for planning and weekly (bi-weekly) sync meetings https://paper.dropbox.com/doc/Development-planning-for-JupyterHub-and-BinderHub-Q7Y2CMbgICAEjabtFvxlP