Mybinder.org-deploy: Use OVH as a second cluster

[choldgraf]

At a recent meeting @yuvipanda brought up the idea of using OVH basically as a second cluster that runs off of mybinder.org. The idea would be to use the location of users to decide which cluster to send them to. the GCP and OVH clusters would then split the users based on where they were located. Deploys would basically be handled in the same way: we'd include OVH deploy lines to our CI, so both clusters would be updated at the same time.

This issue is to discuss some of the pros and cons of this approach. From my perspective:

Pros

• We can have an N+1 BinderHub deployment for mybinder.og
• We can use this to experiment with federation and UI around this
• We could reduce computing costs
• We could have mybinder.org deployments in Europe, and hopefully grow the Binder community a bit

Cons

• There would now be two clusters to administer (e.g. twice as many random kubernetes bugs that could pop up)
• The user load could be more unstable (e.g. if one of the clusters go down, does that mean that suddenly the other cluster doubles its traffic all at once?)

anybody else have thoughts on this? cc @betatim and @minrk and @sylvaincorlay who were also in on the meeting

[yuvipanda]

Thanks for writing this up, @choldgraf!

There would now be two clusters to administer (e.g. twice as many random kubernetes bugs that could pop up)

This is my biggest concern. A large part of this depends on how mature OVH's managed Kubernetes offering is.

The user load could be more unstable (e.g. if one of the clusters go down, does that mean that suddenly the other cluster doubles its traffic all at once?)

There are ways to manage this. Mostly, we try to distribute load across the two 'data centers' in the following ways:

1. Based on geographic location (with GeoDNS or a GeoIP aware Load Balancer)
2. If one of the data centers is 'full' or 'unhealthy', we always send people to the other data center
3. If both data centers are 'full' or 'unhealthy', we error out.

This requires us to define what 'full' and 'unhealthy' mean. This is generally useful to do anyway, but is a requirement for a multi-DC set up.

This also lets us add more 'Data Centers' in the future - going from 1 to 2 is harder than 2-3, etc.

[yuvipanda]

The user load could be more unstable (e.g. if one of the clusters go down, does that mean that suddenly the other cluster doubles its traffic all at once?)

This feels like an instance of the Thundering herd problem. There are various ways to mitigate this explicitly, and we can do so.

[yuvipanda]

On thinking more about this, I think the way to do this is with GeoIP on a load balancer, rather than with GeoDNS.

1. The two JupyterHubs should live in two different URLs.
2. The two BinderHubs should be accessible by the same URL, behind a global load balancer
3. User request comes in to binderhub, we check if there's already a cookie that says where this user should go to. If not, we determine binderhub to send user to based on GeoIP & cluster health and set a cookie.
4. The binderhub on google cloud knows to send users to the JupyterHub on google cloud, and the binderhub on OVH knows to send users to the JupyterHub on OVH.

This means we need a loadbalancer only for binderhub, which doesn't receive that much traffic. The heavy lifting is done by the JupyterHubs, which would not need a load balancer.

We can then run this BinderHub load balancer in both locations, and do a failover via DNS if there's a total outage.

[minrk]

OVH folks have kindly provided us a staging cluster for getting started with this, so we can discuss a plan in more detail.

Let me (or anyone else with credentials) know, and we can get you credentials for looking at the staging cluster.

Currently folks who have requested credentials and should have them: @minrk, @betatim, @yuvipanda

We will probably want to add the encrypted credentials to this repo when we're ready, so anyone with the creds to view secrets in that repo should be able to edit them without point-to-point sharing of encrypted files.

/cc @mael-le-gal and @jagwar our OVH contacts

[betatim]

It would be great to have a federation consisting of several clusters:

• everything controlled from one repository
• clusters are added/removed from the federation (semi-)automatically depending on their health/load
• mybinder.org redirects users to each cluster in the federation
• each cluster in the federation is directly accessible (for example as COUNTRYCODE.mybinder.org)
• users see which cluster they are being redirected to in their browsers address bar (mybinder.org isn't a proxy but a redirector)
• each cluster can customise (lightly) the HTML page served by BinderHub
• https://archive.analytics.mybinder.org/ continues to provide data on all launches (and builds) across clusters

This is my wish list, what do you think?

---

I think it is important that we expose where/who the clusters are to a) make it obvious to everyone there are multiple clusters (and they can contribute one!) b) users can choose to use only one of the clusters.

We should automate as much as possible of the operations related to adding/removing/health checking a cluster. Maybe plan an option for a manual override?

Anyone willing to run a public BinderHub without restrictions on which repos can be launched can join the federation. We should allow clusters to have higher/lower per repo limits to make it easier to join with a small cluster.

[choldgraf]

Great news! re: centralization and federation, I think we should discuss at the next team meeting! In general Tim's thoughts seem reasonable to me.

Wouldn't we just want to give all of the Binder team members (or only red team members?) direct access to the cluster? I'm not sure if that'd open up potential challenges around billing etc

[betatim]

I'm not sure if that'd open up potential challenges around billing etc

For our current cluster I have "full access" to the cluster but I can see essentially nothing related to billing (who is paying, how it is being paid, how much we have paid, what we spent it on, etc)

[choldgraf]

hmmm, that's not good :-( I didn't realize that nobody else can see that information. You can't see a "reports" tab under hamburger menu -> billing -> linked billing account -> reports? It's not super interesting now that we're running on credits, but if you go back in time it has a readout of how much we were paying.

(if anybody knows of a way to share billing information on google cloud, I'd love to work w/ you to get this information more easily available to the team).

[betatim]

I'm totally fine with the bits I can see. The reason I brought it up was to point out that "givng access" seems to happen on multiple levels and we can have people changing the cluster that can't see the billing info.

[yuvipanda]

A stepping stone to this is probably https://github.com/jupyterhub/mybinder.org-deploy/issues/903

[jagwar]

Is there any news with this subject on your side ?

[betatim]

This thread and the ones linked from it are a good reflection of the state of work on this.

[SylvainCorlay]

@mael-le-gal, @jagwar do you have any update on your side?

[SylvainCorlay]

Hey all, I have created a doodle poll to set up a time for a hands-on discussion with the OVH team on what could be the next steps

https://doodle.com/poll/88e4vxkw4epikn7r

cc @choldgraf @yuvipanda @betatim @willingc

[willingc]

FYI @Zsailer @lheagy @consideRatio

[SylvainCorlay]

Hey all,

Here is the result for the poll:

• Apr 2nd
• 5:00PM - 6:00PM Paris Time (8:00 AM - 9:00 AM Pacific Time)

I will send information on the connection details! (We will probably use Jupyter's zoom account if there is no conflict).

cc @Zsailer @lheagy @consideRatio @choldgraf @yuvipanda @betatim @willingc @mael-le-gal @jagwar @minrk

[choldgraf]

sounds good. Do we have an agenda for this meeting?

[jagwar]

Nice !
cc @jqueguiner

[SylvainCorlay]

sounds good. Do we have an agenda for this meeting?

I would say that the content of that issue constitutes a good agenda. I will put together a google doc for discussion items.

Personnally, I would love if we were able to identify a quick win that is possible without too much investment from the core team.

[lheagy]

:wave: Hi @SylvainCorlay: do you have details available for how to connect? Thanks!

[betatim]

I am at https://calpoly.zoom.us/my/jupyter

[SylvainCorlay]

thanks @betatim I thought I had posted the link already.

[SylvainCorlay]

@choldgraf @yuvipanda @betatim @willingc we are connected at https://calpoly.zoom.us/my/jupyter

[willingc]

I am sorry that I missed you this morning. Tied up with preparations for Guido's Language Creator panel this evening. Hope the call went well.

[betatim]

We discussed the where the binder team is currently thinly stretched and how OVH could contribute there. In particular that we don't have the human resources to operate two clusters and that OVH has experience with that. One outcome is that they have a BinderHub up and running that they want to direct traffic to from mybinder.org to establish a second cluster. Related to this the question in #144.

Some notes from the meeting https://docs.google.com/document/d/1F8uuPnqeC102R-LMFPgdnuLNzNQ8v36JuHkyuYKaoX8/edit?usp=sharing

[choldgraf]

Hey all - just a ping that we now have a page that discusses sub-domains on mybinder.org. It would be great if we could push forward with the OVH deployment so that we can add our first sub-domain! I think from the notes that @jqueguiner was a good person to ping for this.

[betatim]

Has anyone heard from OVH or knows the domain at which their current BinderHub deployment can be reached?

[jqueguiner]

hi guys @jagwar will take the lead for the implementation ;-)

[choldgraf]

@jagwar looking forward to have you involved! Please do reach out, come say hi in the discourse page, etc! Our goal is not simply to get an OVH cluster here, it's to welcome some new folks into the Jupyter community :-)

[jagwar]

Hi ! It will be a pleasure :)
Can you provide me the discourse page link ?
It's currently: Not Found

[consideRatio]

Hi @jagwar :heart: :D

https://discourse.jupyter.org

[choldgraf]

oops - I forgot the HTTP :-)

[jagwar]

Done :)
Do you want to transfer this topic on discourse ? Or can we continue here ?

[betatim]

Do you want to transfer this topic on discourse ? Or can we continue here ?

I'd continue here.

For me the question that needs answering is: What are the next two steps to take?

For me these things are on the list:

• test drive the OVH BinderHub instance, for this we need to know the URL at which to reach it
• assign a sub-domain for the OVH hub (-> someone from OVH opens an issue or PR in https://github.com/jupyterhub/team-compass after reading https://jupyterhub-team-compass.readthedocs.io/en/latest/binder/subdomains.html)
• discuss how the OVH instance is configured (is there a public repo or ...?) so that we can discuss how to keep the two instances inn sync.

[jagwar]

Thanks @betatim

First, we will order mybinder.ovh and provide you OVH BinderHub Instance on this URL.
I have open this issue: https://github.com/jupyterhub/team-compass/issues/154
We can do a PR on https://github.com/jupyterhub/mybinder.org-deploy with our configuration

Let us know if it's OK

[choldgraf]

That sounds good! As @betatim mentions I think we should check these boxes before finalizing ovh.mybinder.org:

• [ ] Folks try out the OVH BinderHub deployment and make sure it all works as planned (@jagwar please let folks know when there's a public URL)
• [ ] The configuration of the OVH BinderHub should either be made public or at least explained somewhere
• [ ] Make sure we've got a plan for keeping the two clusters up-to-date. (I'm wondering here if @henchc 's @henchbot might be useful for this)

Anything else?

[mael-le-gal]

Hello Guys,

Sorry for the delay, we took some time waiting for the ordering of our *.mybinder.ovh certificate.

You can now access the instance on the following URL : https://binder.mybinder.ovh/

Let me know if you spot anything unexpected.

For the deployment configuration I took example on your other repository : https://github.com/jupyterhub/mybinder.org-deploy I guess I could submit a Merge Request so that you can see the changes ?

[SylvainCorlay]

You can now access the instance on the following URL : https://binder.mybinder.ovh/

This is great news!

For the deployment configuration I took example on your other repository : https://github.com/jupyterhub/mybinder.org-deploy I guess I could submit a Merge Request so that you can see the changes ?

That sounds like a good plan to me. cc @minrk @betatim

[mael-le-gal]

There are several other points that we should discuss :

• If I push a PR for the configuration, I’ll need to be in the git-crypt users of the project so that I can upload the ovh related secrets. Do you need something from me to do that ?

• What about the maintenance plan ? internally we used a solution called OpsGenies for supervising the states of our infrastructures. Until now do you have something similar to ensure the run of the platform or do you want us to help you in that direction ?

• As @choldgraf mention we need to defined a plan for keeping the two clusters in sync.

• Last we should talk about the http redirection once everything will be ok.

I think these 4 points are the next priorities and needs to be discuss, maybe we should plan a meeting what do you think ?

[betatim]

1. sharing secrets: we use ssh-vault for sharing. Does that work for you? @minrk @yuvipanda can you give a thumbs up (or down) to sharing our secrets? Related to this: should we setup accounts on the OVH k8s cluster for the mybinder.org team so we can run kubectl commands for debugging? As well as vice versa adding the OVH team to the google cloud account we use.

2. for monitoring we have http://grafana.mybinder.org/ (we should setup a prometheus+grafana like this for the OVH cluster too IMHO) and http://isthehubup.herokuapp.com/ which also posts on https://gitter.im/jupyterhub/mybinder.org-deploy when it thinks that something is down. I can add the OVH instance to the pages it monitors and tries to build repositories on. Other/more sophisticated monitoring and alerting would be welcome though.

3. for keeping things in sync, I think deploying from one shared repo will already get us a long way towards that. We need to update the relevant sections in our SRE guide to mention the new files (if we can't centralise everything into one file)

4. for the HTTP redirects I think we need to write a bit of code, at least I couldn't find an off the shelf solution that could do health checks and answer "is that hub up to date?" with what we have now. So maybe we open a new issue for that?

[betatim]

ps. I just tried https://binder.mybinder.ovh/v2/gh/binder-examples/requirements/master and it worked :)

[choldgraf]

re: @betatim's latest comment, I noticed that while the build+launch seems to work, the image cacheing doesn't seem to work. It re-builds every time. e.g. I just tried hitting

https://binder.mybinder.ovh/v2/gh/binder-examples/requirements/master

a bunch of times and each time it re-triggered a build. Do you have the same behavior @betatim ?

[mael-le-gal]

sharing secrets: we use ssh-vault for sharing. Does that work for you? @minrk @yuvipanda can you give a thumbs up (or down) to sharing our secrets? Related to this: should we setup accounts on the OVH k8s cluster for the mybinder.org team so we can run kubectl commands for debugging? As well as vice versa adding the OVH team to the google cloud account we use.

Yes we can use ssh-vaultfor sharing one to one secrets but it will be better to add me in the git-crypt project so that I can upload the ovh kubectl file that will be used by your Travis workflow for deploying on our cluster.

for monitoring we have http://grafana.mybinder.org/ (we should setup a prometheus+grafana like this for the OVH cluster too IMHO) and http://isthehubup.herokuapp.com/ which also posts on https://gitter.im/jupyterhub/mybinder.org-deploy when it thinks that something is down. I can add the OVH instance to the pages it monitors and tries to build repositories on. Other/more sophisticated monitoring and alerting would be welcome though.

Ok I didn't activate the grafana for now but I'll try to add it also on our side, so that both instance will be exactly the same.

[mael-le-gal]

re: @betatim's latest comment, I noticed that while the build+launch seems to work, the image cacheing _doesn't_ seem to work. It re-builds every time. e.g. I just tried hitting

https://binder.mybinder.ovh/v2/gh/binder-examples/requirements/master

a bunch of times and each time it re-triggered a build. Do you have the same behavior @betatim ?

Indeed it seems I have that problem too. Do you have any idea about what I missed in the setup ?

[betatim]

Yes we can use ssh-vaultfor sharing one to one secrets but it will be better to add me in the git-crypt project so that I can upload the ovh kubectl file that will be used by your Travis workflow for deploying on our cluster.

Nods, we use ssh-vault to share the key that is used by travis to use git-crypt to decrypt the contents of the secret part of the repository.

[betatim]

a bunch of times and each time it re-triggered a build. Do you have the same behavior @betatim ?

Yes, that also happens for me.

[yuvipanda]

+1 for sharing with ssh-vault. As more people use the key, ssh-vault becomes more insecure. See https://github.com/berkeley-dsep-infra/datahub/issues/596 for a detailed analysis of the problem and a proposed solution. However, right now, for this use case, ssh-vault gets a :+1:

[betatim]

As more people use the key, ssh-vault becomes more insecure

The problem isn't ssh-vault, it is the fact that we use one single symmetric key to decrypt the secrets no? Either way, we should address this at some point and I created a new issue for this: #969

[yuvipanda]

The problem isn't ssh-vault, it is the fact that we use one single symmetric key to decrypt the secrets no?

Yep, exactly. Thanks for opening the new issue!

[choldgraf]

@mael-le-gal could you post a thread in the Discourse page about this issue https://discourse.jupyter.org/c/binder/binderhub and we can try to pin down why your binderhub is re-building each time?

[mael-le-gal]

Hello everyone,

Just to let you know, I just pushed a WIP Pull Request here https://github.com/jupyterhub/mybinder.org-deploy/pull/970 containing the configuration I used on the ovh deployment :

@choldgraf yes I'll post a thread on the discourse page for my cache problem

[mael-le-gal]

Now that there is a related PR maybe we could close this issue and discuss the remaining problems directly on the PR feeback ?