Mybinder.org-deploy: Increase limit to 200 for NEUBIAS course

[guiwitz]

Hi,

I'm Guillaume Witz from Bern University. I'm a bioimage analyst for the Microscopy Imaging Center, and I have been asked by the Network of European BioImage Analyst (NEUBIAS, a European COST action http://eubias.org/NEUBIAS/) to give a course on Image processing with Python and Jupyter (http://eubias.org/NEUBIAS/training-schools/neubias-academy-home/) in the frame of their special webinar series organised for scientist who have to work from home.

All the courses of this series are free to attend. People can register for live sessions on Zoom, and the recording of the course is then also published on Youtube. The speakers provide this training without compensation, in the frame of their jobs as bioimage analysts or researchers at various Universities.

The course on Python and Jupyter is available in interactive form thanks to the mybinder.org service and can be found here: https://github.com/guiwitz/neubias_academy_biapy. The course has the following structure: on 7th of May 15.30-17.00 (Brussels time), participants are introduced to the material (including mybinder.org) service during webinar. Students can then explore the material for a week on their own using the interactive binder session. A second webinar takes place on 13th of May 15.30-17.00 (Brussels time) where the material is further explained. At the time of writing, 500 participants registered.

We do not expect 500 participants to work at the exact same time, and it will be made clear that not all participants should connect simultaneously during the webinar. Still this repository might generate a lot of traffic. This GitHub issue serves therefore as a warning for the maintainers of this service.

Thanks for this great service that you provide!

Guillaume

[betatim]

We've increased the limit to 200 concurrent sessions. In a previous online course/webinar with about 700 participants "in the stream" (unsure how many had registered) concurrent usage peaked at around 190. This makes me think for 500 registered viewers this should work.

If people run into the limit now they can switch to colab for today and we can think about raising it for next week.

It is important to remind people that their session will time out after ~10min of inactivity. This happened at the last live event where the speaker asked people to start their mybinder.org session and then continue to talk for >10minutes. Just something to keep in mind.

[arnim]

What happens if the load balancer redirects this to a federation provider with pod_quota > total_pods currently active (i.e. gets redirected) but per_repo_quota < total number of active pods matching this pattern (gets not started)?
Could this be an issue? Do we have to change something / add patterns or think about per_repo_quota values for the fed providers?

But probably I'm thinking in the completely wrong direction here.
(cc @bitnik )

[betatim]

I think right now we get the result we want but only because we are lucky/the particular setup we have.

The federation redirector only considers if a cluster is advertising that it can receive more traffic or not. It knows nothing about per repo quotas. This means the "per repo quota" is only applied per cluster.

For example Repo A with a per repo limit of 150. In a federation with two clusters one of which has an "unlimited size" (A) and one with a size of 100 pods (B) there is the possibility of up to 250 instances of repo A running. This would happen if the fed redirector assigns repo A to cluster B at the start. Then cluster B would fill up to its limit of 100, start advertising that it is "at capacity" and the federation redirector would start sending launch requests to cluster A. Here repo A would be allowed to launch 150 instances again.

I think right now this happens ~never and when it does it isn't as bad as the extreme above. This is because all clusters (except GKE) have a total capacity significantly less than the total traffic to mybinder.org. As a result clusters tend to be full or close to their limit most of the time. This means if the federation redirector sends a new "high traffic" repo to one of the clusters it will very quickly reach the capacity of the cluster and start sending traffic to GKE. GKE then enforces the per repo limit. This seems Ok as the per repo limit is not an exact science anyway.

I think if we had two (or more) clusters in the federation with a spare capacity larger than any per repo quota we'd also have no problem. All traffic for a particular high traffic repo would get sent to one of the clusters and it would enforce the limit.

The changes to the per repo quota have to be propagated by hand to each federation member that isn't controlled by this repository. If this doesn't get done "in time" a high traffic repo could end up on a federation member where it doesn't have a higher limit and never get sent elsewhere because it doesn't exceed the total capacity of that cluster. For example if the cluster has a spare capacity of 200 pods. Right now we don't have any federation members like this though.

---

@guiwitz how did it go? What was the feedback from people?

[guiwitz]

Everything went well, thanks for your responsiveness and support! I checked usage using binderlyzer and we actually never reached the regular limit for the ~370 effective participants we had in the end. But between the two sessions we had ~50 launches per day, so I guess participants found the resource useful. I warned participants not to all start Binder during the webinar. So that warning plus having the material available on Colab as well made things manageable (given the many questions on installation, I think many of them also ran the course locally).

I have one additional question (nothing urgent, I'm just curious): with this binder repo, I regularly get a message saying that "Launch attempt 1 failed, retrying..." and usually then it runs ok the second time. This never happened to me with other repos before, but the image here is quite large as I'm downloading quite a lof of data when building it. Can the size of the image be an issue here?

Thanks again for your help!

[choldgraf]

That's great @guiwitz - for what it's worth, that sounds like similar percentages to what we've experienced in teaching workshops/classes/etc. Congrats on getting it done!

[betatim]

We could take a look at the logs if you note the exact (within a few minutes) time the failed launch attempt happens next time. I will try a few times as well to see if I can trigger it.

I think your idea/guess about the large image sounds reasonable. I haven't seen it happen in a while but if pulling the image from our registry takes a very long time then that could lead to a launch failure (I think). The second attempt then works because the image is in the cache on the node.