Mybinder.org-deploy: Work around launch rate failures whenever a new node is autoscaled up

[yuvipanda]

The cluster autoscaler adds a new node only when the current cluster is 100% full. This means for several minutes all launches fail as the new node has to come up, have images slowly populate themselves on it, etc.

This leads to several minutes of launch failures whenever a new node is added.

We should find workarounds for this. The underlying fix is to have the autoscaler scale up when the cluster is 90% full, but unfortunately that isn't gonna happen for a while thanks to https://github.com/kubernetes/autoscaler/issues/148

[yuvipanda]

I think we need some process in here that does things like:

1. If cluster is at 90% utilization, trigger a new node somehow
2. as we wait for a new node to come up, maybe delete some of the oldest running pods to make space for a few until the new node warms up
3. Pre-pull some images on to new nodes when they come up somehow

[yuvipanda]

@willingc I removed the 'needs: upstream action' label and retitled the issue to make that clearer, since this issue is to track the workaround we need to perform until / if upstream action ever arrives.

[willingc]

Can we possibly 1. alert, 2. create a cordoned node, 3. prepull, 4. uncordon? @yuvipanda

[yuvipanda]

Can you explain that a little more, @willingc?

[willingc]

@yuvipanda I probably used the wrong terms. But the basic idea is similar to Bootstrapping a node in Learn K8 the Hard Way

[choldgraf]

I think that @willingc 's approach sounds like the right idea, I take it to mean:

1. Create an alert for when the cluster is at NN%
2. When we get the alert, create a new node but don't direct user pods to it
3. Pull in the images for some number of binders onto the node
4. Once that's done, start letting users in.

Though I'm unsure what pre-pulling means in this case. In a binder context, the pulling would only happen once a user pod is created for the node, right?

Maybe we could pre-pull the images for the top N repositories over the last 1 hour or something?

[betatim]

The tricky part is that we can't create a new node unless we disable the autoscaler :(

When the autoscaler is on and you add a node by hand the AS will delete that node again because "it isn't needed". You can get around this by upping the minimum size of the node pool, but this means about a minute or two where the k8s API is unavailable.

Can we create a "special" pod that requires so much RAM/CPU that it triggers the autoscaling? We launch it when we reach the trigger point, AS adds a node, the pod watches the cluster utilisation and when it is at 99% it stops running, so now there is room for more user pods and hopefully the time window is short enough that the autoscaler doesn't spot there is a spare node that "isn't needed". (It checks every 10minutes.) The special pod could also do some pre-pulling?

[betatim]

Or, we make a trigger that deletes the oldest N pods just when the new node spawning has started, buying us a few minutes at the cost of booting old pods out that might still be in use. This relies on being able to inspect the k8s cluster so that we can be notified when a new node has just been spawned.

[willingc]

@betatim We should look a bit more at more than one node pool (autoscaled; overflow) too.

[yuvipanda]

I need this for the EdX hub too, and
https://github.com/berkeley-dsep-infra/data8xhub/issues/7 has info on
various approaches

On Thu, Feb 22, 2018 at 1:49 PM, Carol Willing [e-mail]
wrote:

@betatim https://github.com/betatim We should look a bit more at more
than one node pool (autoscaled; overflow) too.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/mybinder.org-deploy/issues/474#issuecomment-367834038,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23jh6YmKqmK-e_ifCyVBJA-_M9rugks5tXeDxgaJpZM4SP2Z2
.

--
Yuvi Panda T
http://yuvi.in/blog

[willingc]

@yuvipanda Do we have heapster deployed on the cluster or does stackdriver do it for us?

[yuvipanda]

@willingc we have heapster deployed (GKE does it for us automatically in the kube-system namespace), but we also collect metrics into prometheus (prometheus.mybinder.org). heapster is not persistent, while prometheus is.

[minrk]

Closing a stale issue - the placeholder pods mitigate this pretty well, as long as we have enough of them.