Mybinder.org-deploy: Fix user pods dying when hub is shut down

[yuvipanda]

@betatim noticed that deleting the hub pod actually kills all the user pods. I am pretty sure this wasn't old behavior, but indeed, we never set c.JupyterHub.cleanup_servers = False in z2jh! I made a PR for it, but:

1. Why now? Am pretty sure this wasn't old behavior...
2. How do we safely deploy this? Deployment would restart the hub, triggering pod deaths. I suspect we'd have to ssh into the node and do a 'kill -9' to be absolutely sure the hub doesn't try to shut everything down before going away.

A mystery, and a ticking time bomb!

[betatim]

This is the related k8s PR https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/302

I am +1 for fixing this ASAP. Should we do a practice on staging to make sure we can kill -9 and redeploy without too much delay?

Is there a way to make bhub or nginx briefly serve a "We are doing something to mybinder.org, back in a few seconds" instead of letting users run into an error? Or rare enough that this is not worth adding?

[minrk]

Testing it out on staging sounds like a good idea. I’m out this morning, but will be able to look in this afternoon if you haven’t done it already.

[minrk]

@betatim a maintenance page (like the default 404 backend) is a good idea. Updating the ingress to point to it ought to work.

[minrk]

Relevant facts needed for this to work:

1. we can talk to the jupyterhub sqlite database directly
2. the shutdown_servers config can be overridden by an explicit shutdown request via the API.
3. configmap can be edited and will be loaded on next pod launch

From that information, we should be able to do the following:

1. edit configmap to add c.JupyterHub.cleanup_servers = False to hub.extra-config.py
2. kubectl exec -it jupyterhub and manually create an API token with admin access in the database.
3. shutdown the Hub via the API with servers=False
4. when the pod next starts, it should load the updated configmap.

I just checked and delete pod --grace-period=0 and --now and --force all send SIGTERM, so they all trigger cleanup, even if SIGKILL will come soon after and interrupt the cleanup process, so they aren't an option if we want to be sure things don't get culled.

[minrk]

This script, run in /srv/jupyterhub on the hub pod will create and print a new token with admin privileges:

from jupyterhub import orm

url = 'sqlite:///jupyterhub.sqlite'

db = orm.new_session_factory(url)()
service = orm.Service(name='recovery', admin=True)
db.add(service)
db.commit()

token = service.new_api_token()
print(token)

[minrk]

And this will shutdown the hub, leaving any running servers alive:

import json
import requests

token = 'retrieved from above'
hub_url = 'https://hub.staging.mybinder.org'

r = requests.post(
    hub_url + '/hub/api/shutdown',
    headers={
        'Authorization': f'token {token}',
    },
    data=json.dumps({
        'servers': False,
    }))
r.raise_for_status()
print(r.text)

[betatim]

This worked well. We should now be able to restart jhub without user pods being disturbed.