Zero-to-jupyterhub-k8s: Kubernetes 1.16 compatibility

[clkao]

It appears there are several issues with running z2jh on kubernetes 1.16, running the latest master with kind:

• helm init failures
• scheduling.userScheduler.enabled: true causing user pod to stuck in Pending state, without any event
• singleuser.events: true causing launch failures, as events seem not to be what EventReflector is expecting: (https://github.com/jupyterhub/kubespawner/issues/354)

[E 2019-10-10 14:30:35.607 JupyterHub gen:593] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py:800> exception=TypeError("'<' not supported between instances of 'datetime.datetime' and 'NoneType'",)> after timeout

I haven't looked further but each of them should probably be of its own issue

[betatim]

I think https://github.com/jupyterhub/kubespawner/pull/355 is another k8s 1.16 related issue (and fix).

[clkao]

@betatim right, i think that's exactly the 3rd one i listed. let me link that as well.

[betatim]

Now merged https://github.com/jupyterhub/kubespawner/pull/356. Maybe it is a temporary fix but it'll get people unblocked and we can make a better fix later. WDYT?

[stv0g]

@betatim Unfortunately, https://github.com/jupyterhub/kubespawner/pull/356/ did not solve the issue for me:

[E 2019-10-14 14:15:11.268 JupyterHub web:1788] Uncaught exception GET /hub/api/users/304957/server/progress (192.168.2.183)
    HTTPServerRequest(protocol='http', host='jupyter.rwth-aachen.de', method='GET', uri='/hub/api/users/304957/server/progress', version='HTTP/1.1', remote_ip='192.168.2.183')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/apihandlers/users.py", line 592, in get
        async for event in events:
      File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 366, in step
        return await ANextIter(self._it, start_fn, *args)
      File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 199, in __next__
        return self._invoke(self._it.__next__)
      File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 209, in _invoke
        result = fn(*args)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/utils.py", line 555, in iterate_until
        await yield_(item_future.result())
      File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 366, in step
        return await ANextIter(self._it, start_fn, *args)
      File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 197, in __next__
        return self._invoke(first_fn, *first_args)
      File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 209, in _invoke
        result = fn(*args)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/spawner.py", line 976, in _generate_progress
        async for event in progress:
      File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 366, in step
        return await ANextIter(self._it, start_fn, *args)
      File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 197, in __next__
        return self._invoke(first_fn, *first_args)
      File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 209, in _invoke
        result = fn(*args)
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1541, in progress
        events = self.events
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1512, in events
        for event in self.event_reflector.events:
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 73, in events
        key=lambda x: x.last_timestamp if x.last_timestamp is not None else 0.,
    TypeError: '<' not supported between instances of 'datetime.datetime' and 'float'

[ktaletsk]

@betatim I have the same issue as @stv0g:

'<' not supported between instances of 'float' and 'datetime.datetime'

[consideRatio]

In #1422 I'm now testing against 1.16 but allowing allow for failures.

https://travis-ci.org/jupyterhub/zero-to-jupyterhub-k8s/jobs/599062620

[betatim]

https://github.com/jupyterhub/kubespawner/pull/357 is up for people who are having trouble running with kubernetes 1.16 to take over and finish off. I won't get around to pushing it over the finish line.

[consideRatio]

Help pushing https://github.com/jupyterhub/kubespawner/pull/357 through the finish line would be appreciated. Note that the current development setup described in CONTRIBUTING.md may make it a plausible to locally develop a functional solution of z2jh+kubespawner+k8s 1.16. I suggest using the dev script to create a kind cluster with k8s 1.16 and run tests etc.

To get the test to use a locally developed kubespawner, I suggest to git clone kubespawner into the images/hub folder, and in the images/hub/Dockerfile add some steps where we copy the local kubespawner repo along with local changes and install it with pip install -e kubespawner or similar.

[manics]

Quick and dirty way:
Replace jupyterhub-kubespawner==0.10.1 with https://github.com/jupyterhub/kubespawner/archive/884f2f6f68e13401a39265d4617ce465a716a4a3.zip:
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/be710d7f0107f64219bbd2eb52c22e2f89840715/images/hub/requirements.txt#L7

And test as normal (or push to GitHub as a temporary branch and let travis test it for you)

[consideRatio]

@clkao, all these issues are now being addressed, mostly in #1483:

• Helm init part is outside the scope of this helm chart, but can be resolved by using a modern version of helm.
• The pending user pods is solved by bumping to a more recent kube-scheduler binary which is used by user-scheduler.
• The kubespawner issues is being solved by a PR in kubespawner and a bump in Z2JH.