Zero-to-jupyterhub-k8s: TypeError: '<' not supported between instances of 'datetime.datetime' and 'NoneType'

[ToniChaz]

hi guys, I am getting this error:

[E 2019-09-27 12:45:45.573 JupyterHub base:1011] Preventing implicit spawn for toni because last spawn failed: '<' not supported between instances of 'datetime.datetime' and 'NoneType'
[E 2019-09-27 12:45:45.573 JupyterHub web:1788] Uncaught exception GET /hub/user/toni/ (10.1.1.1)
    HTTPServerRequest(protocol='http', host='10.152.183.128', method='GET', uri='/hub/user/toni/', version='HTTP/1.1', remote_ip='10.1.1.1')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1013, in get
        raise copy.copy(exc).with_traceback(exc.__traceback__)
      File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 589, in error_callback
        future.result()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 489, in spawn
        raise e
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 409, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1636, in _start
        events = self.events
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1491, in events
        for event in self.event_reflector.events:
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 72, in events
        key=lambda x: x.last_timestamp,
    TypeError: '<' not supported between instances of 'datetime.datetime' and 'NoneType'

[E 2019-09-27 12:45:45.578 JupyterHub log:150] {
      "X-Forwarded-Host": "10.152.183.128",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "80",
      "X-Forwarded-For": "10.1.1.1",
      "Cookie": "jupyterhub-hub-login=\"2|1:0|10:1569588345|20:jupyterhub-hub-login|44:ZDIwZjJjNTlmMzA0NDNiMjhmMmRhZTU1MjRjZDYzYmM=|4a1a8976f6cf225b6ad51e5ff0296f44ea5ef98e3d71541bd99298ad3de65c09\"; jupyterhub-session-id=7b3dd241fc6f478294460bda5cf26124",
      "Accept-Language": "es,en;q=0.9",
      "Accept-Encoding": "gzip, deflate",
      "Referer": "http://10.152.183.128/hub/login",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Cache-Control": "max-age=0",
      "Connection": "close",
      "Host": "10.152.183.128"
    }
[E 2019-09-27 12:45:45.578 JupyterHub log:158] 500 GET /hub/user/toni/ ([email protected]) 30.10ms

I tried to clean jhub namespace and install form zero with helm but i am getting the same error. It stopped to work suddenly, and i can´t create or login any user.

[betatim]

This looks like a similar issue that was posted earlier related to the kubernetes version of the cluster not yet being supported by the Python client library.

@manics I think you replied to that issue and I can't find it :-/

[ToniChaz]

But at the beginning it worked, suddenly stopped working!

Some additional info:

microk8s.kubectl version:
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

helm version:
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

helm list:
NAME    REVISION    UPDATED                     STATUS      CHART               APP VERSION NAMESPACE
elk     4           Fri Sep 27 12:27:46 2019    DEPLOYED    elastic-stack-1.8.0 6           elk      
jhub    2           Fri Sep 27 14:36:39 2019    DEPLOYED    jupyterhub-0.8.2    0.9.6       jhub 

[manics]

It wasn't me, but this is the related issue: https://github.com/jupyterhub/kubespawner/issues/354

[ToniChaz]

Ok, I go to close the issue, you are right @manics.

Thanks