Zero-to-jupyterhub-k8s: HTTP probe failed with 0.9.0 version

[mbalduini]

Bug description

The installation of the jupyterhub version 0.9.0 successfully ended, but the hub pod results running but not ready.
As you can see from the pod description below, the problem is Readiness probe failed: HTTP probe failed with statuscode: 404.

Name:           hub-75c958fc98-wvvnj
Namespace:      jhubt
Priority:       0
Node:           ip-172-20-119-190.eu-west-1.compute.internal/172.20.119.190
Start Time:     Thu, 02 Jul 2020 12:32:00 +0000
Labels:         app=jupyterhub
                component=hub
                hub.jupyter.org/network-access-proxy-api=true
                hub.jupyter.org/network-access-proxy-http=true
                hub.jupyter.org/network-access-singleuser=true
                pod-template-hash=75c958fc98
                release=jhubt
Annotations:    checksum/config-map: e56bc18260532e54c31f0783aaafc3c1a18c5f26f4d9ef0d03aa358062b12902
                checksum/secret: 24278f774ebcdd491d91205c013884f04ce5af859fa4dc54132e148123b5acfe
Status:         Running
IP:             100.120.0.9
IPs:            <none>
Controlled By:  ReplicaSet/hub-75c958fc98
Containers:
  hub:
    Container ID:  docker://61ebab69eae15e44f236d414e6d7e2d9f650b00db5fd4bc005c5c60c851f4c10
    Image:         jupyterhub/k8s-hub:0.9.0
    Image ID:      docker-pullable://jupyterhub/k8s-hub@sha256:ec78bdae0fed0c49474950a1fe4fec3b00448599d4a35e51a93b10a7f342846b
    Port:          8081/TCP
    Host Port:     0/TCP
    Command:
      jupyterhub
      --config
      /etc/jupyterhub/jupyterhub_config.py
      --upgrade-db
    State:          Running
      Started:      Thu, 02 Jul 2020 12:32:23 +0000
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      200m
      memory:   512Mi
    Readiness:  http-get http://:hub/hub/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      PYTHONUNBUFFERED:        1
      HELM_RELEASE_NAME:       jhubt
      POD_NAMESPACE:           jhubt (v1:metadata.namespace)
      CONFIGPROXY_AUTH_TOKEN:  <set to the key 'proxy.token' in secret 'hub-secret'>  Optional: false
    Mounts:
      /etc/jupyterhub/config/ from config (rw)
      /etc/jupyterhub/cull_idle_servers.py from config (rw,path="cull_idle_servers.py")
      /etc/jupyterhub/jupyterhub_config.py from config (rw,path="jupyterhub_config.py")
      /etc/jupyterhub/secret/ from secret (rw)
      /etc/jupyterhub/z2jh.py from config (rw,path="z2jh.py")
      /srv/jupyterhub from hub-db-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from hub-token-kt88q (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      hub-config
    Optional:  false
  secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  hub-secret
    Optional:    false
  hub-db-dir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  hub-db-dir
    ReadOnly:   false
  hub-token-kt88q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  hub-token-kt88q
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                     From                                                   Message
  ----     ------                  ----                    ----                                                   -------
  Warning  FailedScheduling        9m6s (x4 over 9m10s)    default-scheduler                                      pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled               9m1s                    default-scheduler                                      Successfully assigned jhubt/hub-75c958fc98-wvvnj to ip-172-20-119-190.eu-west-1.compute.internal
  Normal   SuccessfulAttachVolume  8m59s                   attachdetach-controller                                AttachVolume.Attach succeeded for volume "pvc-e9576486-b52b-4dac-9a21-754233f56f25"
  Normal   Pulling                 8m55s                   kubelet, ip-172-20-119-190.eu-west-1.compute.internal  Pulling image "jupyterhub/k8s-hub:0.9.0"
  Normal   Pulled                  8m39s                   kubelet, ip-172-20-119-190.eu-west-1.compute.internal  Successfully pulled image "jupyterhub/k8s-hub:0.9.0"
  Normal   Created                 8m38s                   kubelet, ip-172-20-119-190.eu-west-1.compute.internal  Created container hub
  Normal   Started                 8m38s                   kubelet, ip-172-20-119-190.eu-west-1.compute.internal  Started container hub
  Warning  Unhealthy               3m45s (x30 over 8m35s)  kubelet, ip-172-20-119-190.eu-west-1.compute.internal  Readiness probe failed: HTTP probe failed with statuscode: 404

Expected behaviour

I expect to be able to access the hub using the the public IP (or aws ELB) listed as a service

Actual behaviour

The hub pod is running but keep remaining not accessible (not ready)

Your personal set up

I'm working on AWS and using kops to control the cluster

• OS:
  Ubuntu 18.04
• Version:

k8s:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.12", GitCommit:"e2a822d9f3c2fdb5c9bfbe64313cf9f657f0a725", GitTreeState:"clean", BuildDate:"2020-05-06T05:09:48Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

Helm

Client: &version.Version{SemVer:"v2.16.9", GitCommit:"8ad7037828e5a0fca1009dabe290130da6368e39", GitTreeState:"clean"}

Server: &version.Version{SemVer:"v2.16.9", GitCommit:"8ad7037828e5a0fca1009dabe290130da6368e39", GitTreeState:"clean"}

helm chart 0.9.0

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:

Welcome to the Jupyter community! :tada:

[consideRatio]

# restart hub to clean logs
kubectl rollout restart deploy/hub

# wait 30 seconds or so, then check logs
kubectl logs deploy/hub

• Does the hub pod seem to think everything is okay, or does the logs show errors?
• What is your Helm chart configuration?
• Does this reproduce for your without any custom configuration except proxy.secretToken which is required?

I guess this relates to your k8s / cloud setup, where perhaps there is something between kubelet which sends the readiness probe checks and the pod that blocks traffic for some reason if the hub pod actually startup without errors etc.

[mbalduini]

Hi @consideRatio, thanks for the reply.

As you suggested, I tried to set up the hub with a very basic configuration an it worked fine.
I then started to add pieces of custom configuration and i found that the problem is related to a very specific extra-config of the hub: c.JupyterHub.base_url = '/test'

Here my working configuration:

proxy:
  secretToken: "<token>"
  https:
    enabled: false

auth:
  type: dummy
  dummy:
    password: '<password>'
  admin:
    users:
      - admin

hub:
  extraConfig: 
    timeout_config: |
      c.Spawner.http_timeout = 600
      c.Spawner.start_timeout = 600
    dockerspawner_config: |
      c.DockerSpawner.extra_host_config = {"cap_add": "SYS_ADMIN", "device": "/dev/fuse"}

singleuser:
  defaultUrl: /lab
  extraEnv:
    GRANT_SUDO: "yes"
    NOTEBOOK_ARGS: "--allow-root"
  uid: 0
  cmd: start-singleuser.sh
  image:
    name: <image>
    tag: <tag>
  imagePullSecret:
    enabled: true
    username: <username>
    password: <pwd>
  lifecycleHooks:
    postStart:
      exec:
        command:
          - "sh"
          - "-c"
          - >
            <SOME-COMMAND>

  cpu:
    limit: 2
    guarantee: 1
  memory:
    limit: 8G
    guarantee: 4G

scheduling:
  userScheduler:
    enabled: true
  podPriority:
    enabled: true
  userPlaceholder:
    enabled: true
    replicas: 2

cull:
  enabled: true
  timeout: 7200
  every: 600

I use custom configuration in order to run user's container as root, but when I add the extra-config c.JupyterHub.base_url = '/test' the hub pod keep looking in running status but not ready.

Looking in the hub logs I found a couple of exceptions, that look as not related to the base_url config.

[E 2020-08-07 08:57:43.768 JupyterHub web:1220] Uncaught exception in write_error
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1218, in send_error
        self.write_error(status_code, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1195, in write_error
        html = self.render_template('%s.html' % status_code, **ns)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1120, in render_template
        return template.render(**template_ns)
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 1090, in render
        self.environment.handle_exception()
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 832, in handle_exception
        reraise(*rewrite_traceback_stack(source=source))
      File "/usr/local/lib/python3.6/dist-packages/jinja2/_compat.py", line 28, in reraise
        raise value.with_traceback(tb)
      File "/usr/local/share/jupyterhub/templates/404.html", line 1, in top-level template code
        {% extends "error.html" %}
      File "/usr/local/share/jupyterhub/templates/error.html", line 1, in top-level template code
        {% extends "page.html" %}
      File "/usr/local/share/jupyterhub/templates/page.html", line 75, in top-level template code
        {% if not no_spawner_check and user and user.spawner.options_form %}
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 471, in getattr
        return getattr(obj, attribute)
    jinja2.exceptions.UndefinedError: 'jupyterhub.orm.Service object' has no attribute 'spawner'

[W 2020-08-07 08:57:43.770 JupyterHub log:174] 404 GET /test/hub/hub/api/users ([email protected]) 37.86ms
[E 200807 08:57:43 ioloop:763] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7f631dfe8400>>, <Future finished exception=HTTP 404: Not Found>)
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/ioloop.py", line 743, in _run_callback
        ret = callback()
      File "/usr/local/lib/python3.6/dist-packages/tornado/ioloop.py", line 767, in _discard_future_result
        future.result()
      File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 742, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "/etc/jupyterhub/cull_idle_servers.py", line 119, in cull_idle
        resp = yield fetch(req)
      File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 735, in run
        value = future.result()
      File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 748, in run
        yielded = self.gen.send(value)
      File "/etc/jupyterhub/cull_idle_servers.py", line 113, in fetch
        return (yield client.fetch(req))
      File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 735, in run
        value = future.result()
    tornado.httpclient.HTTPClientError: HTTP 404: Not Found

Any hints?
I'm still digging into logs and configuration files, I'll provide more details if I can.

[manics]

The baseurl affects multiple Kubernetes components so you can't just set it in the JupyterHub config, instead you should set hub.baseUrl in the Helm chart:
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/0.9.1/jupyterhub/values.yaml#L10

[mbalduini]

Thank you very much @manics. I completely missed hub.baseUrl configuration.