Zero-to-jupyterhub-k8s: "Error from server [NotFound]: pods "Jupyter-User" not found"

[apachipa]

Hi,

It seems there is a minor bug with Jupyterhub using Kubernetes and Docker.
I tested this in AWS, following instructions on Zero to Jupyterhub.

Below are the symptoms:

User Login to Jupyterhub portal.
User Clicks on "Start My Server"
This will spin up a kubernetes pod with Jupyter Notebook server in a container for that user.
User goes to control Panel and clicks on " My Server" instead of "Stop my Server".
User cant connect to his Jupyter Notebook Server from here on(results in blank gray screen "Try clearing your cookies. ERR_TOO_MANY_REDIRECTS").
If User try to logout and log back in and click "Start My Server". Same error occurs "Try clearing your cookies. ERR_TOO_MANY_REDIRECTS".
I can see Pod running for that user forever running command line "kubectl get pods --all-namepaces", and will not terminate
I tried to forcefully delete pod and i get error "Error from server [NotFound]: pods "Jupyter-User" not found ".

Any help on this issue will be much appreciated.

Thanks In advance.

Cheers,
Ash.

[choldgraf]

For the 503 TOO MANY REDIRECTS error, did you tell them to clear their cookies, or to try from an incognito window?

I believe this bug has been fixed in the latest version of JupyterHub, which is in v0.5 of the helm chart, @yuvipanda do you know if that's released now?

[yuvipanda]

It depends on why this bug exists. Can you provide us the logs from the running user's pod? You can do so with 'kubectl --namespace= logs '

[apachipa]

Hi Yuvi,

Below are the details:

$ kubectl --namespace= logs
error: expected 'logs (POD | TYPE/NAME) [CONTAINER_NAME]'.
POD or TYPE/NAME is a required argument for the logs command
See 'kubectl logs -h' for help and examples.

$ kubectl logs --namespace='kube-system'
error: expected 'logs (POD | TYPE/NAME) [CONTAINER_NAME]'.
POD or TYPE/NAME is a required argument for the logs command
See 'kubectl logs -h' for help and examples.

$ kubectl logs --namespace="kube-system"
error: expected 'logs (POD | TYPE/NAME) [CONTAINER_NAME]'.
POD or TYPE/NAME is a required argument for the logs command
See 'kubectl logs -h' for help and examples.

$ kubectl logs --namespace=kube-system
error: expected 'logs (POD | TYPE/NAME) [CONTAINER_NAME]'.
POD or TYPE/NAME is a required argument for the logs command
See 'kubectl logs -h' for help and examples.

So i ran below command line:

$ kubectl get pods --all-namespaces

NAMESPACE     NAME                                                                     READY     STATUS    RESTARTS   AGE
kube-system   calico-etcd-nc6k5                                                        1/1       Running   0          12d
kube-system   calico-node-9s0f3                                                        2/2       Running   0          12d
kube-system   calico-node-hhv6w                                                        2/2       Running   1          12d
kube-system   calico-node-m5slg                                                        2/2       Running   1          12d
kube-system   calico-policy-controller-1727037546-nhr41                                1/1       Running   0          12d
kube-system   etcd-ip-192-168-11-169.ap-southeast-2.compute.internal                      1/1       Running   0          12d
kube-system   hub-deployment-606657460-1l4wf                                           1/1       Running   0          4d
kube-system   jupyter-admin                                                            1/1       Running   0          4d
kube-system   jupyter-user1                                                            1/1       Running   0          4d
kube-system   kube-apiserver-ip-192-168-11-169.ap-southeast-2.compute.internal            1/1       Running   0          12d
kube-system   kube-controller-manager-ip-192-168-11-169.ap-southeast-2.compute.internal   1/1       Running   0          12d
kube-system   kube-dns-2425271678-0xr64                                                3/3       Running   0          12d
kube-system   kube-proxy-1rshl                                                         1/1       Running   0          12d
kube-system   kube-proxy-3ksll                                                         1/1       Running   0          12d
kube-system   kube-proxy-d4bh4                                                         1/1       Running   0          12d
kube-system   kube-scheduler-ip-192-168-11-169.ap-southeast-2.compute.internal            1/1       Running   0          12d
kube-system   kubernetes-dashboard-3313488171-jjtjx                                    1/1       Running   0          12d
kube-system   proxy-deployment-1227971824-k8vbd                                        1/1       Running   0          4d
kube-system   tiller-deploy-1853538654-kc6wl                                           1/1       Running   0          12d

My config.yaml looks as below:

hub:
  output of first execution of 'openssl rand -hex 32'
  cookieSecret: "cookiescret hex number generated"

proxy:
  output of second execution of 'openssl rand -hex 32'
  secretToken: "secure token hex number generate"

singleuser:
  image:
    name: jupyter/all-spark-notebook
    tag: latest

auth:
  dummy:
    password: testpass123

Helm command i am using is as below:

helm install jupyterhub/jupyterhub \
    --version=v0.4 \
    --name=class1 \
    --namespace=kube-system \
    -f config.yaml

Do i need to change version to --version=0.5 to fix this issue ?

Cheers,
Ash

[choldgraf]

looks like the 0.5 helm chart hasn't been released yet (which is the one that has jupyterhub 0.8, which fixes the bug that I think you're reporting here). Does hitting the page w/ an incognito window work? Or after clearing cookies?

[apachipa]

Clearing cookies and opening in incognito window did not help.
Same result.

[consideRatio]

The config.yaml should look more like...

proxy:
  secretToken: 192de044096f10990a3256355af99f90936eb37189134c277568753ce0cd8d9d

the log command should be run like...

kubectl logs <NAME-OF-POD-HERE> --namespace <YOUR-NAMESPACE-HERE>