Zero-to-jupyterhub-k8s: postStart lifecyclehooks doesn't work properly with the latest notebooks images.

[olifly]

I've been using the postStart lifecyclehook to populate/sync users filesystem with notebooks for a few months and it's been working as documented. (https://zero-to-jupyterhub.readthedocs.io/en/latest/customizing/user-environment.html#about-user-storage-and-adding-files-to-it)

However I recently updated our singleuser notebook image since the old one had a security vulnerability and it seems that the latest notebook images don't play nice with executing scripts with the postStart lifecyclehooks.

Example postStart script that used to work with image: jupyter/r-notebook:605529603889

lifecycleHooks:
        postStart:
          exec:
            command:
              - "sh"
              - "-c"
              - >
                echo 'postStart hook succeeded'

Moving up to a newer image like jupyter/r-notebook:dc9744740e12 or jupyterhub/singleuser:latest results in the singleuser failing to spawn with the following error:

[Warning] Exec lifecycle hook ([sh -c echo 'postStart hook succeeded' ]) for Container "notebook" in Pod "jupyter-olafurh-40wuxinextcode-2ecom_jhub(74030c93-7802-11ea-bc4b-0a37a4c14e67)" failed - error: command 'sh -c echo 'postStart hook succeeded' ' exited with 126: , message: "OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused \"process_linux.go:86: executing setns process caused \\\"exit status 21\\\"\": unknown\r\n"

• K8s environment: AWS EKS (v1.14.9-eks-502bfb)
• Helm chart version: 0.8.2

[Castronova]

I also ran into this problem, and after some debugging I noticed that the pod fails even when the lifecycleHooks are removed. I found that reverting my base image back to jupyterhub/singleuser:1.1.dev resolved the issue.

It appears that this is a bug in the newer jupyterhub images, e.g. 1.2+

[JamesSample]

After updating my Docker image, I was experiencing exactly the same Exec lifecycle hook issue with my JupyterHub. Like @Castronova, I also discovered the problem remained even after removing lifecycleHooks completely.

After a bit of digging, I found the problem was caused by my config.yaml using a deprecated syntax (because I originally followed a now outdated version of the z2jh tutorial). Not sure if it'll help you, but this fix worked for me:

https://github.com/jupyter/docker-stacks/issues/1041#issuecomment-598877334

[consideRatio]

@JamesSample ! Ah nice catch and thank you soo much for documenthing this!

Closing as resolved by https://github.com/jupyter/docker-stacks/issues/1041#issuecomment-598877334, which is also reflecting the current z2jh setup instructions.

[olifly]

@consideRatio - The original issue is still unresolved. I had no problem launching singleuser pods if I removed the postStartup hook.

[consideRatio]

@olifly okay =/ I'd like to confirm this is an issue with this helm chart that needs to be updated in order to open this issue though, and I'd need you to help to escalate this further within this specific repo by confirming it relates to this repo.

It seems like a possible regression of some form relating to the images in jupyter/docker-stacks as you have noticed a change between versions an image built there: jupyter/r-notebook.

You can isolate the debugging of this by simply starting this pod without the entire Helm chart, by doing kubectl apply -f filename.yaml with the yaml file below, and then doing kubectl edit deploy tmp to see how a change in lifecycleHook influence it, or how a change in its cmd (kubernetes name of ENTRYPOINT) influence its startup with the lifecycleHook next to it, or wait I think the first thing to debug is to use jupyter/base-notebook, if that also have a regression you can debug changes here specifically.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: tmp
  name: tmp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tmp
  strategy: {}
  template:
    metadata:
      labels:
        app: tmp
    spec:
      containers:
      - image: jupyter/r-notebook
        name: r-notebook

As a reminder for everyone considering this, here is the error from the lifecycleHook again. I'm not sure what can cause this.

starting container process caused \"process_linux.go:86: executing setns process caused \\\"exit status 21\\\"\": unknown\r\n"