Zero-to-jupyterhub-k8s: Image-puller daemonset fails with restricted PSP

[jatinder91]

Bug description

When restricted PSP is in place, the image-puller daemonset fails as below:-

NAME                       READY   STATUS                            RESTARTS   AGE
hook-image-awaiter-wdgld   1/1     Running                           0          21s
hook-image-puller-dcsbg    0/1     Init:CreateContainerConfigError   0          21s
hook-image-puller-gbw9c    0/1     Init:CreateContainerConfigError   0          21s
hook-image-puller-gztbr    0/1     Init:CreateContainerConfigError   0          21s
hook-image-puller-mvhpr    0/1     Init:CreateContainerConfigError   0          21s
hook-image-puller-qb4rp    0/1     Init:CreateContainerConfigError   0          21s
hook-image-puller-r2txj    0/1     Init:CreateContainerConfigError   0          21s

Error:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  32s                default-scheduler  Successfully assigned default/hook-image-puller-dcsbg to example-slv-02
  Normal   Pulling    31s                kubelet            Pulling image "jupyterhub/k8s-network-tools:0.10.1"
  Normal   Pulled     25s                kubelet            Successfully pulled image "jupyterhub/k8s-network-tools:0.10.1"
  Warning  Failed     12s (x3 over 25s)  kubelet            Error: container has runAsNonRoot and image will run as root
  Normal   Pulled     12s (x2 over 24s)  kubelet            Container image "jupyterhub/k8s-network-tools:0.10.1" already present on machine

Expected behaviour

Helm chart should be able to deployed in restricted PSP enabled cluster.

Actual behaviour

Helm chart is not deploying in restricted PSP enabled cluster.

How to reproduce

Restricted.yaml

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'runtime/default'
    seccomp.security.alpha.kubernetes.io/defaultProfileName:  'runtime/default'
  name: default
spec:
  allowedCapabilities: []  # default set of capabilities are implicitly allowed
  allowPrivilegeEscalation: false
  fsGroup:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  hostIPC: false
  hostNetwork: false
  hostPID: false
  privileged: false
  readOnlyRootFilesystem: false
  runAsUser:
    rule: 'MustRunAsNonRoot'
  seLinux:
    rule: 'RunAsAny'
  supplementalGroups:
    rule: 'MustRunAs'
    ranges:
      # Forbid adding the root group.
      - min: 1
        max: 65535
  requiredDropCapabilities:
  - ALL
  volumes:
  - 'configMap'
  - 'downwardAPI'
  - 'emptyDir'
  - 'persistentVolumeClaim'
  - 'projected'
  - 'secret'

---
# Cluster role which grants access to the default pod security policy
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: default-psp
rules:
- apiGroups:
  - policy
  resourceNames:
  - default
  resources:
  - podsecuritypolicies
  verbs:
  - use

---
# Cluster role binding for default pod security policy granting all authenticated users access
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-psp
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: default-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:authenticated

1. kubectl create -f Restricted.yaml
2. helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
3. helm repo update
4. helm install jupyterhub jupyterhub/jupyterhub --version=0.10.1

FIX:

1. Add variable in values.yml

prePuller:
  securityContext:
    runAsUser: 65534  # nobody user
    runAsGroup: 65534 # nobody group

1. change in _daemonset-helper.yaml on line number 41.

   spec:
      securityContext:
        {{- .Values.prePuller.securityContext  | toYaml | trimSuffix "\n" | nindent 8 }}

output:

NAME                            READY   STATUS    RESTARTS   AGE
continuous-image-puller-gcw8x   1/1     Running   0          2m22s
continuous-image-puller-gp2q5   1/1     Running   0          2m22s
continuous-image-puller-ktnbx   1/1     Running   0          2m22s
continuous-image-puller-lfprz   1/1     Running   0          2m22s
continuous-image-puller-nl7c8   1/1     Running   0          2m22s
continuous-image-puller-wwrf2   1/1     Running   0          2m22s
hub-6669f8989-snnfn             1/1     Running   0          2m22s
proxy-5b98f59b4c-tf7gh          1/1     Running   0          2m22s
user-scheduler-68dd9d78-p2mq7   1/1     Running   0          2m22s
user-scheduler-68dd9d78-pvbxg   1/1     Running   0          2m22s

personal set up

• OS:
  CentOS Linux release 7.7.1908 (Core)
• Version:
  0.10.1
• Configuration:
  ```
  kubectl version
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:56:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9", GitCommit:"94f372e501c973a7fa9eb40ec9ebd2fe7ca69848", GitTreeState:"clean", BuildDate:"2020-09-16T13:47:43Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:

Welcome to the Jupyter community! :tada:

[consideRatio]

@jatinder91 such a beautiful amazing issue! Thank you! I'll fix this right away. I consider the crux to be that we have failed to add containerSecurityContext on all the containers we add in the prePuller, while we add it for some.

[consideRatio]

@jatinder91 0.10.2 published in ~5 minutes (https://travis-ci.com/github/jupyterhub/zero-to-jupyterhub-k8s/jobs/421566688).

[jatinder91]

@jatinder91 0.10.2 published in ~5 minutes (https://travis-ci.com/github/jupyterhub/zero-to-jupyterhub-k8s/jobs/421566688).

@consideRatio : Thank you for your excellent and quick turnaround on the ticket. Its working !!

[consideRatio]

@jatinder91 you made it easy! Thank you for the thorough report!

Ah it is working also! Nice!