Buildkit: rootless image didn't work on OKD (OpenShift) 3.11

[peishuli]

Tried to deploy a rootless buildkit image to OKD 3.11 using https://raw.githubusercontent.com/tektoncd/catalog/master/buildkit/0-buildkitd.yaml and resulted pod in "Crash Loop Back-off" status. The logs showed the following error: "[rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: invalid argument"

[AkihiroSuda]

sudo sh -c "echo 28633 > /proc/sys/user/max_user_namespaces" ?

[chendave]

similar issue found when run a pod with rootless image in Kubernetes.

# cat /proc/sys/user/max_user_namespaces
28633

# kubectl run --generator=run-pod/v1 \
> --image=moby/buildkit:master-rootless \
> bkinstance -- --oci-worker-no-process-sandbox
pod/bk created

# kubectl logs bkinstance 
[rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: permission denied

where can I find the docs on the how to use the rootless image?

[chendave]

@AkihiroSuda

[AkihiroSuda]

The doc is at https://github.com/moby/buildkit/blob/master/docs/rootless.md but needs readability improvement.

Maybe you need sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)".

Can you also try OpenShift 4 with RHEL8-based host?

[AkihiroSuda]

• Maybe you need to disable seccomp if it is automatically enabled with SCC
• Maybe you need to set seLinuxOptions to disable SELinux

[AkihiroSuda]

Seems SELinux needs to be disabled

$ crc version
version: 1.0.0-rc.0+34371d3
OpenShift version: 4.2.0-0.nightly-2019-09-26-192831 (embedded in binary)
$ kubectl --as=system:admin run --generator=run-pod/v1 --image=moby/buildkit:master-rootless bkinstance -- --oci-worker-no-process-sandbox

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "10.128.1.17"
          ],
          "default": true,
          "dns": {}
      }]
    openshift.io/scc: anyuid
  creationTimestamp: "2019-10-05T17:03:55Z"
  labels:
    run: bkinstance
  name: bkinstance
  namespace: foo
  resourceVersion: "429816"
  selfLink: /api/v1/namespaces/foo/pods/bkinstance
  uid: 1d46025b-e792-11e9-97a2-a23a971945de
spec:
  containers:
  - args:
    - --oci-worker-no-process-sandbox
    image: moby/buildkit:master-rootless
    imagePullPolicy: IfNotPresent
    name: bkinstance
    resources: {}
    securityContext:
      capabilities:
        drop:
        - MKNOD
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-vb7vk
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: default-dockercfg-n7tss
  nodeName: crc-chc9n-master-0
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    seLinuxOptions:
      level: s0:c23,c7
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-vb7vk
    secret:
      defaultMode: 420
      secretName: default-token-vb7vk
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-10-05T17:03:55Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-10-05T17:04:04Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-10-05T17:04:04Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-10-05T17:03:55Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: cri-o://8dc443f1b193df00f402289600ca1a68d9e7b68b810ad09291e1836b9305ef9b
    image: docker.io/moby/buildkit:master-rootless
    imageID: docker.io/moby/buildkit@sha256:d20c76a25eaa61a7e834177ae301d1f7ed77bc32d8ca05db71237265e1cc906d
    lastState: {}
    name: bkinstance
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2019-10-05T17:04:04Z"
  hostIP: 192.168.64.9
  phase: Running
  podIP: 10.128.1.17
  qosClass: BestEffort
  startTime: "2019-10-05T17:03:55Z"

$ buildctl --addr=kube-pod://bkinstance build --frontend dockerfile.v0 --local context=. --local dockerfile=.
[+] Building 6.0s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                              0.2s
 => => transferring dockerfile: 58B                                                                               0.1s
 => [internal] load .dockerignore                                                                                 0.1s
 => => transferring context: 2B                                                                                   0.1s
 => [internal] load metadata for docker.io/library/alpine:latest                                                  4.9s
 => [1/2] FROM docker.io/library/alpine@sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01fb   1.0s
 => => resolve docker.io/library/alpine@sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01fb   0.0s
 => => sha256:961769676411f082461f9ef46626dd7a2d1e2b2a38e6a44364bcbecf51e66dd4 1.51kB / 1.51kB                    0.0s
 => => sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01fb 1.64kB / 1.64kB                    0.0s
 => => sha256:acd3ca9941a85e8ed16515bfc5328e4e2f8c128caa72959a58a127b7801ee01f 528B / 528B                        0.0s
 => => sha256:9d48c3bd43c520dc2784e868a780e976b207cbf493eaff8c6596eb871cbd9609 2.79MB / 2.79MB                    0.5s
 => => unpacking docker.io/library/alpine@sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01f  0.2s
 => ERROR [2/2] RUN echo                                                                                          0.0s
------
 > [2/2] RUN echo:
------
error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to prepare v1k1qloq4dpenml2ogrum48b3: copying of parent failed: failed to copy xattrs: failed to set xattr "security.selinux" on /home/user/.local/share/buildkit/runc-native/snapshots/snapshots/new-431277440/bin/arch: operation not supported

[chendave]

will try it later, but seems ubuntu doesn't have SELinux by default.

[AkihiroSuda]

On Ubuntu you need to set apparmor profile to unconfined

https://kubernetes.io/docs/tutorials/clusters/apparmor/#securing-a-pod

[chendave]

works for me!

#cat rooless.yaml
apiVersion: v1
kind: Pod
metadata:
  name: bk
  annotations:
    container.apparmor.security.beta.kubernetes.io/bkinstance: unconfined
spec:
  containers:
  - args:
    - --oci-worker-no-process-sandbox
    image: moby/buildkit:master-rootless
    name: bkinstance

# kubectl create -f rooless.yaml

# buildctl --addr=kube-pod://bk build --frontend dockerfile.v0 --local dockerfile=. --local context=.  --exporter image --exporter-opt name=jungler/rootless:latest --frontend-opt platform=linux/amd64 --exporter-opt push=true
WARN[0000] --exporter <exporter> is deprecated. Please use --output type=<exporter>[,<opt>=<optval>] instead.
WARN[0000] --frontend-opt <opt>=<optval> is deprecated. Please use --opt <opt>=<optval> instead.
[+] Building 9.3s (7/7) FINISHED
 => [internal] load .dockerignore                                                                            0.0s
 => => transferring context: 2B                                                                              0.0s
 => [internal] load build definition from Dockerfile                                                         0.0s
 => => transferring dockerfile: 31B                                                                          0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                             2.8s
 => [1/3] FROM docker.io/library/alpine@sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f  0.0s
 => => resolve docker.io/library/alpine@sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f  0.0s
...

But seems rootless images only support x86 platform?

buildctl --addr=kube-pod://bk debug workers
ID                              PLATFORMS
l4nfzcipal4hbqnwxpfkjvd0s       linux/amd64,linux/386

[peishuli]

Thank you @AkihiroSuda for your help! The nodes under our OKD cluster are CentOS 7 VMs. I ran the following commands on compute nodes:
$ sudo sh -c "echo 28633 > /proc/sys/user/max_user_namespaces
$ sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
per your suggestion and the document below:
https://github.com/moby/buildkit/blob/master/docs/rootless.md#disabling-process-sandbox

It works now!