Buildah: Issue running buildah docker image with kubernetes

[sourabhsharma487]

Description

Briefly describe the problem you are having in a few paragraphs.

When I spin up the docker container on the single/standalone server with below command, I am able to perform all my task mentioned in Dockerfile such as pulling, building and pushing the image onto registry:

FYI - I pulled the docker image "tomkukral/buildah" from https://hub.docker.com/r/tomkukral/buildah

It works fine with below
```docker run --rm --privileged -it tomkukral/buildah bash
Inside container:
buildah bud -t ${IMAGENAME}
buildah push --creds ${registryUSER}:${registryPASS} ${IMAGENAME}

On the other hand, when I run this with k8s, I have been facing a write operations issue with buildah such as below:

Error:
13:23:39 STEP 5: RUN mkdir /apps
13:23:39 error running container: error creating container for [/bin/sh -c mkdir /apps]: container_linux.go:344: starting container process caused "process_linux.go:275: applying cgroup configuration for process caused "mkdir /sys/fs/cgroup/cpuset/buildah-buildah005644356: read-only file system""
13:23:39 : exit status 1

For more detailed logs/info:
While running with k8s, getting below mount points in readonly mode:

sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,cpuset)

Whereas, with docker container:
Getting below mount points in read+write mode, which is helping to create docker image using buildah container:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,name=systemd)

**Steps to reproduce the issue:**
1. Running POD with buildah container with k8s
2.
3.


**Describe the results you received:**
While running with k8s, getting below mount points in readonly mode:
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime,seclabel)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,seclabel,cpuset)

**Describe the results you expected:**
With the k8s, I should be able to get the mount points in read+write mode as same as spinninig 

**Output of `rpm -q buildah` or `apt list buildah`:**

buildah-1.7-2.git873f001.fc29.x86_64

**Output of `buildah version`:**

Version: 1.7
Go Version: go1.11.5
Image Spec: 1.0.0
Runtime Spec: 1.0.0
CNI Spec: 0.4.0
libcni Version:
Git Commit:
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64

**Output of `podman version` if reporting a `podman build` issue:**

Version: 1.1.2
RemoteAPI Version: 1
Go Version: go1.11.5
Git Commit: a95a49d3038462d033f84ac314ec8a3064a99cff
Built: Tue Mar 5 18:10:31 2019
OS/Arch: linux/amd64

**Output of `cat /etc/*release`:**

Fedora release 29 (Twenty Nine)
NAME=Fedora
VERSION="29 (Container Image)"
ID=fedora
VERSION_ID=29
VERSION_CODENAME=""
PLATFORM_ID="platform:f29"
PRETTY_NAME="Fedora 29 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:29"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=29
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=29
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
Fedora release 29 (Twenty Nine)
Fedora release 29 (Twenty Nine)

**Output of `uname -a`:**

Linux 67b1eed5d8d6 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

**Output of `cat /etc/containers/storage.conf`:**

This file is is the configuration file for all tools

that use the containers/storage library.

See man 5 containers-storage.conf for more information

The "container storage" table contains all of the server options.

[storage]

Default Storage Driver

driver = "overlay"

Temporary storage location

runroot = "/var/run/containers/storage"

Primary Read/Write location of container storage

graphroot = "/var/lib/containers/storage"

[storage.options]

Storage options to be passed to underlying storage drivers

AdditionalImageStores is used to pass paths to additional Read/Only image stores

Must be comma separated list.

additionalimagestores = [
]

Size is used to set a maximum size of the container image. Only supported by

certain container storage drivers.

size = ""

Path to an helper program to use for mounting the file system instead of mounting it

directly.

mount_program = "/usr/bin/fuse-overlayfs"

OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version

override_kernel_check = "true"

mountopt specifies comma separated list of extra mount options

mountopt = "root"

Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of

a container, to UIDs/GIDs as they should appear outside of the container, and

the length of the range of UIDs/GIDs. Additional mapped sets can be listed

and will be heeded by libraries, but there are limits to the number of

mappings which the kernel will allow when you later attempt to run a

container.

#

remap-uids = 0:1668442479:65536

remap-gids = 0:1668442479:65536

Remap-User/Group is a name which can be used to look up one or more UID/GID

ranges in the /etc/subuid or /etc/subgid file. Mappings are set up starting

with an in-container ID of 0 and the a host-level ID taken from the lowest

range that matches the specified name, and using the length of that range.

Additional ranges are then assigned, using the ranges which specify the

lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,

until all of the entries have been used for maps.

#

remap-user = "storage"

remap-group = "storage"

[storage.options.thinpool]

Storage Options for thinpool

autoextend_percent determines the amount by which pool needs to be

grown. This is specified in terms of % of pool size. So a value of 20 means

that when threshold is hit, pool will be grown by 20% of existing

pool size.

autoextend_percent = "20"

autoextend_threshold determines the pool extension threshold in terms

of percentage of pool size. For example, if threshold is 60, that means when

pool is 60% full, threshold has been hit.

autoextend_threshold = "80"

basesize specifies the size to use when creating the base device, which

limits the size of images and containers.

basesize = "10G"

blocksize specifies a custom blocksize to use for the thin pool.

blocksize="64k"

directlvm_device specifies a custom block storage device to use for the

thin pool. Required if you setup devicemapper.

directlvm_device = ""

directlvm_device_force wipes device even if device already has a filesystem.

directlvm_device_force = "True"

fs specifies the filesystem type to use for the base device.

fs="xfs"

log_level sets the log level of devicemapper.

0: LogLevelSuppress 0 (Default)

2: LogLevelFatal

3: LogLevelErr

4: LogLevelWarn

5: LogLevelNotice

6: LogLevelInfo

7: LogLevelDebug

log_level = "7"

min_free_space specifies the min free space percent in a thin pool require for

new device creation to succeed. Valid values are from 0% - 99%.

Value 0% disables

min_free_space = "10%"

mkfsarg specifies extra mkfs arguments to be used when creating the base.

device.

mkfsarg = ""

use_deferred_removal marks devicemapper block device for deferred removal.

If the thinpool is in use when the driver attempts to remove it, the driver

tells the kernel to remove it as soon as possible. Note this does not free

up the disk space, use deferred deletion to fully remove the thinpool.

use_deferred_removal = "True"

use_deferred_deletion marks thinpool device for deferred deletion.

If the device is busy when the driver attempts to delete it, the driver

will attempt to delete device every 30 seconds until successful.

If the program using the driver exits, the driver will continue attempting

to cleanup the next time the driver is used. Deferred deletion permanently

deletes the device and all data stored in device will be lost.

use_deferred_deletion = "True"

xfs_nospace_max_retries specifies the maximum number of retries XFS should

attempt to complete IO when ENOSPC (no space) error is returned by

underlying storage device.

xfs_nospace_max_retries = "0"

If specified, use OSTree to deduplicate files with the overlay backend

ostree_repo = ""

Set to skip a PRIVATE bind mount on the storage home directory. Only supported by

certain container storage drivers

skip_mount_home = "false"

```

[rhatdan]

@sourabhsharma487 I really do not know what you are reporting here. What buildah commands were you trying to run in a container. We are working on documenting how to run buildah within a locked down container. But most people currently need to do it in a privileged container. With /var/lib/containers volume mounted in.

[sourabhsharma487]

@rhatdan : My bad, I had updated the issue with proper information however that Description block was commented. Please have a look at the description and let me know in case further information is required.

[rhatdan]

@sourabhsharma487 Inside a locked down container we are recommending that people use
buildah bud --isolation chroot ...

Which might get your further.
The Buildah container we are experimenting with has BUILDAH_ISOLATION=chroot environment variable set. We have also found SELinux issues with this, so make sure you have the latest container-selinux-2.94 installed, (If you are running with SELinux).

@TomSweeneyRedHat Could you paste in the Dockerfile we are using now for the experiments.

[TomSweeneyRedHat]

@sourabhsharma487 We've a Demo in progress that covers this: https://github.com/containers/Demos/pull/17

However @rhatdan found an issue with the way volumes were being recursively bind mounted, so I changed the Dockerfile from that script to build Buildah from GitHub and to pull in a PR that I'd put together. This is all a big WIP, no production machines should come near this. I'm still having issues getting stuff to run fully, but it's closer.

# cat /root/Dockerfile.plus
FROM fedora
ENV GOPATH=/root/buildah

#RUN yum -y install buildah --exclude container-selinux --enablerepo updates-testing; rm -rf /var/cache /var/log/dnf* /var/log/yum.*



RUN dnf -y install --enablerepo=updates-testing \
     make \
     golang \
     bats \
     btrfs-progs-devel \
     device-mapper-devel \
     glib2-devel \
     gpgme-devel \
     libassuan-devel \
     libseccomp-devel \
     ostree-devel \
     git \
     bzip2 \
     go-md2man \
     runc \
fuse-overlayfs \
fuse3 \
     containers-common; \
     mkdir /root/buildah; \
     git clone https://github.com/containers/buildah /root/buildah/src/github.com/containers/buildah; \
     cd /root/buildah/src/github.com/containers/buildah; \
     git checkout -b TomSweeneyRedHat-dev/tsweeney/cinc master;\
     git pull https://github.com/TomSweeneyRedHat/buildah.git dev/tsweeney/cinc;\
     make;\
     make install

RUN sed 's|^#mount_program|mount_program|g' -i /etc/containers/storage.conf
ENV _BUILDAH_STARTED_IN_USERNS="" BUILDAH_ISOLATION=chroot
COPY Dockerfile.plus /root

Then to run it:

podman build -t buildahimage -f /root/Dockerfile.plus .
podman run --net=host --device=/dev/fuse -v /var/lib/mycontainers:/var/lib/containers:Z -ti --uidmap 0:5000:1000 buildahimage buildah --debug bud -f /root/Dockerfile.plus  --isolation=chroot /root

You can remove the --debug from the second line to get a little less output. Currently it's having a problem bind mouting /dev

[sourabhsharma487]

@rhatdan , @TomSweeneyRedHat : After adding "--isolation chroot " in buildah bud command as below, I am getting a different error:

buildah bud --isolation chroot -t ${IMAGENAME} .

Error

20:56:48 STEP 5: RUN mkdir /apps /root/.ssh
20:56:48 process exited with error: fork/exec /bin/sh: no such file or directorysubprocess exited with status 1

Just for quick oversight on this, I have updated my k8s YAML with below securityContext but no luck:

securityContext:
      privileged: true
      runAsUser: 0
      fsGroup: 0
      capabilities:
        add: ["SYS_ADMIN"]

FYI - I don`t see any challenge with standalone docker, however, it is an issue while running with kubernetes (k8s) only.

Please, can you help me with this? Appreciate your quick help.

[rhatdan]

@sourabhsharma487 Could you try this with Podman and make sure we can get it to work there, Then we can attempt this in K8s.

[sourabhsharma487]

Hi @rhatdan : I managed myself fixing and running buildah container on k8s to create docker image(s) successfully by adding the following parameters:

spec:
  volumes:                          
  - name: volume
  containers:
  - name: buildah-image
    image: <REGISTRY_NAME>/<IMAGE_NAME>:<TAG>
    imagePullPolicy: Always
    securityContext:
      privileged: true
    args:
    command: [ "sleep", "infinity" ]
    env:
    resources:
    requests:
        cpu: 250m
        memory: 500Mi
    volumeMounts:                 
    - mountPath: /var/lib/containers
      name: volume
    command:
    - cat
    tty: true

I am attaching this k8s sample template so that if anyone else tries to run buildah container on k8s may use this and this will help for sure to run it on k8s.

Closing this Issue now.

[marshallford]

I am also seeing this error:

➜ docker run -it --privileged quay.io/buildah/stable:v1.9.0
[root@aa04fe2dafff /]# buildah bud -t foo:latest .
STEP 1: FROM alpine
Getting image source signatures
Copying blob 921b31ab772b done
Copying config 4d90542f06 done
Writing manifest to image destination
Storing signatures
STEP 2: RUN date
process exited with error: fork/exec /bin/sh: no such file or directorysubprocess exited with status 1
                                                                                                      error building at STEP "RUN date": exit status 1
[root@aa04fe2dafff /]# cat Dockerfile
FROM alpine
RUN date
[root@aa04fe2dafff /]#

[TomSweeneyRedHat]

@marshallford This worked for me, but I think the seccomp bit is not happy in k8s:

podman run -it --rm --security-opt label=disable --security-opt seccomp=unconfined --device /dev/fuse:rw -v /var/lib/mycontainer:/var/lib/containers:Z  quay.io/buildah/stable

[marshallford]

@TomSweeneyRedHat Thanks. Apologizes for the similar questions in multiple threads. I was trying to tie similar errors together. See my comments here: https://github.com/containers/buildah/issues/1335#issuecomment-501537272 for a better explanation of what I am trying to accomplish. Day job security policies around limiting container permissions in k8s is driving a lot of my questions and confusion.

[mnuttall]

I've just seen this problem in quay.io/buildah/stable:v1.10.1 and quay.io/buildah/stable:latest. Reverting to quay.io/buildah/stable:v1.10.0 has fixed it for me.