Buildah: Permission problems for buildah in rootless container

[matpen]

Description

My current goal is to run buildah in a rootless podman container, possibly as non-root inside the container, but I am facing some permission issues. #1480 is similar, but the symptoms are different.

Steps to reproduce the issue:

In the following I try several variations, attempting to lower the granted permissions at each step:

1. Prepare a test image and install buildah in it:

container="$(buildah from docker.io/library/ubuntu:18.04)"
buildah run "$container" -- bash -e <<- EOF
    # Install buildah
    apt-get update -qq
    apt-get install -yqq software-properties-common uidmap
    add-apt-repository -y ppa:projectatomic/ppa
    apt-get update -qq
    apt-get install -yqq buildah slirp4netns libdevmapper1.02.1 runc

    # Add an unprivileged user
    useradd -ms /bin/bash matteo
EOF
buildah copy "$container" "/usr/bin/fuse-overlayfs" "/usr/bin/fuse-overlayfs"
buildah commit --rm "$container" "test"

also, share this image with the host root user (btw, it would be nice if we could skip this, see #1224, closed but seemingly unresolved):

temp="$(mktemp)"
podman save test > "$temp"
sudo podman load < "$temp"
rm "$temp"

1. First attempt; rootful, privileged container (works):

sudo podman run --rm --privileged --volume /tmp/test/run_root:/var/run/containers/storage:rw --volume /tmp/test/storage_root:/var/lib/containers/storage:rw test bash -ec '
    container="$(buildah from docker.io/library/ubuntu:18.04)"
    buildah run "$container" apt-get -y update
    buildah commit --rm "$container" "new-image"
    buildah rmi "new-image"
'

1. Second attempt; rootless, privileged container (does not work): simply remove sudo from the previos command. On my system (I have virtualbox installed), I get:

Error: open /dev/vboxusb: permission denied

However I figured that this is probably expected, and even without the virtualbox entries, there will be other devices that cannot (and should not) be shared with the rootless container. So I tried to replace the --privileged with --cap-add SYS_ADMIN (see next attempt).

1. Third attempt; rootful, min capabilites, chroot isolation (works):

sudo podman run --rm --cap-add SYS_ADMIN --volume /tmp/test/run_root:/var/run/containers/storage:rw --volume /tmp/test/storage_root:/var/lib/containers/storage:rw test bash -ec '
    container="$(buildah from docker.io/library/ubuntu:18.04)"
    buildah run --isolation=chroot "$container" apt-get -y update # Note the --isolation option
    buildah commit --rm "$container" "new-image"
    buildah rmi "new-image"
'

the above works, but without the --isolation=chroot or with --isolation=rootless I would get:

error running container: error creating container for [/usr/bin/apt-get -y update]: container_linux.go:345: starting container process caused "process_linux.go:281: applying cgroup configuration for process caused \"mkdir /sys/fs/cgroup/cpuset/buildah-buildah999412522: read-only file system\""

not sure if this is expected.

1. Fourth attempt; rootless, min capabilites, chroot isolation (does not work):

mkdir -p /tmp/test2/run_rootless /tmp/test/storage_rootless
podman run --rm --cap-add SYS_ADMIN --volume /tmp/test2/run_rootless:/var/run/containers/storage:rw --volume /tmp/test/storage_rootless:/var/lib/containers/storage:rw test bash -ec '
    container="$(buildah from docker.io/library/ubuntu:18.04)"
    buildah run --isolation=chroot "$container" apt-get -y update
    buildah commit --rm "$container" "new-image"
    buildah rmi "new-image"
'

this yields

error committing container "ubuntu-working-container" to "new-image": error copying layers and metadata for container "19fcd6b5196dcc6...": Error committing the finished image: error adding layer with blob "sha256:32f5e4917e...": Error processing tar file(exit status 1): operation not permitted

Searching the internet for the error message led me to this issue for docker, but I do not think it is related.

1. Fifth attempt; rootless, min capabilites, chroot isolation, unprivileged user (does not work):

podman run --rm --cap-add SYS_ADMIN --volume /tmp/test2/run_rootless:/var/run/containers/storage:rw --volume /tmp/test/storage_rootless:/var/lib/containers/storage:rw --user matteo:matteo --userns=keep-id test bash -ec '
    container="$(buildah from docker.io/library/ubuntu:18.04)"
    buildah run --isolation=chroot "$container" apt-get -y update
    buildah commit --rm "$container" "new-image"
    buildah rmi "new-image"
'

which fails with error:

Error committing the finished image: error adding layer with blob "sha256:35c102085707f703de2d9eaad8752d6fe1b8f02b5d2149f1d8357c9cc7fb7d0a": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
time="2019-08-23T20:07:18Z" level=error msg="exit status 1"

Describe the results you received:

Various permission problems as described above.

Describe the results you expected:

Some way to run buildah in rootless containers would be useful (e.g. for CI).

Output of rpm -q buildah or apt list buildah:

buildah/bionic,now 1.10.1-1~ubuntu18.04~ppa1 amd64 [installed]

Output of buildah version:

Note that the buildah versions on the host and inside the container are exactly the same (at the time of this writing, they are the latest available releases).

Version:         1.10.1
Go Version:      go1.10.4
Image Spec:      1.0.1
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Thu Aug  8 22:29:48 2019
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

Version:            1.5.0
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of cat /etc/*release:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Output of uname -a:

Linux matteo-laptop 5.0.0-25-generic #26~18.04.1-Ubuntu SMP Thu Aug 1 13:51:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information

# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary read-write location of container storage
graphroot = "/home/matteo/MyData/data/podman/root"

[storage.options]
# AdditionalImageStores is used to pass paths to additional read-only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers (currently overlay, zfs, vfs, btrfs)
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

[rhatdan]

Could you try

podman run --rm --device /dev/fuse --volume /tmp/test2/run_rootless:/var/run/containers/storage:rw,Z --volume /tmp/test/storage_rootless:/var/lib/containers/storage:rw,Z -ti quay.io/buildah/stable bash -ec '

[matpen]

Thanks for the hint, @rhatdan. I tried the following:

1. Running the suggested command as root; I get the following error while executing buildah from inside the image

Error during unshare(CLONE_NEWUSER): Operation not permitted
ERRO[0000] error parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (unable to determine exit status)            

1. Adding the flag --cap-add SYS_ADMIN to the previous command, still running as root; this works both with --isolation=chroot and without.

2. Running the previous command as normal user: this works! No --isolation=chroot needed. In my environment, the ,Z flag for bind-propagation is also not required.

3. However, running the same command with "my image" does not work: I still get the error (even after trying and installing fuse):

error running container: error creating container for [/usr/bin/apt-get -y update]: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"join session keyring: create session key: operation not permitted\""
: exit status 1

So, it seems that there is "something" in the image on quay.io that makes it work: ideas?

[rhatdan]

The magic is in the Dockerfile, mainly configuring fuse-overlay to run, this eliminates the need for SYS_ADMIN.

https://github.com/containers/buildah/blob/master/contrib/buildahimage/stable/Dockerfile

# stable/Dockerfile
#
# Build a Buildah container image from the latest
# stable version of Buildah on the Fedoras Updates System.
# https://bodhi.fedoraproject.org/updates/?search=buildah
# This image can be used to create a secured container
# that runs safely with privileges within the container.
#
FROM fedora:latest

# Don't include container-selinux and remove
# directories used by dnf that are just taking
# up space.
RUN yum -y install buildah fuse-overlayfs --exclude container-selinux; rm -rf /var/cache /var/log/dnf* /var/log/yum.*

# Adjust storage.conf to enable Fuse storage.
RUN sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock

# Set up environment variables to note that this is
# not starting with usernamespace and default to
# isolate the filesystem with chroot.
ENV _BUILDAH_STARTED_IN_USERNS="" BUILDAH_ISOLATION=chroot

[matpen]

this eliminates the need for SYS_ADMIN

Wait, does that mean that the flag --cap-add SYS_ADMIN should not be needed? This appears not to be true on my system:

rm -rf /tmp/test/run /tmp/test/storage
mkdir -p /tmp/test/run /tmp/test/storage
podman run \
    --rm --interactive \
    --device /dev/fuse \
    --volume /tmp/test/run:/var/run/containers/storage:rw,Z \
    --volume /tmp/test/storage:/var/lib/containers/storage:rw,Z \
    quay.io/buildah/stable bash -e <<- "EOF"
        container="$(buildah from docker.io/library/ubuntu:18.04)"
        buildah run "$container" apt-get -y update
        buildah commit --rm "$container" "new-image"
        buildah rmi "new-image"
EOF

this fails with error:

Error during unshare(CLONE_NEWUSER): Operation not permitted
time="2019-08-30T13:52:30Z" level=error msg="error parsing PID \"\": strconv.Atoi: parsing \"\": invalid syntax"
time="2019-08-30T13:52:30Z" level=error msg="(unable to determine exit status)"

[rhatdan]

With new enough versions of podman and container-storage, and perhaps container-selinux, SYS_ADMIN should not be required.

[matpen]

With new enough versions of podman and container-storage, and perhaps container-selinux, SYS_ADMIN should not be required.

Thank you for the information. I think I will wait for these features to be released: I can live with the flag for now.

However, there are two more issues to iron out:

1. I am trying to remove the need to run buildah as root inside the container:

# Prepare image from quay, adding an unprivileged user
container="$(buildah from quay.io/buildah/stable)"
buildah run "$container" -- bash -e <<- EOF
    useradd -ms /bin/bash matteo
EOF
buildah commit --rm "$container" "test"

# TEST: run container directly as unprivileged user
rm -rf /tmp/test/run /tmp/test/storage
mkdir -p /tmp/test/run /tmp/test/storage
podman run \
    --rm \
    --device /dev/fuse --cap-add SYS_ADMIN \
    --volume /tmp/test/run:/var/run/containers/storage:rw \
    --volume /tmp/test/storage:/var/lib/containers/storage:rw \
    --user matteo:matteo --userns=keep-id \
    test \
    bash -ec '
        container="$(buildah from docker.io/library/ubuntu:18.04)"
        buildah run "$container" apt-get -y update
        buildah commit --rm "$container" "new-image"
        buildah rmi "new-image"
'

# TEST: run container directly as unprivileged user, try to fix with uidmap
rm -rf /tmp/test/run /tmp/test/storage
mkdir -p /tmp/test/run /tmp/test/storage
podman run \
    --rm \
    --device /dev/fuse --cap-add SYS_ADMIN \
    --volume /tmp/test/run:/var/run/containers/storage:rw \
    --volume /tmp/test/storage:/var/lib/containers/storage:rw \
    --user matteo:matteo --uidmap=0:1000:2000 \
    test bash -ec '
        container="$(buildah from docker.io/library/ubuntu:18.04)"
        buildah run "$container" apt-get -y update
        buildah commit --rm "$container" "new-image"
        buildah rmi "new-image"
'

whereby both tests fail with error:

Error committing the finished image: error adding layer with blob "sha256:35c102085707f7...": Error processing tar file(exit status 1): there might not be enough IDs available in the namespace (requested 0:42 for /etc/gshadow): lchown /etc/gshadow: invalid argument
time="2019-08-30T15:56:02Z" level=error msg="exit status 1"

1. Reproducing the image on quay.io from a bas Ubuntu image, trying to simulate the official Dockerfile:

# Prepare image from ubuntu
container="$(buildah from docker.io/library/ubuntu:18.04)"
buildah run "$container" -- bash -e <<- EOF
    # Install buildah
    apt-get -yqq update
    apt-get -yqq install software-properties-common uidmap
    add-apt-repository -y ppa:projectatomic/ppa
    apt-get -yqq update
    apt-get -yqq install buildah slirp4netns libdevmapper1.02.1 runc

    # Adjust storage.conf to enable Fuse storage.
    apt-get -yqq install fuse
    sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' /etc/containers/storage.conf
    mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers
    touch /var/lib/shared/overlay-images/images.lock
    touch /var/lib/shared/overlay-layers/layers.lock
EOF
buildah copy "$container" "/usr/bin/fuse-overlayfs" "/usr/bin/fuse-overlayfs"
buildah config --env _BUILDAH_STARTED_IN_USERNS="" --env BUILDAH_ISOLATION=chroot "$container"
buildah commit --rm "$container" "test"

# TEST: run rootless, as root user inside container
sudo rm -rf /tmp/test/run /tmp/test/storage
mkdir -p /tmp/test/run /tmp/test/storage
podman run \
    --rm \
    --device /dev/fuse --cap-add SYS_ADMIN \
    --volume /tmp/test/run:/var/run/containers/storage:rw \
    --volume /tmp/test/storage:/var/lib/containers/storage:rw \
    test \
    bash -ec '
        container="$(buildah from docker.io/library/ubuntu:18.04)"
        buildah run --isolation=chroot "$container" -- apt-get -y update
        buildah commit --rm "$container" "new-image"
        buildah rmi "new-image"
'

unfortunately buildah commit (inside the container) gives:

error committing container "ubuntu-working-container" to "new-image": error copying layers and metadata for container "a2c67f8a66b2d29e8c5e...": Error committing the finished image: error adding layer with blob "sha256:03fcdcd506bd613...": Error processing tar file(exit status 1): operation not permitted

[rhatdan]

@giuseppe PTAL

[matpen]

Friendly ping.

[TomSweeneyRedHat]

@giuseppe thoughts?

[gdm85]

This issue can be consistently reproduced using non-root podman v1.6.2 and the overlay storage driver; it is however documented here: https://github.com/containers/buildah/blob/999fa43623e88fb80f474373e08acace834e1810/troubleshooting.md#6-rootless-buildah-bud-fails-when-using-overlayfs

Maybe the error could be improved to point users in the right direction?

I could not build fuse-overlays because it requires fuse3 which is not available on the current Ubuntu LTS (bionic); can anyone verify that it will solve the issue, as it is explained in the troubleshooting documentation?

[matpen]

@gdm85 Unfortunately, fuser-overlayfs does not solve the problem. If you recheck this thread carefully, you will notice that I documented many attempts with it (e.g. in the OP where I copy it from my local system inside the container).

I could not build fuse-overlays because it requires fuse3 which is not available on the current Ubuntu LTS (bionic)

fuse3 is not required: see updated instructions for the build here. I too use Ubuntu 18.04, and perform the build with those steps.

[rhatdan]

@giuseppe Could you take a look.

[giuseppe]

if you are not using fuse-overlays, you need to make sure the vfs backend is used. Otherwise, I think it falls back to using native overlay which is allowed on Ubuntu for rootless users but we fail to deal with wthiteout files.

[rhatdan]

I believe @giuseppe answered the question, reopen if I am mistaken.