Buildah: Buildah'ing an image inside an unprivileged container

[cevich]

Description
I was thinking, it'd be handy to not need a bloated-host in order to build and push container images. This would enable deploying new images directly from inside a container-based CI environment (like travis), once testing passed.

Steps to reproduce the issue:

1. $ sudo podman run -it --rm fedora bash
2. # dnf install -y podman buildah
3. # sed -i -r 's/"overlay"/"vfs"/g' /etc/containers/storage.conf
4. Follow blog post about making a buildah container image.
5. # buildah commit $containerid buildah

Describe the results you received:

error committing container "working-container" to "buildah": error copying layer and metadata: error committing new image "containers-storage:[vfs@/var/lib/containers/storage+/var/run/containers/storage:vfs.override_kernel_check=true]docker.io/library/buildah:latest": error adding layer with blob "sha256:35f1120a42795a4020d54635d7cf52d32c2d7bd69deb8143fc738fe198cc6b38": ApplyLayer exit status 1 stdout:  stderr: Error creating mount namespace before pivot: operation not permitted

Describe the results you expected:
Success, so that I may follow with # podman push buildah <wherever>

Output of rpm -q buildah or apt list buildah:

[root@b78316e0e81e /]# rpm -q buildah podman
buildah-0.15-1.gitd1330a5.fc27.x86_64
podman-0.3.2-1.gitf79a39a.fc27.x86_64

Output of buildah version:

[root@b78316e0e81e /]# buildah version
Version:       0.15
Go Version:    go1.9.4
Image Spec:    1.0.0
Runtime Spec:  1.0.0
Git Commit:    d1330a5
Built:         Tue Feb 27 13:05:39 2018
OS/Arch:       linux/amd64

Output of cat /etc/*release:

# cat /etc/*release
Fedora release 27 (Twenty Seven)
NAME=Fedora
VERSION="27 (Twenty Seven)"
ID=fedora
VERSION_ID=27
PRETTY_NAME="Fedora 27 (Twenty Seven)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:27"
HOME_URL="https://fedoraproject.org/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=27
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=27
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
Fedora release 27 (Twenty Seven)
Fedora release 27 (Twenty Seven)

Output of uname -a:

[root@b78316e0e81e /]# uname -a
Linux b78316e0e81e 4.15.8-300.fc27.x86_64 #1 SMP Fri Mar 9 18:11:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

[root@b78316e0e81e /]# cat /etc/containers/storage.conf
# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information

# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "vfs"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

[cevich]

@nalind This is the thing we talked about the other day, that "should work". All details above, because I'm sure you remember perfectly :smile: NP if this is not possible / will never work / don't bother, I was just playing with an idea, so nothing critical.

[rhatdan]

@cevich I believe you would still need SYS_ADMIN to do this.
@giuseppe has played the most with this situation.

[giuseppe]

yes that won't work. You will still need CAP_SYS_ADMIN (or an user namespace) to run the containers created by buildah run.

I got it to work with bwrap-oci and bubblewrap (it requires this PR: https://github.com/projectatomic/bubblewrap/pull/256) running as non-root user in the host.
I wrote about the current status here: https://www.scrivano.org/2018/02/25/current-status-problems-running-buildah-non-root/

With the correct config.json you could run it directly with runc as non privileged user. As soon as the change in bubblewrap gets into a release, it will be quite easy to run the container with atomic run --user

[cevich]

@giuseppe 'cept I dunnawanna RUN the image, I just want to create it (then I'd just push it up to a registry). Still need CAP_SYS_ADMIN just for building?

[alanbchristie]

Sounds like something might be in the pipeline but I'd like to voice my desire for a root-free OCI builder(ah). I was attracted to buildah because of a presentation at a recent OpenShift Commons meeting. The ability to avoid the "big fat Docker daemon" was compelling but I've just swapped one problem for another.

Ideally I'd like to build from within a Docker container in an OpenShift CI/CD workflow (such as a slave-agent in a Jenkins pipeline). And, no, BuildConfigs don't solve the problem.

Like @cevich I'm not running the image at this stage - I will run some unit/functional testing (outside of the container) in a Jenkins slave-agent and then push the image. Surely that can/could be done as any user?

I've built a buildah-agent slave agent [https://github.com/alanbchristie/openshift-jenkins-buildah-slave] but I just run into the "can't run agent as root" problem. So (in another conversation) I've asked a question in the OpenShift discussion about running slave-agents as privileged users.

[rhatdan]

We are continuing to work on this problem. We hope to have buildah fully integrated into OpenShift and as we role out UserNS we should be able to get buildah to run in a "non-privilged" container.

[redbaron]

Another attempt to have pure unprivileged builds: https://github.com/GoogleCloudPlatform/kaniko

[cevich]

FWIW: The use-case I was attempting is similar to @alanbchristie just substitute Jenkins for Travis. Being able to build images inside unprivileged container, then push elsewhere is much faster b/c Travis gives priority to jobs that run inside containers (as opposed to on a full, bloated, host). There are other CI/CD systems that are even less flexible, only allowing testing in containers. So for me it was a performance workaround of sorts.

[rhatdan]

Another version of the same issue. @nalind remove the issues as dups if you want.

[cevich]

Yes please. I'd do it if I knew what it was a dupe of :smile:

[rhatdan]

Ok we have people doing this now, closing this issue, open new issues for newer features.

[TomSweeneyRedHat]

Darn it @rhatdan! You beat me to the punch. I thought this was fixed now and was in the midst of testing it when you closed this. I can verify that this now works with the latest and greatest bits of Buildah from GitHub.

[cevich]

W00T! Nice work guys!