Buildah: Is buildah meant to be Linux only tool?

[jorgemoralespou]

Given that buildah provides a binary that helps you create containers, and that many developers would use different Operating Systems on their laptop, is buildah targeting to work on multiple architectures, or it's just meant to run on linux?

[rhatdan]

We would be open to it running on any OS. Pull requests accepted.

[jorgemoralespou]

I would if I knew how, but you decided to do it in a different language than Java or bash, which are the only ones I can barely write.

[rhatdan]

Probably easiest thing to do would be launch VMs with linux in them to build linux containers.

[jorgemoralespou]

@rhatdan so, that means developers will always need to use Docker to build containers, even if on the "cloud platform" (e.g. OpenShift) we run containers with runc through CRI-O and we build them with buildah.

Thanks for the answer. I guess from a user/developer point of view, we don't need to look into buildah then.

[rhatdan]

No, You should be able to build a container through CRI-O/openshift. Even in the docker case the user is using a VM to build it docker does not running natively on a MAC or windows (yet).

I think we should continue to discuss what you would need from buildah.

[jorgemoralespou]

@rhatdan From a developer perspective, buildah needs to be a client tool that seamlessly build containers in Mac,Win and Linux.

I know that Docker uses a VM, but the client side of it works on mac, win and linux, hence whether the container builds in a VM or not, the experience for a user is transparent.

One of my "considerations" of us going cri-o, is the developer side of it. A developer creates containers in 2 different ways, either on his laptop or on a hosted platform (k8s,openshift,...). As you say, the hosted building is no problem, as the user will submit a Dockerfile to be built. How that's done, he don't care, and probably he'll be using a different experience that Docker (e.g. oc, draft,...)

On the other hand, when a developer works on his local machine, he'll want the easiest experience, which is what he currently get's using Docker. That means I will want to build and run containers locally. To run, I can use minishift, minikube, oc cluster, ...., (eventually kpod, runc,...). So that means that as a developer I will provide a Dockerfile for it to build.

As a developer I'm, left with 2 options, one, keep using Docker/moby or the other would be to build a container with buildah. If I go the Docker way, I can do "docker build ... && docker push" in my mac (even if things work on a B2D image). I would want to do the same with buildah, and not having to "create me a VM with a linux, install buildah, and use it in a VM via ssh or similar".

These developers can be of two types: the first, those that will never operate a platform in any way (minikube,minishift, k8s, openshift,...) so those will keep using Docker without an issue. But there are also those that will want to do builds and use same tools and technology on both sides, on their local as well as on the hosted platform, as otherwise they'll have to learn yet another way.

Do you see where I'm going?

[jorgemoralespou]

Also, does this mean that buildah is meant to build only linux containers? Or are also windows containers in the scope?

[rhatdan]

Not something Linux engineers are working on, but if Windows engineers want to contribute, Patches welcome. :^)

[henrik242]

@jorgemoralespou Did you get buildah running on OSX?

[jorgemoralespou]

No

[devurandom]

A recent blog post by @haraldh on a Podman remote access solution that works also on macOS and Windows gives hope that this might eventually be addressed, i.e. I can see how varlink could be used to build a solution similar to "Docker for Mac", running the actual builder in a Linux VM on macOS: https://podman.io/blogs/2019/01/16/podman-varlink.html

[rhatdan]

@devurandom Yes we are concentrating on making podman work on Macs right now, which would inclue podman build. This is based on top of the varlink calls talked about in this blog. If you have a MAC and want to play with podman, the code can be built on a MAC Now.

[konrad-ohms]

What about using docker for portability at the moment as long as buildah does not work nativity on the desired platforms. I guess it could help to increase the adoption of buildah. I am mainly using buildah because of its scripting features, usually I am on Linux and this can be tested on my workstation directly. Nevertheless, if you are not on Linux and want to use buildah, you could add a kind of wrapper script around it which leverages docker.

Example

Directory structure:

├── buildah-docker-wrapper.sh
├── build-buildah.sh
├── Dockerfile
└── index.js

1) Package buildah into a docker image, e.g. with that Dockerfile (The build instructions can be found at https://github.com/containers/buildah/blob/master/install.md#ubuntu-1):

Dockerfile:

FROM ubuntu:18.04
RUN apt-get update && \
    apt-get -y install software-properties-common && \
    add-apt-repository -y ppa:alexlarsson/flatpak && \
    add-apt-repository -y ppa:gophers/archive && \
    apt-add-repository -y ppa:projectatomic/ppa && \
    apt-get -y -qq update && \
    apt-get -y install \
        bats \
        btrfs-tools \
        git \
        libapparmor-dev \
        libdevmapper-dev \
        libglib2.0-dev \
        libgpgme11-dev \
        libostree-dev \
        libseccomp-dev \
        libselinux1-dev \
        skopeo-containers \
        go-md2man \
        golang-1.10 && \
    apt-get clean
RUN mkdir ~/buildah && \
    cd ~/buildah && \
    export GOPATH=`pwd` && \
    git clone --branch v1.7.2 https://github.com/containers/buildah ./src/github.com/containers/buildah && \
    cd ./src/github.com/containers/buildah && \
    PATH=/usr/lib/go-1.10/bin:$PATH make runc all SECURITYTAGS="apparmor seccomp" && \
    make install install.runc && \
    buildah --help && \
    buildah --version

2) Add an example application, e.g. a Node.js hello world app, see: https://nodejs.org/es/docs/guides/getting-started-guide/

index.js:

const http = require('http');
const hostname = '0.0.0.0';
const port = 3000;
const server = http.createServer((req, res) => {
  res.statusCode = 200;
  res.setHeader('Content-Type', 'text/plain');
  res.end('Hello World\n');
});
server.listen(port, hostname, () => {
  console.log(`Server running at http://${hostname}:${port}/`);
});

3) Add a buildah shell script which would be executed directly with buildah.

build-buildah.sh:

#!/bin/bash
set -xe
CONTAINER=`buildah from docker.io/node:10.15.3-alpine`
buildah copy ${CONTAINER} `pwd` /home/node/
buildah config --workingdir /home/node ${CONTAINER}
buildah run ${CONTAINER} -- npm install
buildah run ${CONTAINER} -- chown -R node /home/node
buildah config \
    --user node \
    --port 3000 \
    --cmd "node index.js" \
    ${CONTAINER}
buildah commit --rm ${CONTAINER} mynodeimage

4) And finally add a docker wrapper.

buildah-docker-wrapper.sh:

#!/bin/bash
set -xe
# Build container with buildah installed
docker build -t buildah:1.7.2 .
echo "buildah has been compiled on ubuntu:18.04, now building app image with buildah"

# Privileged execution, because of CLONE_NEWUSER syscall
# -v /var/lib/containers because of overlay filesystem
DOCKER_CONTAINER_NAME=`docker run -d --privileged -v /var/lib/containers buildah:1.7.2 sleep 1000`

# Define trap handler to remove buildah container on script exit
function removeContainer {
    docker rm -f ${DOCKER_CONTAINER_NAME}
}
trap removeContainer EXIT

# Transfer files into docker build container
docker cp `pwd` ${DOCKER_CONTAINER_NAME}:/app
# Execute buildah build, on Linux host this could be executed nativly
docker exec -it ${DOCKER_CONTAINER_NAME} /bin/bash -c "cd /app && ./build-buildah.sh"
# Workaround to get back docker image on the host, export docker image as tar file
docker exec -it ${DOCKER_CONTAINER_NAME} buildah push mynodeimage docker-archive:/root/image.tar:mynodeimage
# Copy tar file to local host
docker cp ${DOCKER_CONTAINER_NAME}:/root/image.tar .
# Load image with local docker daemon
docker load -i image.tar
# Cleanup transfered image
rm -f image.tar

set +x
echo "Docker image has been built successfully with buildah!"
echo "Run the image with: docker run --rm -p 3000:3000 mynodeimage"

Please note that the wrapper script uses the --privileged flag, so it might not be the safest way to run buildah, but I think it should work on Mac as well (did not try that). It also uses a workaround to copy a tar file back on the host and import it with the docker cli in order to make the produced image available to the docker host.

Example execution (on RHEL 7)

The first time ./buildah-docker-wrapper.sh is executed, it will run a usual docker build. Afterwards builds will be using the docker build cache and directly started a buildah container.

If your build server is using RHEL or Ubuntu, you can execute ./build-buildah.sh directly.

$ ./buildah-docker-wrapper.sh 
+ docker build -t buildah:1.7.2 .
Sending build context to Docker daemon  7.168kB
Step 1/3 : FROM ubuntu:18.04
 ---> 47b19964fb50
Step 2/3 : RUN apt-get update &&     apt-get -y install software-properties-common &&     add-apt-repository -y ppa:alexlarsson/flatpak &&     add-apt-repository -y ppa:gophers/archive &&     apt-add-repository -y ppa:projectatomic/ppa &&     apt-get -y -qq update &&     apt-get -y install         bats         btrfs-tools         git         libapparmor-dev         libdevmapper-dev         libglib2.0-dev         libgpgme11-dev         libostree-dev         libseccomp-dev         libselinux1-dev         skopeo-containers         go-md2man         golang-1.10 &&     apt-get clean
 ---> Using cache
 ---> f33879ddaabb
Step 3/3 : RUN mkdir ~/buildah &&     cd ~/buildah &&     export GOPATH=`pwd` &&     git clone --branch v1.7.2 https://github.com/containers/buildah ./src/github.com/containers/buildah &&     cd ./src/github.com/containers/buildah &&     PATH=/usr/lib/go-1.10/bin:$PATH make runc all SECURITYTAGS="apparmor seccomp" &&     make install install.runc &&     buildah --help &&     buildah --version
 ---> Using cache
 ---> 14eb76a8b536
Successfully built 14eb76a8b536
Successfully tagged buildah:1.7.2
+ echo 'buildah has been compiled on ubuntu:18.04, now building app image with buildah'
buildah has been compiled on ubuntu:18.04, now building app image with buildah
++ docker run -d --privileged -v /var/lib/containers buildah:1.7.2 sleep 1000
+ DOCKER_CONTAINER_NAME=8027eea826e5a1999e723ae9a8195d6965be221afa22a3ceaf5f9f93e827cc6f
+ trap removeContainer EXIT
++ pwd
+ docker cp /home/ohmsk/git/rba/buildah-demo/buildah-in-docker 8027eea826e5a1999e723ae9a8195d6965be221afa22a3ceaf5f9f93e827cc6f:/app
+ docker exec -it 8027eea826e5a1999e723ae9a8195d6965be221afa22a3ceaf5f9f93e827cc6f /bin/bash -c 'cd /app && ./build-buildah.sh'
++ buildah from docker.io/node:10.15.3-alpine
Getting image source signatures
Copying blob 5d2ca3ce7154 done
Copying blob ca2222b63ac5 done
Copying blob 8e402f1a9c57 done
Copying config 94f3c89564 done
Writing manifest to image destination
Storing signatures
+ CONTAINER=node-working-container
++ pwd
+ buildah copy node-working-container /app /home/node/
fc96c64aa176b4be461697dc709f69813d1f7fe6763a1a0bed9a08abb548bac6
+ buildah config --workingdir /home/node node-working-container
+ buildah run node-working-container -- npm install
npm WARN saveError ENOENT: no such file or directory, open '/home/node/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/home/node/package.json'
npm WARN node No description
npm WARN node No repository field.
npm WARN node No README data
npm WARN node No license field.

up to date in 0.828s
found 0 vulnerabilities

+ buildah run node-working-container -- chown -R node /home/node
+ buildah config --user node --port 3000 --cmd 'node index.js' node-working-container
+ buildah commit --rm node-working-container mynodeimage
Getting image source signatures
Copying blob bcf2f368fe23 skipped: already exists
Copying blob 980eb43e895d skipped: already exists
Copying blob d64a7e460154 skipped: already exists
Copying blob 97db84a37dc3 done
Copying config 6c9ab0783d done
Writing manifest to image destination
Storing signatures
6c9ab0783d7f143bc7609cd09ec4e3248d6cea5b95c8b3670778148393267b3d
+ docker exec -it 8027eea826e5a1999e723ae9a8195d6965be221afa22a3ceaf5f9f93e827cc6f buildah push mynodeimage docker-archive:/root/image.tar:mynodeimage
Getting image source signatures
Copying blob bcf2f368fe23 done
Copying blob 980eb43e895d done
Copying blob d64a7e460154 done
Copying blob 97db84a37dc3 done
Copying config 6c9ab0783d done
Writing manifest to image destination
Storing signatures
Successfully pushed /root/image.tar:docker.io/library/mynodeimage:latest@sha256:473cdd17be6ec9192554c08bcb208d5ee96d285e7f4fe81233848707a226a352
+ docker cp 8027eea826e5a1999e723ae9a8195d6965be221afa22a3ceaf5f9f93e827cc6f:/root/image.tar .
+ docker load -i image.tar
97db84a37dc3: Loading layer [==================================================>]  17.41kB/17.41kB
The image mynodeimage:latest already exists, renaming the old one with ID sha256:30797341064ed3a157b0d8ae217147996a766982167083d0052193c8f17cfbc7 to empty string
Loaded image: mynodeimage:latest
+ rm -f image.tar
+ set +x
Docker image has been built successfully with buildah!
Run the image with: docker run --rm -p 3000:3000 mynodeimage
8027eea826e5a1999e723ae9a8195d6965be221afa22a3ceaf5f9f93e827cc6f

Finally you can run docker run --rm -p 3000:3000 mynodeimage to start the Node.js app.

[konrad-ohms]

Looks like there is a more offical tutorial available now, at least for running buildah in a container: https://developers.redhat.com/blog/2019/04/04/build-and-run-buildah-inside-a-podman-container/

[brianbruggeman]

I think the goal, from a Mac and/or Windows perspective, is to use buildah because we don't want to have to run a resource intensive service like docker _just_ to build and manage the life cycle of an image.

More generally, if any tool worked natively on my desktop platform without the requirement for an extra service to run, that's a win (think of it like a standard compiler). If we have to use docker as a service to build, then buildah actually becomes a barrier: I've already got docker, why add the extra overhead.

I think people are looking at buildah because it looks like it could potentially provide that non-service path.

[rhatdan]

We are working with podman build to run on a Mac or Windows. Not Buildah, although podman build embeds the buildah code within it.
On a MAC and Windows we require you to use a VM, just Like docker does, that is running podman inside of it.

On Windows WSL2, people have had success with running podman there as well.

[jorgemoralespou]

You mean containers/libpod will create binaries for mac, windows and linux? AFAIK, it doesn't even have a binary that can be directly downloaded from GitHub, but comes only packaged in Linux distro managers. Where would we be able to find these binaries?

[rhatdan]

On a MAC you should be able to do
brew install podman.

If you go to https://github.com/containers/libpod
and search for Windows or MAC, there are links their to install.

[cristaloleg]

Well, yes, but no. Works via brew cask (works fine)

Error: No available formula with the name "podman"
Found a cask named "podman" instead. Try
  brew cask install podman