Buildah: ERRO[0000] 'overlay' is not supported over overlayfs

[cmoulliard]

When I execute within a fedora:28 docker container this command

sudo buildah --debug images

I get this error

DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: override_kernelcheck=true           
ERRO[0000] 'overlay' is not supported over overlayfs    
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: override_kernelcheck=true           
ERRO[0000] 'overlay' is not supported over overlayfs    
ERRO[0000] 'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver
'overlay' is not supported over overlayfs: backing file system is unsupported for this graph driver 

Here is the dockerfile used to create the buildah container on Macos

FROM fedora:28

RUN dnf install buildah sudo -y

#
# Create a user and group used to launch processes
# The user ID 1000 is the default for the first "regular" user on Fedora/RHEL,
# so there is a high chance that this ID will be equal to the current user
# making it easier to use volumes (no permission issues)
#
RUN groupadd -r default -g 1000 && useradd -u 1000 -r -g default -m -d /home/me -s /sbin/nologin -c "Default user" default

#
# Add user to sudo
#
RUN echo 'default:secret' | chpasswd
RUN echo '%default ALL=(ALL) ALL' >> /etc/sudoers

# Set the working directory to default' user home directory
WORKDIR /home/me
USER default

Docker info (command executed within macos's terminal)

docker info
Containers: 67
 Running: 11
 Paused: 0
 Stopped: 56
Images: 160
Server Version: 1.13.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: 
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version:  (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: N/A (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 3.10.0-693.21.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.954GiB
Name: minishift
ID: PWZQ:O5OI:AMTH:2IVK:E3FW:IX2G:BJ6X:JTWS:MW2N:YCY3:BDUI:LFAZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 provider=xhyve
Experimental: false
Insecure Registries:
 172.30.0.0/16
 127.0.0.0/8
Live Restore Enabled: false

[rhatdan]

You need to volume mount storage on top of /var/lib/containers for this to work.
Or you could use a storage driver of something other then overlay, like vfs.

[cmoulliard]

I changed the storage to use "vfs" and I can download some images but when this error is next reported -> failed to mount shm tmpfs

Info

[default@a0bd912f65f8 ~]$ sudo podman run -dt -e HTTPD_VAR_RUN=/var/run/httpd -e HTTPD_MAIN_CONF_D_PATH=/etc/httpd/conf.d                     -e HTTPD_MAIN_CONF_PATH=/etc/httpd/conf                     -e HTTPD_CONTAINER_SCRIPTS_PATH=/usr/share/container-scripts/httpd/                     registry.fedoraproject.org/f27/httpd /usr/bin/run-httpd
[sudo] password for default: 
Trying to pull registry.fedoraproject.org/f27/httpd...Getting image source signatures
Copying blob sha256:ff3dab903f926d26db009e0dcd575b8ccea7635b6b225835c35df765662707ec
 80.73 MB / 80.73 MB [=====================================================] 31s
Copying blob sha256:9347d6e9d8644ec22930b7dd78ba0cfcab8fd9a09d57545d4d74649ca929df0c
 7.30 MB / 7.30 MB [========================================================] 3s
Copying blob sha256:2fc5c44251d42d4e8bdc547a79e7ad1efffdf11f74ceea6a83e2fb6743aeff7d
 44.82 MB / 44.82 MB [===================================================] 2m14s
Copying config sha256:18f01f6f77ef941f5b31dd007d113620cbb1939ac33d1922edce7d9d2c9ebad7
 6.55 KB / 6.55 KB [========================================================] 0s
Writing manifest to image destination
Storing signatures
failed to mount shm tmpfs "/var/lib/containers/storage/vfs-containers/7a1b389dd27738244235fa6208f649637b0be651b74136c975f9ebd68fd43240/userdata/shm": operation not permitted

[default@a0bd912f65f8 ~]$ sudo ls -la /var/lib/containers/storage
[sudo] password for default: 
total 4
drwx------. 10 root root 149 Jul 17 19:41 .
drwx------.  3 root root  21 Jul 17 19:36 ..
drwxr-xr-x.  2 root root  27 Jul 17 19:41 libpod
drwx------.  2 root root   6 Jul 17 19:36 mounts
drwx------.  2 root root   6 Jul 17 19:36 overlay
-rw-------.  1 root root  64 Jul 18 04:25 storage.lock
drwx------.  2 root root   6 Jul 17 19:36 tmp
drwx------.  3 root root  17 Jul 17 19:42 vfs
drwx------.  3 root root 124 Jul 18 04:25 vfs-containers
drwx------.  4 root root 188 Jul 18 04:24 vfs-images
drwx------.  2 root root 384 Jul 18 04:25 vfs-layers
[default@a0bd912f65f8 ~]$ sudo ls -la /var/lib/containers/storage/vfs-containers
total 8
drwx------.  3 root root 124 Jul 18 04:25 .
drwx------. 10 root root 149 Jul 17 19:41 ..
drwx------.  3 root root  22 Jul 18 04:25 7a1b389dd27738244235fa6208f649637b0be651b74136c975f9ebd68fd43240
-rw-------.  1 root root 513 Jul 18 04:25 containers.json
-rw-------.  1 root root  64 Jul 18 04:25 containers.lock

[rhatdan]

Currently we only support these with SYS_ADMIN Capabilities. Although @giuseppe Is close to having it run without any privs. He has been looking at running podman while not root, but it should also run inside of a container.

[giuseppe]

https://github.com/containers/fuse-overlayfs can be used to mount overlay within an user namespace. It requires Linux 4.18, that is at rc5 right now. It works on Rawhide. The biggest problem I see with running it inside of a container is that the container itself is running on top of overlay, so we need to check how that will work with whiteouts files. There is a fallback mode to use files instead of devices in fuse-overlayfs, but should be checked on top of another overlay file system.

[rhatdan]

Closing this because I believe it has been answered.

[elgalu]

@giuseppe how can we do this by "use files instead of devices" via fuse-overlayfs ? I hit this Error: 'overlay' is not supported over overlayfs

[giuseppe]

@elgalu I am not sure I understand the question. Would you like to use fuse-overlayfs on top of the overlay file system?

[elgalu]

Yes, on top of the overlay running inside a K8s unprivileged pod, is that possible?

[rhatdan]

Podman and Buildah need to be launched with at least SETUID and SETGID capabilities. Even if they run in a User Namespace, setting up the Namespace will require these CAPs in order to use more then one UID inside of the container.

[elgalu]

thanks @rhatdan , the pod has those capabilities:

Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+i
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=1000(nbuser)
gid=100(users)
groups=100(users),2000(???)

[rhatdan]

As long as you mount a volume on /var/lib/containers inside of you container then it should work.

You are not allowed to run overlayfs on top of a file system that is already overlay.

[elgalu]

We're not allowed to mount host's paths in this K8s setup, is there any other way beyond slower solutions like VFS ?

[rhatdan]

Set up fuse-overlayfs inside of the container, I think fuse-overlay will work on overlay.
@giuseppe Correct?

[giuseppe]

yes, fuse-overlayfs should work on top of overlay.

[liubin]

Thanks all, this worked for me to run buildah inside a container:

docker run --privileged -it -v /tmp/csivvv:/var/lib/containers --rm  centos:centos7 bash
# # mount |grep containers
/dev/nvme0n1p2 on /var/lib/containers type ext4 (rw,relatime,errors=remount-ro)

Just use /var/lib/containers as a volume.

And I wonder how to use fuse-overlayfs inside of the container to let it works, I have tried but failed:

[root@9237b68aec97 /]# mkdir lower upper work merged
[root@9237b68aec97 /]# 
[root@9237b68aec97 /]# fuse-overlayfs -o lowerdir=lower,upperdir=upper,workdir=work /var/lib/containers
[root@9237b68aec97 /]# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/YEOKEZEWHFHL7WMM27BXFNQU6T:/var/lib/docker/overlay2/l/C4XFPXQ3OPSOKWEVRJ5B6FT5UW,upperdir=/var/lib/docker/overlay2/20f6a7c025dd2c6bdeebc32c00a561bb49119682c828c271432653f13f90c068/diff,workdir=/var/lib/docker/overlay2/20f6a7c025dd2c6bdeebc32c00a561bb49119682c828c271432653f13f90c068/work,xino=off)
fuse-overlayfs on /var/lib/containers type fuse.fuse-overlayfs (rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
[root@9237b68aec97 /]# buildah from --name myvolume --pull reg.docker.alibaba-inc.com/liubin/pouch-volume:latest
ERRO 'overlay' is not supported over <unknown> at "/var/lib/containers/storage/overlay" 
kernel does not support overlay fs: 'overlay' is not supported over <unknown> at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver

Any suggestion? @giuseppe @rhatdan

[giuseppe]

docker run --privileged -it -v /tmp/csivvv:/var/lib/containers --rm centos:centos7 bash

you'll need at least to add the /dev/fuse device to the container (--device /dev/fuse)