Buildah: overlay not working on ubuntu: Error processing tar file(exit status 1): operation not permitted

[ianmiell]

Description

When trying to use overlay, I get a permission error regarding permissions.

It defaults to vfs, which works (but uses far too much disk space).

I assume it's some kind of mapping error, but I haven't found a way to resolve.

Steps to reproduce the issue:

1. rm -rf ~/.local/shared/containers
2. podman build --storage-driver overlay .

Describe the results you received:

imiell@basquiat:~/tmp  🐧  podman build --storage-driver overlay .
STEP 1: FROM docker.io/ubuntu:xenial
Getting image source signatures
Copying blob edf72af6d627 done
Copying blob 3e4f86211d23 done
Copying blob 8d3eac894db4 done
Copying blob f7277927d38a done
Copying config 5e13f8dd4c done
Writing manifest to image destination
Storing signatures
Error: error creating build container: Error committing the finished image: error adding layer with blob "sha256:8d3eac894db4dc4154377ad28643dfe6625ff0e54bcfa63e0d04921f1a8ef7f8": Error processing tar file(exit status 1): operation not permitted

Describe the results you expected:

Simple docker image is built.

Output of rpm -q buildah or apt list buildah:

imiell@basquiat:~/tmp  🐧  apt list buildah
Listing... Done
buildah/bionic,now 1.9.0-1~dev~ubuntu18.04~ppa17 amd64 [installed]

Output of buildah version:

imiell@basquiat:~/tmp  🐧  buildah version
Version:         1.9.0-dev
Go Version:      go1.10.4
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Fri May 31 00:39:16 2019
OS/Arch:         linux/amd64

Output of podman version if reporting a podman build issue:

imiell@basquiat:~/tmp  🐧  podman version
Version:            1.4.3
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of cat /etc/*release:

imiell@basquiat:~/tmp  🐧  cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Output of uname -a:

imiell@basquiat:~/tmp  🐧  uname -a
Linux basquiat 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

imiell@basquiat:~/tmp  🐧  cat /etc/containers/storage.conf 
# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information

# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary read-write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# AdditionalImageStores is used to pass paths to additional read-only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers (currently overlay, zfs, vfs, btrfs)
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"```

It works fine as root.

Strace gives me more output than I can parse, and results in a different error:

Error: error running newgidmap: exit status 1: newgidmap: write to gid_map failed: Operation not permitted

ERRO[0000] error writing "0 1000 1n" to /proc/10449/gid_map: write /proc/10449/gid_map: operation not permitted
ERRO[0000] (unable to determine exit status)

For reference:

imiell@basquiat:/etc 🐧 cat subuid
imiell:100000:65536
jira:165536:65536
nexus:231072:65536
strongswan:296608:65536

imiell@basquiat:/etc 🐧 cat subgid
imiell:100000:65536
jira:165536:65536
nexus:231072:65536
strongswan:296608:65536

In my kernel config:

CONFIG_USER_NS=y
```

[ianmiell]

Looks like: https://github.com/containers/buildah/issues/1709

[TomSweeneyRedHat]

@ianmiell thanks for the issue. FWIW I edited the description slightly to get the back ticks in the right place for happier formatting. I didn't change any of the other text.

Do you by chance have a Dockerfile that you can share that you used in your test?

[ianmiell]

I’m out and about right now but it was a trivial one, something like

FROM docker.io/ubuntu:xenial
RUN ls
CMD bash

On Mon, 29 Jul 2019 at 15:24, Tom Sweeney [e-mail] wrote:

@ianmiell https://github.com/ianmiell thanks for the issue. FWIW I
edited the description slightly to get the back ticks in the right place
for happier formatting. I didn't change any of the other text.

Do you by chance have a Dockerfile that you can share that you used in
your test?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/buildah/issues/1745?email_source=notifications&email_token=ABVNZTIPYLMBQRY5FGPSK3TQB34QXA5CNFSM4IHNKLW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3A33NQ#issuecomment-516013494,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABVNZTNYABHQTAYY2SSGLCTQB34QXANCNFSM4IHNKLWQ
.

[giuseppe]

Looks like: #1709

it is indeed a duplicate of #1720

The issue is that an unprivileged user has no privileges for using mknod so we cannot (easily) create whiteout files when extracting an image.

Our suggestion is to use fuse-overlayfs on Ubuntu as well, you'd need to install fuse-overlayfs and add mount_program = "/usr/bin/fuse-overlayfs under [storage.options] in your ~/.config/containers/storage.conf file.

@lsm5 can we add a dependency to fuse-overlayfs?

We probably need a more descriptive error message since the issue was already reported twice.

[ianmiell]

imiell@basquiat:/space/git/alertonchange ⑂ master + 🐧 buildah --debug
bud .

DEBU[0000] running [buildah-in-a-user-namespace --debug bud .] with
environment [SHELL=/bin/bash HISTCONTROL=erasedups:ignorespace
HISTSIZE=1000000 HOSTNAME=basquiat LANGUAGE=en_GB:en
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
HISTTIMEFORMAT=%d/%m/%y %T EDITOR=vim
GOBIN=/space/go/bin PWD=/space/git/alertonchange LOGNAME=imiell
QT_QPA_PLATFORMTHEME=appmenu-qt5 XDG_SESSION_TYPE=tty
GEM_ROOT=/opt/chefdk/embedded/lib/ruby/gems/2.5.0 TZ=GB
CDPATH=.:/space/git:/space/git/work HOME=/home/imiell LANG=en_GB.UTF-8
HISTFILE=/home/imiell/.bash_history
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:*.xspf=00;36:
SSH_CONNECTION=86.137.120.162 49837 192.168.1.124 22 GOROOT=/space/go
GEM_PATH=/home/imiell/.chefdk/gem/ruby/2.5.0:/opt/chefdk/embedded/lib/ruby/gems/2.5.0
GEM_HOME=/home/imiell/.chefdk/gem/ruby/2.5.0 LESSCLOSE=/usr/bin/lesspipe %s
%s XDG_SESSION_CLASS=user TERM=xterm-256color LESSOPEN=| /usr/bin/lesspipe
%s LIBVIRT_DEFAULT_URI=qemu:///system USER=imiell VISUAL=vim SHLVL=1
PAGER=less XDG_SESSION_ID=12103 XDG_RUNTIME_DIR=/run/user/1000
SSH_CLIENT=86.137.120.162 49837 22 ANSIBLE_HOST_KEY_CHECKING=False
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
PATH=/Users/imiell/anaconda3/bin:/usr/local/bin:/usr/local/opt/openssl/bin:/home/imiell/.krew/bin:/opt/chefdk/bin:/home/imiell/.chefdk/gem/ruby/2.5.0/bin:/opt/chefdk/embedded/bin:/home/imiell/bin:/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/bin:~/.dotfiles/bin::/opt/chefdk/gitbin/space/git/work/bin:/space/git/work/bin/archive:/space/git/work/bin/asciidoctor:/space/git/work/bin/git:/space/git/work/bin/.gitignore:/space/git/work/bin/home:/space/git/work/bin/jenkins:/space/git/work/bin/man:/space/git/work/bin/media:/space/git/work/bin/music:/space/git/work/bin/notes:/space/git/work/bin/openstack:/space/git/work/bin/state_street:/space/git/work/bin/utils:/space/git/work/bin/vms:/space/git/home/bin:
HISTFILESIZE=1000000 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
MAIL=/var/mail/imiell SSH_TTY=/dev/pts/0 GOPATH=/home/imiell/go
_=/usr/bin/buildah OLDPWD=/home/imiell _CONTAINERS_USERNS_CONFIGURED=1
BUILDAH_ISOLATION=rootless], UID map [{ContainerID:0 HostID:1000 Size:1}
{ContainerID:1 HostID:100000 Size:65536}], and GID map [{ContainerID:0
HostID:1000 Size:1} {ContainerID:1 HostID:100000 Size:65536}]

DEBU[0000] [graphdriver] trying provided driver "vfs"

imiell@basquiat:/space/git/alertonchange ⑂ master + 🐧 cat !$

cat /etc/containers/storage.conf

storage.conf is the configuration file for all tools

that share the containers/storage libraries

See man 5 containers-storage.conf for more information

The "container storage" table contains all of the server options.

[storage]

Default Storage Driver

driver = "overlay"

mount_program = "/usr/bin/fuse-overlayfs"

Temporary storage location

runroot = "/var/run/containers/storage"

Primary read-write location of container storage

graphroot = "/var/lib/containers/storage"

[storage.options]

AdditionalImageStores is used to pass paths to additional read-only image

stores

Must be comma separated list.

additionalimagestores = [

]

Size is used to set a maximum size of the container image. Only

supported by

certain container storage drivers (currently overlay, zfs, vfs, btrfs)

size = ""

OverrideKernelCheck tells the driver to ignore kernel checks based on

kernel version

override_kernel_check = "true"

imiell@basquiat:/space/git/alertonchange ⑂ master + 🐧 ls -l
/usr/bin/fuse-overlayfs

-rwxr-xr-x 1 root root 63920 Jan 15 2019

/usr/bin/fuse-overlayfs

I've confirmed with strace that that's the config file being read.

On Tue, Jul 30, 2019 at 2:41 PM Giuseppe Scrivano [e-mail]
wrote:

Looks like: #1709 https://github.com/containers/buildah/issues/1709

it is indeed a duplicate of #1720
https://github.com/containers/buildah/pull/1720

The issue is that an unprivileged user has no privileges for using mknod
so we cannot (easily) create whiteout files when extracting an image.

Our suggestion is to use fuse-overlayfs on Ubuntu as well, you'd need to
install fuse-overlayfs and add mount_program = "/usr/bin/fuse-overlayfs
under [storage.options] in your ~/.config/containers/storage.conf file.

@lsm5 https://github.com/lsm5 can we add a dependency to fuse-overlayfs?

We probably need a more descriptive error message since the issue was
already reported twice.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/buildah/issues/1745?email_source=notifications&email_token=ABVNZTO4UE4HRLMUFSBW6NDQCBAG3A5CNFSM4IHNKLW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3D77RA#issuecomment-516423620,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABVNZTPQ52QW3ZQDT7IARDLQCBAG3ANCNFSM4IHNKLWQ
.

[giuseppe]

/etc/containers/storage.conf is the configuration for root containers.

When you are running rootless, Podman/Buildah will use the configuration files under $HOME/.config/containers.

An issue I see in the configuration file you are using is that mount_program = "/usr/bin/fuse-overlayfs" is not in the correct place. It must be under the [storage.options] TOML section.

Another solution is to delete the configuration files and let podman recreate them It will automatically configure the storage to use fuse-overlayfs when it is available:
rm -rf $HOME/.config/containers && podman system migrate

[ianmiell]

If $HOME/.config/containers/storage.conf does not exist, then strace reports that it looks at /etc/containers/storage.conf. Maybe it acts differently under strace (I notice I get a different error message, presumably due to some interaction between strace and the syscalls (?).

In any case, the last suggestion rm -rf... worked a treat. Thanks.

[giuseppe]

unfortunately strace breaks creating the user namespace, as setuid binaries (such as /usr/bin/newuidmap and /usr/bin/newgidmap) cannot be traced.

/etc/containers/storage.conf is used only to cherry pick some settings, like runtime paths, but the whole configuration file cannot be used as most of the settings won't work when running without root privileges

[ianmiell]

Ah, thanks - that's interesting. Explains why it was complaining to me
about new*idmap.

On Wed, Jul 31, 2019 at 1:58 PM Giuseppe Scrivano [e-mail]
wrote:

unfortunately strace breaks creating the user namespace, as setuid
binaries (such as /usr/bin/newuidmap and /usr/bin/newgidmap) cannot be
traced.

/etc/containers/storage.conf is used only to cherry pick some settings,
like runtime paths, but the whole configuration file cannot be used as most
of the settings won't work when running without root privileges

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/buildah/issues/1745?email_source=notifications&email_token=ABVNZTLSWIKMSGJRZJKXLHLQCGEADA5CNFSM4IHNKLW2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3HFBRA#issuecomment-516837572,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABVNZTNTLJKUEFTAFFUNX2DQCGEADANCNFSM4IHNKLWQ
.

[TeNNoX]

I like that you wrote a troubleshooting section, but do you have plans to detect / use fuse-overlay as default on ubuntu?
I ran into this too, and took me now ~2 hours to debug and find this.

[rhatdan]

I guess the question is does the Podman package require the fuse-overlay package. On Fedora we have this as a Recommends, which means it is installed by default and Podman will use it.