Buildah: Buildah Info not working for rootless

[TomSweeneyRedHat]

Description

Running as non-root, the buildah info command fails.

Steps to reproduce the issue:

1. Run a non-root
2. buildah info
   3.

Describe the results you received:

buildah info

ERRO[0000] 'overlay' is not supported over xfs at "/home/tom/.local/share/containers/storage/overlay" 
ERRO[0000] [graphdriver] prior storage driver overlay failed: kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/tom/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver 
ERRO[0000] 'overlay' is not supported over xfs at "/home/tom/.local/share/containers/storage/overlay" 
ERRO[0000] [graphdriver] prior storage driver overlay failed: kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/tom/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver 
kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/tom/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/tom/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
ERRO[0000] exit status 1                                

Describe the results you expected:

Information to be displayed appropriately from buildah info

Output of rpm -q buildah or apt list buildah:

# rpm -q buildah
buildah-1.6-1.git3ec6a6b.fc29.x86_64

Output of buildah version:

# buildah version 
Version:         1.6
Go Version:      go1.11.4
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Wed Dec 31 19:00:00 1969
OS/Arch:         linux/amd64

Output of cat /etc/*release:

# cat /etc/*release
Fedora release 29 (Twenty Nine)
NAME=Fedora
VERSION="29 (Server Edition)"
ID=fedora
VERSION_ID=29
PLATFORM_ID="platform:f29"
PRETTY_NAME="Fedora 29 (Server Edition)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:29"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f29/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=29
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=29
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Server Edition"
VARIANT_ID=server
Fedora release 29 (Twenty Nine)
Fedora release 29 (Twenty Nine)

Output of cat /etc/containers/storage.conf:

# cat /etc/containers/storage.conf
# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base.
# device.
# mkfsarg = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver 
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

# If specified, use OSTree to deduplicate files with the overlay backend
ostree_repo = ""

# Set to skip a PRIVATE bind mount on the storage home directory.  Only supported by
# certain container storage drivers
skip_mount_home = "false"

[TomSweeneyRedHat]

@giuseppe any thoughts?

[giuseppe]

@TomSweeneyRedHat it works well for me. How does your ~/.config/containers/storage.conf look like?

[TomasTomecek]

I feel like it's finally time to change the functionality so that buildah (and podman) errors out when the parser can't process the storage.conf; this is like the 3rd issue related to this.

[TomSweeneyRedHat]

@giuseppe I'm not able to replicate it now. @ipbabble ran into this last week too, seems like it might be related to upgrading from 1.5 to 1.6, but I've tried to get it to break in that environment too without luck.

[giuseppe]

I've tried with an invalid configuration file:

Failed to parse /home/gscrivano/.config/containers/storage.conf Near line 0 (last key parsed ''): Bare keys cannot contain '\n'.
Failed to parse /home/gscrivano/.config/containers/storage.conf Near line 0 (last key parsed ''): Bare keys cannot contain '\n'.
Failed to parse /home/gscrivano/.config/containers/storage.conf Near line 0 (last key parsed ''): Bare keys cannot contain '\n'.

or to chmod it to 000 and:

Failed to read /home/gscrivano/.config/containers/storage.conf open /home/gscrivano/.config/containers/storage.conf: permission denied

so the problem that was seen here should be different.

Was perhaps only overlay specified but no mount_program?

In that case, the error is correct as the kernel implementation of overlay cannot be used by unprivileged users.

[rhatdan]

Yes I think this situation is happening when someone runs podman without fuse-ovleray installed and the storage.conf file gets created as overlay. I don't know how this is happening.

[jkonecny12]

I'm having the same issue for any buildah command as non-root. I used the old version before without any problems.

buildah images

ERRO[0000] 'overlay' is not supported over xfs at "/home/jkonecny/.local/share/containers/storage/overlay" 
ERRO[0000] [graphdriver] prior storage driver overlay failed: kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/jkonecny/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver 
ERRO[0000] 'overlay' is not supported over xfs at "/home/jkonecny/.local/share/containers/storage/overlay" 
ERRO[0000] [graphdriver] prior storage driver overlay failed: kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/jkonecny/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver 
kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/jkonecny/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/jkonecny/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
ERRO[0000] exit status 1

cat ~/.config/containers/storage.conf

RunRoot = "/run/user/1000"
GraphRoot = "/home/jkonecny/.local/share/containers/storage"
GraphDriverName = "overlay"
GraphDriverOptions = ["overlay.mount_program=/usr/bin/fuse-overlayfs"]

cat /etc/containers/storage.conf

# This file is is the configuration file for all tools
# that use the containers/storage library.
# See man 5 containers-storage.conf for more information
# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# Storage options to be passed to underlying storage drivers

# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# Path to an helper program to use for mounting the file system instead of mounting it
# directly.
#mount_program = "/usr/bin/fuse-overlayfs"

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

# mountopt specifies comma separated list of extra mount options
mountopt = "nodev"

# Remap-UIDs/GIDs is the mapping from UIDs/GIDs as they should appear inside of
# a container, to UIDs/GIDs as they should appear outside of the container, and
# the length of the range of UIDs/GIDs.  Additional mapped sets can be listed
# and will be heeded by libraries, but there are limits to the number of
# mappings which the kernel will allow when you later attempt to run a
# container.
#
# remap-uids = 0:1668442479:65536
# remap-gids = 0:1668442479:65536

# Remap-User/Group is a name which can be used to look up one or more UID/GID
# ranges in the /etc/subuid or /etc/subgid file.  Mappings are set up starting
# with an in-container ID of 0 and the a host-level ID taken from the lowest
# range that matches the specified name, and using the length of that range.
# Additional ranges are then assigned, using the ranges which specify the
# lowest host-level IDs first, to the lowest not-yet-mapped container-level ID,
# until all of the entries have been used for maps.
#
# remap-user = "storage"
# remap-group = "storage"

[storage.options.thinpool]
# Storage Options for thinpool

# autoextend_percent determines the amount by which pool needs to be
# grown. This is specified in terms of % of pool size. So a value of 20 means
# that when threshold is hit, pool will be grown by 20% of existing
# pool size.
# autoextend_percent = "20"

# autoextend_threshold determines the pool extension threshold in terms
# of percentage of pool size. For example, if threshold is 60, that means when
# pool is 60% full, threshold has been hit.
# autoextend_threshold = "80"

# basesize specifies the size to use when creating the base device, which
# limits the size of images and containers.
# basesize = "10G"

# blocksize specifies a custom blocksize to use for the thin pool.
# blocksize="64k"

# directlvm_device specifies a custom block storage device to use for the
# thin pool. Required if you setup devicemapper.
# directlvm_device = ""

# directlvm_device_force wipes device even if device already has a filesystem.
# directlvm_device_force = "True"

# fs specifies the filesystem type to use for the base device.
# fs="xfs"

# log_level sets the log level of devicemapper.
# 0: LogLevelSuppress 0 (Default)
# 2: LogLevelFatal
# 3: LogLevelErr
# 4: LogLevelWarn
# 5: LogLevelNotice
# 6: LogLevelInfo
# 7: LogLevelDebug
# log_level = "7"

# min_free_space specifies the min free space percent in a thin pool require for
# new device creation to succeed. Valid values are from 0% - 99%.
# Value 0% disables
# min_free_space = "10%"

# mkfsarg specifies extra mkfs arguments to be used when creating the base.
# device.
# mkfsarg = ""

# use_deferred_removal marks devicemapper block device for deferred removal.
# If the thinpool is in use when the driver attempts to remove it, the driver 
# tells the kernel to remove it as soon as possible. Note this does not free
# up the disk space, use deferred deletion to fully remove the thinpool.
# use_deferred_removal = "True"

# use_deferred_deletion marks thinpool device for deferred deletion.
# If the device is busy when the driver attempts to delete it, the driver
# will attempt to delete device every 30 seconds until successful.
# If the program using the driver exits, the driver will continue attempting
# to cleanup the next time the driver is used. Deferred deletion permanently
# deletes the device and all data stored in device will be lost.
# use_deferred_deletion = "True"

# xfs_nospace_max_retries specifies the maximum number of retries XFS should
# attempt to complete IO when ENOSPC (no space) error is returned by
# underlying storage device.
# xfs_nospace_max_retries = "0"

# If specified, use OSTree to deduplicate files with the overlay backend
ostree_repo = ""

# Set to skip a PRIVATE bind mount on the storage home directory.  Only supported by
# certain container storage drivers
skip_mount_home = "false"

buildah version

Version:         1.6
Go Version:      go1.11.4
Image Spec:      1.0.0
Runtime Spec:    1.0.0
CNI Spec:        0.4.0
libcni Version:  
Git Commit:      
Built:           Thu Jan  1 01:00:00 1970
OS/Arch:         linux/amd64

I'm running F29 newest updates.

Interesting thing is that when I changed GraphDriverName = "overlay" to vfs in ~/.config/containers/storage.conf I'm still getting the same error. Not sure if that is intentional.

[jkonecny12]

I don't mind loosing all containers and settings. Nothing have changed from default as far as I'm aware and containers are just working containers. What is the correct way to fix this. Re-install the podman and buildah?

[rhatdan]

Install fuse-overlayfs and then remove the ~/.config/containers/storage.conf and see if it works.

[jkonecny12]

HA! It works!
Thanks a lot @rhatdan

[jkonecny12]

If someone is interested the newly created configuration file looks like this:

cat ~/.config/containers/storage.conf

[storage]
  driver = "overlay"
  runroot = "/run/user/1000"
  graphroot = "/home/jkonecny/.local/share/containers/storage"
  [storage.options]
    mount_program = "/usr/bin/fuse-overlayfs"

[jkonecny12]

As @TomasTomecek already wrote in https://github.com/containers/buildah/issues/1286#issuecomment-458566636 we really should test if the configuration file is valid instead of giving overlayfs errors.

[rhatdan]

Please open up a separate issue on that. The original issue is fixed.