Buildah: overlay/overlay2 not supported by kernel 4.20.3 or extfs?

[AndydeCleyre]

Arch Linux
Ext4

I thought that the overlay(2) driver was pretty much always recommended, but since I upgraded buildah to 1.6, it seems to say it's not supported over extfs anymore, though I can't find info on that anywhere. Can anyone clear things up for me? I believe vfs performance is expected to be awful. What should I be using?

buildah containers

```
ERRO[0000] 'overlay' is not supported over extfs at "/home/andy/.local/share/containers/storage/overlay2"
ERRO[0000] 'overlay' is not supported over extfs at "/home/andy/.local/share/containers/storage/overlay2"
kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/andy/.local/share/containers/storage/overlay2": backing file system is unsupported for this graph driver
kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/andy/.local/share/containers/storage/overlay2": backing file system is unsupported for this graph driver
ERRO[0000] exit status 1

Version: 1.6
Go Version: go1.11.5
Image Spec: 1.0.0
Runtime Spec: 1.0.0
CNI Spec: 0.4.0
libcni Version: v0.7.0-alpha1
Git Commit: 5f95bd9a
Built: Sat Jan 26 13:04:12 2019
OS/Arch: linux/amd64

Linux Ingram 4.20.3-zen2-1-zen #1 ZEN SMP PREEMPT Sun Jan 20 22:17:49 UTC 2019 x86_64 GNU/Linux

**`/etc/containers/storage.conf`:**

```ini
[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
size = ""
override_kernel_check = "true"
mountopt = "nodev"
[storage.options.thinpool]
ostree_repo = ""
skip_mount_home = "false"

/~/.config/containers/storage.conf:

[storage]
  driver = "overlay2"
  runroot = "/run/user/1000"
  graphroot = "/home/andy/.local/share/containers/storage"
  [storage.options]
    mount_program = ""

[TomasTomecek]

There is a bug somewhere, I erased ~/.local/share/containers and ~/.config/containers to start from scratch and then it started to work as expected. I think there is an issue related to this.

Also, there is no overlay2 in podman/world; just overlay.

[AndydeCleyre]

@TomasTomecek work as expected with overlay, or did it change your config to vfs?

[TomasTomecek]

I think your issue is a dupe of https://github.com/containers/libpod/issues/2172

Overlay. I don't use vfs, it is slow and inefficient.

[rhatdan]

The overlay driver is doing some testing to see if the file system as full support on file system it is attempting to use. Could you turn on full debugging to see if containers/storage gives you more information on why it says the storage is not working?

@nalind @rhvgoyal. Should overlay work on extfs?

[rhatdan]

This fails in this function. What is the file system that this is happening on, It is being reported as extfs.

func supportsOverlay(home string, homeMagic graphdriver.FsMagic, rootUID, rootGID int) (supportsDType bool, err error) {
    // We can try to modprobe overlay first

    exec.Command("modprobe", "overlay").Run()

    layerDir, err := ioutil.TempDir(home, "compat")
    if err != nil {
        patherr, ok := err.(*os.PathError)
        if ok && patherr.Err == syscall.ENOSPC {
            return false, err
        }
    }
    if err == nil {
        // Check if reading the directory's contents populates the d_type field, which is required
        // for proper operation of the overlay filesystem.
        supportsDType, err = fsutils.SupportsDType(layerDir)
        if err != nil {
            return false, err
        }
        if !supportsDType {
            return false, overlayutils.ErrDTypeNotSupported("overlay", backingFs)
        }

        // Try a test mount in the specific location we're looking at using.
        mergedDir := filepath.Join(layerDir, "merged")
        lower1Dir := filepath.Join(layerDir, "lower1")
        lower2Dir := filepath.Join(layerDir, "lower2")
        defer func() {
            // Permitted to fail, since the various subdirectories
            // can be empty or not even there, and the home might
            // legitimately be not empty
            _ = unix.Unmount(mergedDir, unix.MNT_DETACH)
            _ = os.RemoveAll(layerDir)
            _ = os.Remove(home)
        }()
        _ = idtools.MkdirAs(mergedDir, 0700, rootUID, rootGID)
        _ = idtools.MkdirAs(lower1Dir, 0700, rootUID, rootGID)
        _ = idtools.MkdirAs(lower2Dir, 0700, rootUID, rootGID)
        flags := fmt.Sprintf("lowerdir=%s:%s", lower1Dir, lower2Dir)
        if len(flags) < unix.Getpagesize() {
            err := mountFrom(filepath.Dir(home), "overlay", mergedDir, "overlay", 0, flags)
            if err == nil {
                logrus.Debugf("overlay test mount with multiple lowers succeeded")
                return supportsDType, nil
            } else {
                logrus.Debugf("overlay test mount with multiple lowers failed %v", err)
            }
        }
        flags = fmt.Sprintf("lowerdir=%s", lower1Dir)
        if len(flags) < unix.Getpagesize() {
            err := mountFrom(filepath.Dir(home), "overlay", mergedDir, "overlay", 0, flags)
            if err == nil {
                logrus.Errorf("overlay test mount with multiple lowers failed, but succeeded with a single lower")
                return supportsDType, errors.Wrap(graphdriver.ErrNotSupported, "kernel too old to provide multiple lowers feature for overlay")
            } else {
                logrus.Debugf("overlay test mount with a single lower failed %v", err)
            }
        }
        logrus.Errorf("'overlay' is not supported over %s at %q", backingFs, home)
        return supportsDType, errors.Wrapf(graphdriver.ErrIncompatibleFS, "'overlay' is not supported over %s at %q", backingFs, home)
    }

    logrus.Error("'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.")
    return supportsDType, errors.Wrap(graphdriver.ErrNotSupported, "'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.")
}

[AndydeCleyre]

It's happening on ext4, as reported. I notice now this only happens when running buildah as a regular user (buildah-in-a-user-namespace).

I'll include the comment-stripped /etc/containers/storage.conf:

[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
size = ""
override_kernel_check = "true"
mountopt = "nodev"
[storage.options.thinpool]
ostree_repo = ""
skip_mount_home = "false"

Back to regular user:

buildah --debug images

running [buildah-in-a-user-namespace --debug images] with environment [<snip> _BUILDAH_STARTED_IN_USERNS=1 BUILDAH_ISOLATION=rootless], UID map [{HostID:1000 ContainerID:0 Size:1} {HostID:100000 ContainerID:1 Size:65536}], and GID map [{HostID:1002 ContainerID:0 Size:1} {HostID:100000 ContainerID:1 Size:65536}]

[graphdriver] trying provided driver "overlay"
overlay test mount with multiple lowers failed mountfrom re-exec error: exit status 1: output: operation not permitted
overlay test mount with a single lower failed mountfrom re-exec error: exit status 1: output: operation not permitted
'overlay' is not supported over extfs at "/home/andy/.local/share/containers/storage/overlay"
[graphdriver] trying provided driver "overlay"
overlay test mount with multiple lowers failed mountfrom re-exec error: exit status 1: output: operation not permitted
overlay test mount with a single lower failed mountfrom re-exec error: exit status 1: output: operation not permitted
'overlay' is not supported over extfs at "/home/andy/.local/share/containers/storage/overlay"
kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/andy/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/andy/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
exit status 1

Maybe I haven't properly configured user namespace stuff. What I've done:

/etc/subuid:
andy:100000:65536

/etc/subgid:
andy:100000:65536

/etc/sysctl.d/96-userns.conf:
kernel.unprivileged_userns_clone = 1

[ghost]

Well, Fedora workstation and Cloud image uses ext4 as the default rootfs. Many other distros use ext4 as default too. I don't see any reasons podman should not support ext4.

[rhatdan]

Do you have fuse-overlayfs installed on your box?

[rhatdan]

I can get something like this to happen if I have a bogus entry in ~/.config/containers/storage.conf

buildah bud .
ERRO[0000] 'overlay' is not supported over xfs at "/home/dwalsh/.local/share/containers/storage/overlay" 
ERRO[0000] 'overlay' is not supported over xfs at "/home/dwalsh/.local/share/containers/storage/overlay" 
kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/dwalsh/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
kernel does not support overlay fs: 'overlay' is not supported over xfs at "/home/dwalsh/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
ERRO[0000] exit status 1         

 cat ~/.config/containers/storage.conf
[storage]
  driver = "overlay"
  runroot = "/run/user/3267"
  graphroot = "/home/dwalsh/.local/share/containers/storage"
  [storage.options]
    mount_program = ""

If you remove this and install fuse-overlayfs does the problem go away?

[rhatdan]

Overlayfs is not supported in rootless mode UNLESS fuse-overlay is configured as the mount-program.

[AndydeCleyre]

YES, thank you! After nuking the config and installing fuse-overlayfs-git from the AUR, buildah generated the following, working-as-user config:

[storage]
  driver = "overlay"
  runroot = "/run/user/1000"
  graphroot = "/home/andy/.local/share/containers/storage"
  [storage.options]
    mount_program = "/bin/fuse-overlayfs"

So in all, the following will get rootless working on Arch:

sudo echo "$USER:100000:65536" >> /etc/subuid
sudo echo "$USER:100000:65536" >> /etc/subgid
sudo sysctl kernel.unprivileged_userns_clone=1
sudo echo "kernel.unprivileged_userns_clone = 1" >> /etc/sysctl.d/96-userns.conf
git clone [email protected]:fuse-overlayfs-git
cd fuse-overlayfs-git
makepkg -sci

That 96 is arbitrary, and I don't know if a particular ordering is recommended. I only have a couple other items there, at 90 and 99.

Thanks so much!