Podman: no good choice for storage driver on backing btrfs

Created on 30 Dec 2019 · 15Comments · Source: containers/podman

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

If the backing filesystem is btrfs, there is no good choice for a backing driver:

overlay: not supported over btrfs
btrfs: unusable due to outstanding #3963
vfs: inefficient and discouraged

The recommended way forward is unclear.

Steps to reproduce the issue:

Have a system with btrfs as /
Try running podman run -it --rm alpine with overlay storage.
Try running podman run -it --rm alpine with btrfs storage.

Describe the results you received:
With overlay:
Error: could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over btrfs at "/home/toast/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
With btrfs:
ERRO[0003] Error removing container b76e0f109ea4810afe02b2dd9202c2323f69f66d5e9179545bb9feb06f51337a: error removing container b76e0f109ea4810afe02b2dd9202c2323f69f66d5e9179545bb9feb06f51337a root filesystem: Failed to destroy btrfs snapshot /home/toast/.local/share/containers/storage/btrfs/subvolumes for abfa41012e571c659f0d90be2f6c12a2b6e83e5171ecf13035f23b9aa4618c2c: operation not permitted

Describe the results you expected:
At least one of those to work.
As seen in #3963, fuse-overlayfs is strongly recommended, but it will not run on top of btrfs.
Meanwhile (in the same bug), btrfs itself remains unusable.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.6.3
RemoteAPI Version:  1
Go Version:         go1.13.4
Git Commit:         9d087f6a766259ba53b224944f1b7b778035c370-dirty
Built:              Sat Nov 23 06:56:15 2019
OS/Arch:            linux/amd64

Output of podman info --debug:
On btrfs:

debug:
  compiler: gc
  git commit: 9d087f6a766259ba53b224944f1b7b778035c370-dirty
  go version: go1.13.4
  podman version: 1.6.3
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.3, commit: eb5fa88c26fde5ce1e3f8a1d2a8a9498b2d7dbe6'
  Distribution:
    distribution: arch
    version: unknown
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 7982092288
  MemTotal: 33663705088
  OCIRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc9
      commit: d736ef14f0288d6993a1845745d6756cfc9ddd5a
      spec: 1.0.1-dev
  SwapFree: 15999168512
  SwapTotal: 15999168512
  arch: amd64
  cpus: 16
  eventlogger: journald
  hostname: ToastTop
  kernel: 5.4.6-arch3-1
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: Unknown
    Version: |-
      slirp4netns version 0.4.2
      commit: 69153b0d1cb82216d6782179ff7c3c5e91c731a1
  uptime: 54h 2m 49.91s (Approximately 2.25 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/toast/.config/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: btrfs
  GraphOptions: {}
  GraphRoot: /home/toast/.local/share/containers/storage
  GraphStatus:
    Build Version: 'Btrfs v5.3.1 '
    Library Version: "102"
  ImageStore:
    number: 2
  RunRoot: /run/user/1000
  VolumePath: /home/toast/.local/share/containers/storage/volumes

On overlay it simply errors out.

Package info (e.g. output of rpm -q podman or apt list podman):
pacman -Qi podman fuse-overlayfs

Name            : podman
Version         : 1.6.3-1
Description     : Tool and library for running OCI-based containers in pods
Architecture    : x86_64
URL             : https://github.com/containers/libpod
Licenses        : Apache
Groups          : None
Provides        : None
Depends On      : cni-plugins  conmon  device-mapper  iptables  libseccomp  ostree  runc  skopeo  btrfs-progs  slirp4netns  libsystemd
Optional Deps   : podman-docker: for Docker-compatible CLI
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 103.35 MiB
Packager        : Morten Linderud <[email protected]>
Build Date      : Sat 23 Nov 2019 06:56:15 AM EST
Install Date    : Sun 29 Dec 2019 07:44:46 PM EST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature

Name            : fuse-overlayfs
Version         : 0.7.2-1
Description     : FUSE implementation of overlayfs
Architecture    : x86_64
URL             : https://github.com/containers/fuse-overlayfs
Licenses        : GPL3
Groups          : None
Provides        : None
Depends On      : fuse3
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 87.47 KiB
Packager        : Unknown Packager
Build Date      : Sun 29 Dec 2019 08:27:10 PM EST
Install Date    : Sun 29 Dec 2019 08:27:23 PM EST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : None

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical.

kinbug

Source

CosmicToast

Most helpful comment

This was merged into podman on Jan 23rd. We are looking to cut a new release within the next week or two.

rhatdan on 28 Jan 2020

👍3

All 15 comments

Overlayfs and fuse-overlayfs definitely work over btrfs. I'm using Podman with Overlay for root and fuse-overlay for rootless atop a btrfs / and /home on my work laptop.

mheon on 30 Dec 2019

That error indicates that you're trying to use kernel overlayfs instead of fuse-overlayfs as rootless. We don't have permission to use kernel overlay without root. The error is a bit confusing, admittedly (AFAIK it's wrapping EPERM, which as root means we can't mount an overlay atop the given filesystem, but as rootless just means we don't have the mount capability).

mheon on 30 Dec 2019

@mheon I have fuse-overlayfs installed, how am I supposed to select fuse-overlayfs over kernel overlayfs?
containers-storage.conf(5) only mentions "overlay" as an option

CosmicToast on 30 Dec 2019

You should install fuse-overlayfs (if not already installed) and add a line like the following to your rootless storage.conf:

mount_program = "/usr/bin/fuse-overlayfs"

mheon on 30 Dec 2019

👍1

would it be possible to add this to the docs?
containers-storage.conf(5) is really barren considering I had to do the above manually
(yes, it works now)

CosmicToast on 30 Dec 2019

My complete rootless storage.conf, for reference:

[storage]
  driver = "overlay"
  runroot = "/run/user/1000/containers"
  graphroot = "/home/mheon/.local/share/containers/storage"
  [storage.options]
    size = ""
    remap-uids = ""
    remap-gids = ""
    remap-user = ""
    remap-group = ""
    ostree_repo = ""
    skip_mount_home = ""
    mount_program = "/usr/bin/fuse-overlayfs"
    mountopt = ""
    [storage.options.thinpool]
      autoextend_percent = ""
      autoextend_threshold = ""
      basesize = ""
      blocksize = ""
      directlvm_device = ""
      directlvm_device_force = ""
      fs = ""
      log_level = ""
      min_free_space = ""
      mkfsarg = ""
      mountopt = ""
      use_deferred_deletion = ""
      use_deferred_removal = ""
      xfs_nospace_max_retries = ""

mheon on 30 Dec 2019

Probably better to just fix the error message in c/storage to indicate that kernel overlay is in use if we try a mount without CAP_SYS_MOUNT so we can have sane error messages

mheon on 30 Dec 2019

Reopening until we can fix error messages in c/storage

mheon on 30 Dec 2019

The issue is that you ran podman without fuse-overlay installed, and it created the storage.conf in your homedir. Later you installed fuse-overlayfs but the storage.conf file exists so it was ignored. The way it is supposed to work is if the storage.conf file did not exists and the fuse-overlay was installed, podman would create the file correctly.

I would prefer that we don't create the fuse-overlay at all or created a version that is commented out and podman just use fields that that were uncommented. This is what we will do when we add containers.conf.

https://github.com/containers/storage/pull/502

rhatdan on 31 Dec 2019

It looks like there was an apparently sane fix given in https://github.com/containers/libpod/issues/3963 that was never presented as a PR, so the issue just aged out.

vwbusguy on 2 Jan 2020

👍1

I don't think anyone ever opened a PR to fix it.

rhatdan on 3 Jan 2020

It looks like the poster forked and made the change, but perhaps didn't submit it back as a PR? https://github.com/t-msn/storage/commit/41c2a90841cfc42f1373dac0de240773b405f536
(Ref this comment: https://github.com/containers/libpod/issues/3963#issuecomment-533630331 )

vwbusguy on 10 Jan 2020

@vwbusguy Hello,

It seemed that the original reporter in #3963 lost interest and I didn't make PR at that time. But now I have submitted: https://github.com/containers/storage/pull/508

Can you please check it?

t-msn on 12 Jan 2020

👍1

Given that https://github.com/containers/storage/pull/508 has been merged for a bit now, my understanding is all that's left to do is bump the dependency in libpod and make a bugfix release.
It's been a couple of weeks now, is there any general ETA?

CosmicToast on 27 Jan 2020

This was merged into podman on Jan 23rd. We are looking to cut a new release within the next week or two.

rhatdan on 28 Jan 2020

👍3

Was this page helpful?

0 / 5 - 0 ratings