Podman: Hitting "Error committing the finished image" "no space left on device" due to nonoptimized use of disk

[oblitum]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1. Have 50GB left on disk

2. Have a Dockerfile whose second step will take 7.5GB of data, like installing many programs

3. Have additional steps (around >5 more steps) that won't require space at all

Describe the results you received:

I failed to build an image many times that I had no issues building with docker. I noticed that at each try podman took too long to pick each cached step after the second, while docker did it instantly. I realized that for each time podman was expending time on a step picking cache, it was producing a copy of around 7.5GB on ~/.local/share/containers/storage/vfs/dir, for all the steps following the second, even though those steps didn't produce data, only the second did. It looks like podman is simply copying the second step at all following steps. There's no space left on disk to finish this operation, so it reaches "no space left on device".

Describe the results you expected:

I was hopping for podman to work like docker and not require 10x in disk size over the size of final image to produce it, nor was I expecting that trivial steps would be slow to finish because of the previous.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 1.5.1

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.8
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.0, commit: e217fdff82e0b1a6184a28c43043a4065083407f'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 6003105792
  MemTotal: 16747802624
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: leibniz
  kernel: 5.2.9-arch1-1-ARCH
  os: linux
  rootless: true
  uptime: 42m 51.56s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/francisco/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/francisco/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 9
  RunRoot: /run/user/1000
  VolumePath: /home/francisco/.local/share/containers/storage/volumes

[mheon]

This is a known limitation of the VFS storage driver. We recommend using
fuse-overlayfs instead, as it is capable of deduplicating storage

On Sun, Aug 18, 2019, 11:11 Francisco Lopes [e-mail]
wrote:

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

1.

Have 50GB left on disk
2.

Have a Dockerfile whose second step will take 7.5GB of data, like
installing many programs
3.

Have additional steps (around >5 more steps) that won't require space
at all

Describe the results you received:

I failed to build an image many times that I had no issues building with
docker. I noticed that at each try podman took too long to pick each cached
step after the second, while docker did it instantly. I realized that for
each time podman was expending ona step to use cache, it was producing a
copy of around 7.5GB on ~/.local/share/containers/storage/vfs/dir, for
all the steps following the second, even though those steps didn't produce
data, only the second did. It looks like podman is simply copying the
second step at all following steps. There's no space left on disk to finish
this operation, so it reaches "no space left on device".

Describe the results you expected:

I was hopping for podman to work like docker and not require 10x in disk
size over the size of final image to produce, nor was I expecting that
trivial steps would be slow to finish because of the previous.

Additional information you deem important (e.g. issue happens only
occasionally):

Output of podman version:

podman version 1.5.1

Output of podman info --debug:

debug:
compiler: gc
git commit: ""
go version: go1.12.8
podman version: 1.5.1
host:
BuildahVersion: 1.10.1
Conmon:
package: Unknown
path: /usr/bin/conmon
version: 'conmon version 2.0.0, commit: e217fdff82e0b1a6184a28c43043a4065083407f'
Distribution:
distribution: arch
version: unknown
MemFree: 6003105792
MemTotal: 16747802624
OCIRuntime:
package: Unknown
path: /usr/bin/runc
version: |-
runc version 1.0.0-rc8
commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
spec: 1.0.1-dev
SwapFree: 0
SwapTotal: 0
arch: amd64
cpus: 8
eventlogger: journald
hostname: leibniz
kernel: 5.2.9-arch1-1-ARCH
os: linux
rootless: true
uptime: 42m 51.56s
registries:
blocked: null
insecure: null
search:

• docker.io
• registry.fedoraproject.org
• quay.io
• registry.access.redhat.com
• registry.centos.org
  store:
  ConfigFile: /home/francisco/.config/containers/storage.conf
  ContainerStore:
  number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/francisco/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
  number: 9
  RunRoot: /run/user/1000
  VolumePath: /home/francisco/.local/share/containers/storage/volumes

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/3846?email_source=notifications&email_token=AB3AOCGEGKVPQLRDBYNX2OTQFFRBHA5CNFSM4IMTIRO2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HF2Z2WA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB3AOCBA6QNBONIMAVWFU3DQFFRBHANCNFSM4IMTIROQ
.

[oblitum]

@mheon I'm newbie to podman, where should I configure that?

[oblitum]

Is it enough if I just install fuse-overlayfs?

[baude]

Does your ~/.config/containers/storage.conf look like this?

[storage]
  driver = "overlay"
  runroot = "/run/user/1000"
  graphroot = "/home/bbaude/.local/share/containers/storage"
  [storage.options]
    mount_program = "/usr/bin/fuse-overlayfs"

Btw, if you make an adjustment, you may need to remove your current storage ~/.local/share/containers/storage to make the change

[oblitum]

@baude thanks for the tip. After installing fuse-overlayfs, it was still using vfs as driver. I've started anew by removing ~/.config/containers/ and ~/.local/share/containers/. The storage.conf is now using "overlay".