Podman: Failed to prune image: image is in use by a container

[bmaupin]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I'm frequently unable to run podman prune -a because I get the following error:

$ podman image prune -a
Error: failed to prune image: Image used by 90b9fa5820c4f1a050ce1bd00f937a5cc63a5e3e4b08a52cf0333a8d9eb45f94: image is in use by a container

More info:

$ buildah containers --all
CONTAINER ID  BUILDER  IMAGE ID     IMAGE NAME                       CONTAINER NAME
a5b76fd2b773           815d76a89459 registry.example.org/containers/base-images/rhel7-mysql-80:latest mysql
348fd256a08b           cedbe259e0a0 docker.io/bmaupin/pspdev:gcc-4.6.4 thirsty_moore
815ebf0b8933           556e8499d9f9 localhost/impersonator:latest    impersonator
7c0af2f8e427     *     36bd7e8bcf55 registry.access.redhat.com/rhoar-nodejs/nodejs-8:latest nodejs-8-working-container
90b9fa5820c4     *     9b574591fcab                                  9b574591fcab120dfdcf52ecbb6354661605ea19beaadbc37579205073145673-working-container
cf2da324a492     *     36bd7e8bcf55 registry.access.redhat.com/rhoar-nodejs/nodejs-8:latest nodejs-8-working-container-1
e1e0571d438f     *     da54719fe50c                                  da54719fe50cc5fe1fedc0452b5125ec4996945e2af09e10804b75ebe478e080-working-container
bff42f40b7a8     *     0fcff1eab16c                                  0fcff1eab16c2eb86fa17044d1bbc567193e548de6460edb08eb40fde5e7e458-working-container
4dab95a379dd     *     76b9b33d5878                                  76b9b33d5878a48e4310a172685b15e74649f4fd06b40b5343395e150bcf96bb-working-container
c05010d15a76     *     36bd7e8bcf55 registry.access.redhat.com/rhoar-nodejs/nodejs-8:latest nodejs-8-working-container-2
91e1e8274022     *     da54719fe50c                                  da54719fe50cc5fe1fedc0452b5125ec4996945e2af09e10804b75ebe478e080-working-container-1
34f8f6a69f5e     *     0fcff1eab16c                                  0fcff1eab16c2eb86fa17044d1bbc567193e548de6460edb08eb40fde5e7e458-working-container-1
b3d0cf7b54d9     *     20b82b08b854                                  20b82b08b8543d5b02419498ddae1ac19b163f1db7260e9d58349138c6884041-working-container
48d592dde705     *     7030953edf66                                  7030953edf669a9fcdd40594d588ea7a6af389c65f6231ee1126066267605963-working-container
131af825d0cc           fa7398dcd3cb localhost/seed-loopback3:latest  seed-loopback3

$ podman image ls
ERRO[0000] error checking if image is a parent "7030953edf669a9fcdd40594d588ea7a6af389c65f6231ee1126066267605963": error reading image "f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21": image not known 
ERRO[0000] error checking if image is a parent "f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21": error reading image "f0c5b4e82e11c970a2f38198b2eaeacd6479b6bc12b90960887cf456ad83ae21": image not known 
REPOSITORY                                                       TAG         IMAGE ID       CREATED             SIZE
<none>                                                           <none>      e0052b5acc35   About an hour ago   568 MB
<none>                                                           <none>      4e13c4def0df   About an hour ago   568 MB
<none>                                                           <none>      7c5da149659b   About an hour ago   568 MB
localhost/seed-loopback3                                         latest      fa7398dcd3cb   19 hours ago        632 MB
<none>                                                           <none>      94ef4d389a1e   19 hours ago        699 MB
<none>                                                           <none>      f0c5b4e82e11   3 days ago          699 MB
<none>                                                           <none>      7030953edf66   3 days ago          699 MB
<none>                                                           <none>      493974907437   3 days ago          699 MB
localhost/impersonator                                           latest      556e8499d9f9   4 days ago          930 MB
<none>                                                           <none>      c0bcb0376d5a   4 days ago          692 MB
<none>                                                           <none>      b5769e0bd61a   4 days ago          692 MB
registry.example.org/containers/base-images/rhel7-node-oracle   8           fe5c0145a44a   5 days ago          793 MB
registry.example.org/containers/redhat/nodejs-8                 latest      36bd7e8bcf55   5 days ago          568 MB
registry.access.redhat.com/rhoar-nodejs/nodejs-8                 latest      36bd7e8bcf55   5 days ago          568 MB
registry.example.org/containers/base-images/rhel7-mysql-80      latest      815d76a89459   13 days ago         478 MB
docker.io/bmaupin/pspdev                                         gcc-4.6.4   cedbe259e0a0   2 months ago        911 MB

$ podman container ls -a
CONTAINER ID  IMAGE                                                               COMMAND               CREATED       STATUS                     PORTS                   NAMES
131af825d0cc  localhost/seed-loopback3:latest                                     node_modules/.bin...  19 hours ago  Exited (127) 19 hours ago                          seed-loopback3
815ebf0b8933  localhost/impersonator:latest                                       node_modules/.bin...  4 days ago    Exited (130) 4 days ago                            impersonator
348fd256a08b  docker.io/bmaupin/pspdev:gcc-4.6.4                                  ./libretro-buildb...  5 days ago    Created                                            thirsty_moore
a5b76fd2b773  registry.example.org/containers/base-images/rhel7-mysql-80:latest  sh -c run-mysqld ...  11 days ago   Exited (0) 5 days ago      0.0.0.0:3306->3306/tcp  mysql

Steps to reproduce the issue:

See Description

Describe the results you received:

See Description

Describe the results you expected:

I would've expected podman image prune -a to work the same way docker image prune -a does:

Remove all unused images, not just dangling ones

https://docs.docker.com/engine/reference/commandline/image_prune/#options

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 3998068736
  MemTotal: 16756355072
  OCIRuntime:
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree: 8161837056
  SwapTotal: 8189374464
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: host
  kernel: 5.0.0-27-generic
  os: linux
  rootless: true
  uptime: 88h 40m 44.79s (Approximately 3.67 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
store:
  ConfigFile: /home/user/.config/containers/storage.conf
  ContainerStore:
    number: 17
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/user/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 65
  RunRoot: /tmp/1000
  VolumePath: /home/user/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/bionic,now 1.5.1-1~ubuntu18.04~ppa1 amd64 [installed]

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical machine running Ubuntu 18.04.

Thanks!

[TomSweeneyRedHat]

Thanks for the issue @bmapuin. You've added a link to #3982, I just wanted to note that further background discussion can be found there.

[mheon]

3984 to fix

[bmaupin]

I don't understand how I'm the first one to encounter this issue. Am I the only podman user using both podman build (I haven't ever explicitly used buildah) and podman prune -a? So bizarre!

[mheon]

From issue reports I've seen, most people are using podman build which cleans up its containers automatically as builds complete... But you're right, it does seem unlikely that nobody is using prune on a system with Buildah or CRI-O installed.

[ecs-hk]

I don't understand how I'm the first one to encounter this issue. Am I the only podman user using both podman build (I haven't ever explicitly used buildah) and podman prune -a? So bizarre!

You're not alone.

[jilin]~> podman container list
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES

[jilin]~> podman image prune
Error: failed to prune image: Image used by 62d0dcaf932e512f8f57d728adb3cd7cf00027ec03deb9d254f53071bd2c1ec4: image is in use by a container

[jilin]~> podman container kill 62d0dcaf932e512f8f57d728adb3cd7cf00027ec03deb9d254f53071bd2c1ec4
Error: no container with name or ID 62d0dcaf932e512f8f57d728adb3cd7cf00027ec03deb9d254f53071bd2c1ec4 found: no such container

Pretty mysterious. (I never use buildah either.)

[qhaas]

Ran into this today, given that buildah/podman are typically used together (iirc, the latter calls the former's libraries when building images), figured that something was amiss with the former when I was seeing this behavior with the latter:

podman image prune -a
Error: failed to prune image: Image used by 693470ed1189ed805908dca2ea408fff862f14d0d5832ff71c41f1de5b005748: image is in use by a container

However, podman images -a and podman ps -a showed nothing. I then checked buildah with buildah containers -a and buildah images -a and found plenty there. The solution? I just cleared out everything buildah was aware of with buildah rm -a and buildah rmi -a, after that, podman system prune -a and podman image prune -a worked like a charm.