podman build --pull (--pull behaviour is different from docker)

[asottile]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

the behaviour of podman build --pull differs from that of docker build --pull (the behaviour in docker is more like that of podman build --pull-always)

Steps to reproduce the issue:

1. have an outdated base image

2. podman build --pull ...

Describe the results you received:

It does not pull the base image before building

Describe the results you expected:

I expect it to pull the base image before building

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 5
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: babibox
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-48-generic
  linkmode: dynamic
  memFree: 3150012416
  memTotal: 8348299264
  ociRuntime:
    name: runc
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 1963335680
  swapTotal: 1964396544
  uptime: 17h 22m 6.04s (Approximately 0.71 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/asottile/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/asottile/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 45
  runRoot: /run/user/1000/containers
  volumePath: /home/asottile/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/unknown,now 2.1.1~2 amd64 [installed]
podman/unknown 2.1.1~2 arm64
podman/unknown 2.1.1~2 armhf
podman/unknown 2.1.1~2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

VirtualBox + AWS

[mheon]

@TomSweeneyRedHat PTAL

[rhatdan]

We probably should fix this, but it would involve differences between buildah bud and podman build
But that is probably alright since we already differ on --layers flag.

[TomSweeneyRedHat]

@asottile can you tell me which version of Docker you're comparing against please? One of the latest ones from Docker/Moby upstream or something else?

[asottile]

I actually don't have docker installed on my current machine (I've gone full podman + buildah), but I'm familiar with the behaviour -- the docs also mention this:

--pull | Always attempt to pull a newer version of the image

I can try and get a specific version if you'd like, though this option has worked this way as long as I can remember

[TomSweeneyRedHat]

@asottile thanks I just wanted to make sure the behaviour in the version of Docker you were using aligned with what Podman should be doing. Based on an article that I wrote several months ago (https://www.redhat.com/sysadmin/podman-image-pulling) podman build --pull which should default to podman build --pull=true should pull the image if it is not present locally, or if there's a new version in the registry.

To verify that the default is not off, could you try adding the true and seeing if that pulls the image? i.e. podman build --pull=true .... This was working when I wrote the article, but perhaps something got nicked accidentally a long the way. Anyway, I'll go setup a vm to play with. Thanks!

[asottile]

yeah it doesn't seem to do the "if there's a new version in the registry" part:

$ podman images | grep ubuntu.*bionic
docker.io/library/ubuntu                        bionic              6526a1858e5d  8 weeks ago   66.6 MB
$ podman tag ubuntu:bionic ubuntu:bionic-backup
$ podman build --pull=true -t test .
STEP 1: FROM ubuntu:bionic
STEP 2: RUN echo hi > f
STEP 3: COMMIT test
--> 7bf553e4b03
7bf553e4b03ecda292bbfc882f38ab0489c9cd9f96e999ccdc9c37ba1e2999d9
$ podman pull ubuntu:bionic
Trying to pull docker.io/library/ubuntu:bionic...
Getting image source signatures
Copying blob 171857c49d0f done  
Copying blob 419640447d26 done  
Copying blob 61e52f862619 done  
Copying config 56def654ec done  
Writing manifest to image destination
Storing signatures
56def654ec22f857f480cdcc640c474e2f84d4be2e549a9d16eaba3f397596e9
$ podman images | grep ubuntu.*bionic
docker.io/library/ubuntu                        bionic              56def654ec22  2 weeks ago     65.6 MB
localhost/ubuntu                                bionic-backup       6526a1858e5d  8 weeks ago     66.6 MB

[TomSweeneyRedHat]

@asottile Where is your registry located? Is it on quay.io by chance? I'm having some real funkiness going on where I keep only getting the odler variant of the image regardless of the pull type I use. More concerning, after I delete the tag on quay.io, then push the new version it to quay.io, remove all images on my host, and pull the image again, I still get the old version which should not be available....

I'm going to wrap up for the night, will look more over the weekend.

[asottile]

the image above (ubuntu:bionic) is from dockerhub (I believe docker.io is the canonical address?)

[asottile]

the code here _seems correct_:

https://github.com/containers/buildah/blob/73ae0013fc6a05b74078ef4d1108ed6221ff7fde/cmd/buildah/bud.go#L114-L119

my guess is the "if newer" code is broken 🤔

https://github.com/containers/buildah/blob/ab1adf15470799177d86293f7e0d7c7b7b91bded/new.go#L168-L190

[TomSweeneyRedHat]

OK, I must have been messing something up with my testing Thursday night, but I'm able to reliably replicate now. Interestingly, if I use "buildah bud --pull=true" it works as expected, in other words the Docker way. So I think that code in Buildah is probably OK, my suspicion is that Podman is passing the wrong PullPolicy, it may have been tweaked after the change to the new REST API.

[asottile]

ah yep, didn't find this the first search but here it is! https://github.com/containers/podman/blob/6ec96dc009d73cc2b0a3fa81149ca599b04252e4/cmd/podman/images/build.go#L243-L249

that's defaulting incorrectly

it also seems that neither podman nor buildah handle --pull=false if I'm reading the code correctly

[TomSweeneyRedHat]

@asottile that sure looks like the spot. The PullIfMissing should be used if --pull=false is passed. I'll have to play with the code a bit tomorrow, to see how to determine when the pull option is set and what it is set to. For now I'm going to enjoy the rest of the weekend. Thanks for the help/pointers.

[asottile]

ah actually it looks like podman's --pull=false kinda works:

$ podman build --pull=false -t test .
STEP 1: FROM ubuntu:lucid
Error: error creating build container: The following failures happened while trying to pull image specified by "ubuntu:lucid" based on search registries in /etc/containers/registries.conf:
* "localhost/ubuntu:lucid": no such image "containers-storage:[overlay@/home/asottile/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]localhost/ubuntu:lucid"
* "docker.io/library/ubuntu:lucid": no such image "containers-storage:[overlay@/home/asottile/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/ubuntu:lucid"
* "quay.io/ubuntu:lucid": no such image "containers-storage:[overlay@/home/asottile/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]quay.io/ubuntu:lucid"

but buildah's is incorrect:

$ buildah bud --pull=false -t test .
STEP 1: FROM ubuntu:lucid
Getting image source signatures
Copying blob a3ed95caeb02 done  
Copying blob a3ed95caeb02 done  
Copying blob 86b54f4b6a4e done  
^C

[TomSweeneyRedHat]

@asottile #8085 in flight, thanks for all the help with the diagnosis!

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[TomSweeneyRedHat]

This has been addressed in #8505 and a number of follow up PRs, closing