Kind: [podman] kind 0.8.1 fails to download node image because podman is incompatible with image:tag@sha256

[axelsimon]

What happened:
Tried to run kind create cluster. Failed with:

ERROR: failed to create cluster: failed to pull image "kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f": command "docker pull kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f" failed 
with error: exit status 125 

What you expected to happen:
Create a cluster :)

How to reproduce it (as minimally and precisely as possible):
Run GO111MODULE="on" go get sigs.k8s.io/[email protected] && kind create cluster

Anything else we need to know?:
This is using podman and not docker, not that it should make a difference.

The SHA256 hash that is pinned for kindest/node-v.1.18.2 in the pull command is "7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f" but on docker hub, the hash is "1ab458c61fbd408081c9d321ceec1a50adffc2d060e1013d823af3c61fe90ff2".
This makes the pull command fail. Manually pulling the image and removing the hash downloads the image fine.

Environment:

• kind version: 0.8.1
• Kubernetes version: (use kubectl version): none (yet?)
• Docker version: (use docker info): podman version 1.9.3
• OS (e.g. from /etc/os-release): Fedora 32 (Workstation Edition)

[aojea]

/assign

[aojea]

@axelsimon this is curious, I'm able to reproduce the issue but seems that podman does not like that image

docker pull kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f: Pulling from kindest/node
Digest: sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
Status: Image is up to date for kindest/node@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
docker.io/kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f

 podman pull kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
Error: invalid image reference "kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f"

[BenTheElder]

It's not the wrong hash.
The point of the hash is to pull the correct image with the tag being informational only.
This is a podman bug.

[BenTheElder]

The current tag is a different version of that image, which is exactly why we use the hash.

[BenTheElder]

The podman driver is supposed to mitigate this by dropping the ~hash~ tag.

Is podman in your path as a fake docker?

[BenTheElder]

Setting KIND_EXPERIMENTAL_PROVIDER=podman will force the podman specific codepath, which you may need because podman fails to be docker compatible in other ways too.

In HEAD we have some trickery to detect when docker in PATH is not actually docker

[axelsimon]

The current tag is a different version of that image, which is exactly why we use the hash.

Thanks for clarifying this. That makes sense.

Is podman in your path as a fake docker?

which docker points to /usr/bin/docker, which is just a wrapper around podman. So yes, running docker will run podman.

I found the KIND_EXPERIMENTAL_PROVIDER=podman workaround later, but after i'd already downloaded the image manually and so thought that was why it was then clearing that step.

I'm running into another issue now, but it seems unrelated.

Thanks for jumping on this so fast, it's nice to see :)

[BenTheElder]

[being bad at being on vacation => delayed responses]

gotcha yeah, in HEAD we have detection of "is this actually docker" => if not then employ podman specific code to deal with things like this.

we should also file a bug upstream about this particular problem, it shouldn't be terribly difficult, the expected behavior is that the sha256 is accepted and the :tag part is ignored / just informational for the humans.

closing as a known / already fixed issue.

[tao12345666333]

Upstream issue https://github.com/containers/libpod/issues/6721