kind load docker-image should optimize loading new tags for already loaded images

[adampl]

What happened:
Kind doesn't recognize an existing image with the same digest but a different tag as already present.

What you expected to happen:
The second load should just create a new tag, instead of loading the whole image again.

How to reproduce it:

# docker tag my-image repo/name:tag-one
# docker tag my-image repo/name:tag-two

# kind load docker-image repo/name:tag-one
Image: "repo/name:tag-one" with ID "sha256:812c9b2b4173e8414c51ac368333de5a22b8bccce106e486b1f564bdd04eca33" not present on node "kind-worker"

# kind load docker-image repo/name:tag-two
Image: "repo/name:tag-two" with ID "sha256:812c9b2b4173e8414c51ac368333de5a22b8bccce106e486b1f564bdd04eca33" not present on node "kind-worker"

Environment:

# docker --version 
Docker version 19.03.5, build 633a0ea838
# kind version
kind v0.6.1 go1.13.4 linux/amd64

Related issues and PRs:

380

382

680

776

[BenTheElder]

The second load should just create a new tag, instead of loading the whole image again.

containerd and CRI have no such mechanism as far as I know. we can possibly emulate it, but it will be ... interesting.

you should take a look at https://kind.sigs.k8s.io/docs/user/local-registry/ as one alternative

[BenTheElder]

so also, while the docker save is not optimized out, the layer loading should in fact skip existing layers.

[adampl]

Maybe the image should be loaded as a digest (without any tag so comparison would be simple), and tags could be transmitted separately and just created if missing?

[BenTheElder]

it looks like actually the containerd API (not CRI) does have the right methods to make it possible to create an image referencing existing metadata content which is not surprising given the architecture, but the CLI does not have this so we'd need an API client somewhere, besides a comparison of the image layers instead of the manifest / image ID

[BenTheElder]

Maybe the image should be loaded as a digest (without any tag so comparison would be simple), and tags could be transmitted separately and just created if missing?

So right now we confirm that the image ~manifests are the same and load image tarballs.

Image archive tarballs have a manifest in them, the hash of which defines the image ID, and the contents of which defines the image. there are then sub-tarballs for each of the layers.

Right now we check if the ID is present on the node(s) and if not we load the image.

We could check if the _layers_ exist, and then try to populate the equivalent of the manifest via the API, but that's not really a normal flow and we'd have to be careful to produce the same manifest else the image will not be the same as if we'd just imported an image archive.

[adampl]

Hmm, I thought it's just a bug, because it looks like an obvious functionality to me... but I'm not really an expert on the Docker internals. Which CLI did you mean in the previous comment?

[BenTheElder]

I think better registry support might be the better thing to focus on right now, it's reasonably efficient about this out of the box (only pushing / pulling missing layers individually) and fits in better with workflows that are not kind specific.

For larger images pushing over loopback will probably be faster than streaming over docker exec.

It also has the benefit of persisting across cluster create / delete and more efficiently supporting loading images into just the nodes actually using them without thinking about it.

kind load ... is simple and reliable but registries will probably give better UX and more portable workflows.

I'll be working more on registry UX after sorting out some possibly breaking network changes related to host reboot support that would break this.

[BenTheElder]

Hmm, I thought it's just a bug, because it looks like an obvious functionality to me... but I'm not really an expert on the Docker internals.

Yeah, it's definitely obvious functionality but the way image import works is that an entire tarball including all the image details and layers is loaded.

Which CLI did you mean in the previous comment?

ctr (or crictl) on the nodes, to manage the images etc.

[adampl]

Yes, I also noticed that kind load is terribly slow, especially with images above 1 GB.
Maybe a local registry is worth the hassle...

[BenTheElder]

we aim to make it less of a hassle in the future. it's _possible_ today but definitely not ideal.

I recommend using a variant of the example script to automate it for now.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[BenTheElder]

We looked into this, it doesn't look straightforward to do portably, docker doesn't expose a way to get image layers natively, you can sort of read them from local storage or something but that gets brittle quickly.

I think for now local registries are the way forward to more optimal image loading.