Kind: Accept .gz or stream in load image-archive

Created on 30 May 2020 · 16Comments · Source: kubernetes-sigs/kind

What would you like to be added:
Accepting of either .tar.gz files, or streams.

Why is this needed:
In my case this is a nice to have for better parity with docker load $archive, which might have saved me some time.

Example of failure using tar.gz directly:

$ kind load image-archive archive.tar.gz
ERROR: failed to load image: command "docker exec --privileged -i kind-worker3 ctr --namespace=k8s.io images import -" failed with error: exit status 1
Command Output: ctr: archive/tar: invalid tar header
read unix @->/run/docker.sock: read: connection reset by peer

Example of failure reading from stream:

$ kind load image-archive <(gzip --decompress --stdout archive.tar.gz)
ERROR: failed to load image: command "docker exec --privileged -i kind-worker3 ctr --namespace=k8s.io images import -" failed with error: exit status 1
Command Output: ctr: archive/tar: invalid tar header
read unix @->/run/docker.sock: read: connection reset by peer

Example workaround:

$ gzip --decompress archive.tar.gz
$ kind load image-archive archive.tar

kinfeature lifecyclrotten triagnot-reproducible

Source

Code0x58

All 16 comments

we can support reading compressed archives pretty easily but it will confer only performance penalties if we implement it on the kind side (decompress on the host before loading).

ctr in containerd could be updated to support this but I'm not sure if they want it. It's worth noting that this is what we're calling, NOT docker load, because the nodes do not run docker.

The stream not working seems like a bug, I'll look into that.

BenTheElder on 30 May 2020

👍1

the following worked fine for me:

docker save alpine:latest | gzip >$HOME/junk/alpine.tar.gz
kind load image-archive <(gzip --decompress --stdout $HOME/junk/alpine.tar.gz)

is it possible the image actually wasn't a valid tarball ...?

BenTheElder on 31 May 2020

Thanks for digging into this!

I rechecked just now to make sure I didn't mix things up, but got the same result:

$ kind load image-archive <(cat archive.tar)
ERROR: failed to load image: command "docker exec --privileged -i kind-control-plane ctr --namespace=k8s.io images import -" failed with error: exit status 1
Command Output: ctr: archive/tar: invalid tar header
read unix @->/run/docker.sock: read: connection reset by peer
$ kind load image-archive archive.tar

The file I am using has a tar magic:

$ hexdump -Cn8 -s257 archive.tar 
00000101  75 73 74 61 72 00 30 30                           |ustar.00|
00000109

It was also made with docker save like your test :s

An environment info dump from my machine (Ubuntu 20.04):

$ kind version
kind v0.8.1 go1.14.2 linux/amd64
$ uname -a
Linux ubuntu-pc 5.4.0-31-generic #35-Ubuntu SMP Thu May 7 20:20:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ $0 --version | head -n1
GNU bash, version 5.0.16(1)-release (x86_64-pc-linux-gnu)
$ docker version
Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 23:42:35 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 22:48:33 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2
  GitCommit:        
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:        
 docker-init:
  Version:          0.18.0
Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 23:42:35 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 22:48:33 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2
  GitCommit:        
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:

Code0x58 on 31 May 2020

👀1

hmm, but now you're failing to load the uncompressed tar?
does it load when you don't use redirection? (kind load image-archive archive.tar)
what shell are you using?

BenTheElder on 31 May 2020

It was failing with the uncompressed tar before when using <(...), but from gzip's stdout instead (I cut out the middleman as I only had the .tar lying around after doing gzip --decompress archive.tar.gz.

It does load when not using redirection. The shell I'm using (it's also in the above post but I edited it in shortly before you posted - you're not going mad):

$ $0 --version | head -n1
GNU bash, version 5.0.16(1)-release (x86_64-pc-linux-gnu)

shell+gzip passed a sanity check:

$ gzip --decompress --keep archive.tar.gz
$ sha256sum archive.tar <(gzip --decompress --stdout archive.tar.gz)
6fc970e8558d23d3756251d6a8af5c7c51bb149247d137bb8698798ab209e5e3  archive.tar
6fc970e8558d23d3756251d6a8af5c7c51bb149247d137bb8698798ab209e5e3  /dev/fd/63

Code0x58 on 31 May 2020

👍1

tested on GNU bash, version 5.0.16(1)-release (x86_64-pc-linux-gnu) against kind v0.8.1 again with the commands I used above (previously on mac) ... hmm

BenTheElder on 31 May 2020

do you also see this using alpine:latest? or is it some particular image? my docker is 19.03.5 but otherwise a very similar environment, I'm not yet able to reproduce, process subsitution is working just fine.

BenTheElder on 31 May 2020

same when using alpine the latest alpine image of this moment. Could this be a kindest/node image version difference from caching?

$ docker image list
REPOSITORY                       TAG                   IMAGE ID            CREATED             SIZE
...
kindest/node                     <none>                de6eb7df13da        4 weeks ago         1.25GB

Code0x58 on 31 May 2020

the default image is pinned down to the image digest, and for the images in the release notes we request including the digest as well, so if we both kind create cluster from the same kind commit (no options) then we get the _same_ image.

BenTheElder on 31 May 2020

That makes sense. I suppose a remaining possibility is a in the kernel (5.4.0-31 for me)

Code0x58 on 31 May 2020

I wouldn't expect that to matter, but i'm at something of a loss for how to reproduce.
IIRC istio is actually using process substitution to load images actively in their CI setup.

I think maybe it would be best for ctr in containerd to support .tar.gz natively anyhow, which wouldn't require any changes to kind other than picking up the new containerd version. This helps the most people and is the cleanest separation of concerns I think.

BenTheElder on 31 May 2020

We're pretty unlikely to have time to replicate down to a specific kernel for this bug and I'm unable to reproduce.

Can you try to debug this more? Does it fail with process substitution of cat a normal archive?

BenTheElder on 3 Jun 2020

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 1 Sep 2020

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot on 1 Oct 2020

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

fejta-bot on 31 Oct 2020

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.