K3s: why k3s ctr images pull timeout but docker can pull success

[damozhiying]

Version:
v1.17.0-rc.4-k3s.1

Describe the bug

sudo docker pull docker.io/rancher/pause:3.1
3.1: Pulling from rancher/pause
Digest: sha256:d22591b61e9c2b52aecbf07106d5db313c4f178e404d660b32517b18fcbf0144
Status: Image is up to date for rancher/pause:3.1

[ecf@server16 system]$ sudo k3s ctr images pull docker.io/rancher/pause:3.1
docker.io/rancher/pause:3.1: resolving      |--------------------------------------| 
elapsed: 29.9s               total:   0.0 B (0.0 B/s)                                         
ERRO[2020-01-06T17:31:41.470872098+08:00] active check failed                           error="context canceled"
ctr: failed to resolve reference "docker.io/rancher/pause:3.1": failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: dial tcp 34.197.189.129:443: i/o timeout

[ryepup]

I'm seeing a similar problem on a raspberry pi.

$ k3s --version
k3s version v1.17.0+k3s.1 (0f644650)

I get TLS Handshare timeouts when trying to fetch images, which results in errors and ultimately the node being unable to pull any docker images.

I can recreate this directly:

$ sudo k3s ctr image pull docker.io/rancher/klipper-lb:v0.1.2
docker.io/rancher/klipper-lb:v0.1.2:                                              resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:2fb97818f5d64096d635bc72501a6cb2c8b88d5d16bc031cf71b5b6460925e4a:    waiting        |--------------------------------------| 
docker.io/rancher/klipper-lb:v0.1.2:                                              resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:2fb97818f5d64096d635bc72501a6cb2c8b88d5d16bc031cf71b5b6460925e4a:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:05a8fb40c1a0e1edde509c654c58a211e60b3162e41bf275a6de645af8c7debf: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:f3d053078311027dabc30837b8909f3bc388935120a0f2fb85c5b74b57ccb7ab:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:7d23a14d38d241a90765cc08848ef5901932d66e0f697adc94f0bb10ccfadbb7:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5b678b67777fc7983d3563839cc9d511de267ec6de1961f2b590d552d8bfa105:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:d9f0b2b885d968636a597331169fce72a69964c911558554f1b2a0d21959f34f:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:721c43048222f4ab09255f9ded01963f23714c3f4aac7d7ca073a8b6feccd78f:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 19.7s                                                                    total:  1.1 Ki (58.0 B/s)                                        
ctr: failed to copy: httpReaderSeeker: failed open: failed to do request: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/72/721c43048222f4ab09255f9ded01963f23714c3f4aac7d7ca073a8b6feccd78f/data?verify=1579550636-TcODE28BJgq%2FBmEvfHLhB3RWoAw%3D: net/http: TLS handshake timeout

In this case, my pi is an older Model B+; I think it's just not fast enough. I do not see this on a new pi model. I'm going to try upgrading my hardware, but it'd be great if there was a way to configure the TLS handshake timeout.

[anirtek]

I am also running through the same issue. My image is of size 3GB and when I am trying to perform the deployment through yaml file, it fails with ImgPullBackError.

I tried using sudo k3s ctr image pull quay.io/cloudian/hap-spark-tf:0.0.1 command. But I have to do this many times and everytime it pulls some more data and finally gives me error -
ctr: failed to copy: httpReaderSeeker: failed open: failed to do request: Get https://quay.io/v2/cloudian/hap-spark-tf/blobs/sha256:aadf4ce80f06eb3861fbe013843671eed35033d2c1de571cdb71241812fd4a5e: net/http: TLS handshake timeout. I have made sure that this deployment works on cluster created with kubeadm.

How do I resolve this?