K3os: system-upgrade-controller is missing in k3s-airgap-images

[pagong]

Is your feature request related to a problem? Please describe.
I'm trying to do an airgap installation of k3os by copying the file k3s-airgap-images-amd64.tar to directory /var/lib/rancher/k3s/agent/images. However, the pod for the system-upgrade-controller does not come up.

Describe the solution you'd like
Please include image rancher/system-upgrade-controller:v0.4.0 into the airgap tar ball for v1.17.4+k3s1 (and future releases).

Describe alternatives you've considered
None.

[dweomer]

The air gap story could be more coherent, yes, but since v0.3.0 the SUC has been pushing tarballs as checksummed release artifacts (the .tar assets). See https://github.com/rancher/system-upgrade-controller/releases

[pagong]

OK, I'll try that route: I'm going to put https://github.com/rancher/system-upgrade-controller/releases/download/v0.4.0/system-upgrade-controller-amd64.tar also into /var/lib/rancher/k3s/agent/images and watch what happens.

[pagong]

I've finally managed to get the system-upgrade-controller running in an air-gapped environment:

1. Copy the TAR file system-upgrade-controller-amd64.tar into directory /var/lib/rancher/k3s/agent/images.
2. Execute sudo crictl images:

IMAGE                                         TAG                 IMAGE ID            SIZE
docker.io/rancher/coredns-coredns             1.6.3               c4d3d16fe508b       44.4MB
docker.io/rancher/klipper-helm                v0.2.3              274808e7f6b83       138MB
docker.io/rancher/klipper-lb                  v0.1.2              897ce3c5fc8ff       6.46MB
docker.io/rancher/library-traefik             1.7.19              aa764f7db3051       86.6MB
docker.io/rancher/local-path-provisioner      v0.0.11             9d12f9848b99f       36.5MB
docker.io/rancher/metrics-server              v0.3.6              9dd718864ce61       41.2MB
docker.io/rancher/pause                       3.1                 da86e6ba6ca19       746kB
docker.io/rancher/system-upgrade-controller   v0.4.0-amd64        7b04ae4440239       33.7MB
docker.io/rancher/system-upgrade-controller   latest              7b04ae4440239       33.7MB
````
3. This is not enough. The POD keeps looking for  "rancher/system-upgrade-controller:v0.4.0"
4. Retag the image without the `-amd64` suffix by using command:

sudo ctr image tag docker.io/rancher/system-upgrade-controller:v0.4.0-amd64 docker.io/rancher/system-upgrade-controller:v0.4.0

5. Watch the result with `sudo crictl images`:

IMAGE TAG IMAGE ID SIZE
docker.io/rancher/coredns-coredns 1.6.3 c4d3d16fe508b 44.4MB
docker.io/rancher/klipper-helm v0.2.3 274808e7f6b83 138MB
docker.io/rancher/klipper-lb v0.1.2 897ce3c5fc8ff 6.46MB
docker.io/rancher/library-traefik 1.7.19 aa764f7db3051 86.6MB
docker.io/rancher/local-path-provisioner v0.0.11 9d12f9848b99f 36.5MB
docker.io/rancher/metrics-server v0.3.6 9dd718864ce61 41.2MB
docker.io/rancher/pause 3.1 da86e6ba6ca19 746kB
docker.io/rancher/system-upgrade-controller latest 7b04ae4440239 33.7MB
docker.io/rancher/system-upgrade-controller v0.4.0 7b04ae4440239 33.7MB
docker.io/rancher/system-upgrade-controller v0.4.0-amd64 7b04ae4440239 33.7MB
```

1. Now k3s is happy and the POD gets started.

[pagong]

Okay, S-U-C is working now.
But that doesn't make sense in an air-gapped environment:
```
kubectl -n k3os-system logs pod/system-upgrade-controller-69c695c7fc-r77b4
W0505 17:32:16.344349 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
time="2020-05-05T17:32:16Z" level=info msg="Creating CRD plans.upgrade.cattle.io"
time="2020-05-05T17:32:16Z" level=info msg="Waiting for CRD plans.upgrade.cattle.io to become available"
time="2020-05-05T17:32:16Z" level=info msg="Done waiting for CRD plans.upgrade.cattle.io to become available"
time="2020-05-05T17:32:16Z" level=info msg="Starting /v1, Kind=Node controller"
time="2020-05-05T17:32:16Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2020-05-05T17:32:16Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2020-05-05T17:32:16Z" level=info msg="Starting upgrade.cattle.io/v1, Kind=Plan controller"
E0505 17:32:26.926414 1 controller.go:135] error syncing 'k3os-system/k3os-latest': handler system-upgrade-controller: Get https://github.com/rancher/k3os/releases/latest: dial tcp: lookup github.com on 10.107.17.10:53: no such host, requeuing
E0505 17:32:26.934943 1 controller.go:135] error syncing 'k3os-system/k3os-latest': handler system-upgrade-controller: Get https://github.com/rancher/k3os/releases/latest: dial tcp: lookup github.com on 10.107.17.10:53: no such host, requeuing
E0505 17:32:26.946619 1 controller.go:135] error syncing 'k3os-system/k3os-latest': handler system-upgrade-controller: Get https://github.com/rancher/k3os/releases/latest: dial tcp: lookup github.com on 10.107.17.10:53: no such host, requeuing
````

S-U-C keeps looking for github.com. 😢
Maybe a local repo or registry is needed?
Or disable S-U-C for air-gapped envs, by using --no-deploy system-upgrade-controller?

[dweomer]

The k3os-latest plan is meant to serve as an example. It is was I use to test upgrades and the SUC on k3OS. That said, if you are using k3os-latest plan you are welcome to edit it as you see fit. In this instance, to prevent it from attempting to talk to github.com you can remove the .spec.channel from the plan but then you will want to explicitly specify the .spec.version as a plan without either is in error.

[gbonnefille]

Please, can you clarify who is responsible of downloading the fresh version of K3OS? Is it S.U.C. or the k3os upgrade command?