K3os: K3OS upgrade staging

[joharper54SEG]

Is your feature request related to a problem? Please describe.
We are planning on running single k3os nodes is some very remote locations that have questionable network links. When using the upgrade controller, the first step it performs is to cordon the node which takes all the pods down. It then starts pulling all its upgrades which can take several hours to complete in some locations. During this time, all services provided by the pods in these locations are down.

Additionally, I don't see any options to schedule when upgrade controller executes, so to plan for this downtime is problematic.

Describe the solution you'd like
It would be nice to have a option to stage upgrades so when then upgrade controller executes it does its downloads first before the cordon.

Describe alternatives you've considered
An alternative path, for now, would be to use the old upgrade scripts provided pre 0.9.0. These seem to be much less disruptive for the pods.

[dweomer]

With https://github.com/rancher/system-upgrade-controller/pull/61 this can be made to work with a prepare container:

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3os-latest
  namespace: k3os-system
spec:
  concurrency: 1
  channel: https://github.com/rancher/k3os/releases/latest
  nodeSelector:
    matchExpressions:
      - {key: plan.upgrade.cattle.io/k3os-latest, operator: Exists}
      - {key: k3os.io/mode, operator: Exists}
      - {key: k3os.io/mode, operator: NotIn, values: ["live"]}
  serviceAccountName: k3os-upgrade
  prepare:
    image: rancher/k3os
    command: [k3os, --version]
    args: []
  drain:
    force: true
  upgrade:
    image: rancher/k3os
    command: [k3os, --debug]
    args:
      - upgrade
      - --kernel
      - --rootfs
      - --remount
      - --sync
      - --reboot
      - --lock-file=/host/run/k3os/upgrade.lock
      - --source=/k3os/system
      - --destination=/host/k3os/system

This is the default plan with:

  prepare:
    image: rancher/k3os
    command: [k3os, --version]
    args: []

The prepare init container is called before the cordon/drain init container so this would satisfy your use case I think. I am whipping up a new release of k3OS that incorporates this.

[dweomer]

Scheduling is still an open issue, as per https://github.com/rancher/system-upgrade-controller/issues/63

[joharper54SEG]

Circling back to test this now that a new release is out. The prepare container appears to be working as the non-system pods are staying running. I did run into 1 issue with the prepare container as it's defined in the plan. I am specifying the version in the plan as I wanted some control over versioning instead of rolling to latest automatically. If version is specified the prepare container fails to download as it is not being tagged properly.

docker.io/rancher/k3os:latest: not found

If I tag the prepare image, everything works.

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: k3os-latest
  namespace: k3os-system
spec:
  concurrency: 1
  version: v0.10.2
  channel: https://github.com/rancher/k3os/releases/latest
  nodeSelector:
    matchExpressions:
      - {key: plan.upgrade.cattle.io/k3os-latest, operator: Exists}
      - {key: k3os.io/mode, operator: Exists}
      - {key: k3os.io/mode, operator: NotIn, values: ["live"]}
  serviceAccountName: k3os-upgrade
  prepare:
    image: rancher/k3os:v0.10.2
    command: [k3os, --version]
    args: []
  drain:
    force: true
  upgrade:
    image: rancher/k3os
    command: [k3os, --debug]
    args:
      - upgrade
      - --kernel
      - --rootfs
      - --remount
      - --sync
      - --reboot
      - --lock-file=/host/run/k3os/upgrade.lock
      - --source=/k3os/system
      - --destination=/host/k3os/system

[dweomer]

@joharper54SEG what version of k3OS were you running prior to the upgrade to v0.10.2? I ask because the tag-less .spec.prepare.image was not supported until v0.10.1 which included the bump to the SUC to v0.5.0

[joharper54SEG]

I was on v0.10.0, so that would explain it.