Cluster-api: Document rollout support for K Worker Nodes

[Arvinderpal]

User Story

As a user/operator I would like to roll out all K worker nodes to new hardware for various reasons.
As a developer/user/operator I would like to have symmetry between control-plane and worker node roll outs for added simplicity.

Detailed Description

KCP.Spec.UpgradeAfter allows machines to be rolled out after a specific date and time even if nothing in the Spec has changed. This approach has some benefits - it can be used to move control-plane nodes to new hardware, perform cert rotation, allow changes in infra machine templates to be reflected in control plane nodes (w/o creating a brand new template), ...

[edited]
CAPI should support a similar approach for worker nodes.
CAPI already supports immediate rollout of worker Nodes -- see comment below. We should improve the docs to reflect this.

/kind feature

[Arvinderpal]

One approach would be to introduce the upgradeAfter field in MachineDeployment and use similar logic to KCP in deciding when an roll out should be triggered.

[detiber]

I don't necessarily see an issue with introducing an upgradeAfter field, however I'm wondering if it would make sense to have more generic support for something similar to kubectl rollout for MachineDeployments and possibly also KubeadmControlPlane?

[vincepri]

+1 Adding a similar functionality to MachineDeployment as well, I'd wait for v0.4.0 and rename these fields to RolloutAfter. @detiber's suggestion to support something similar to kubectl rollout sounds interesting as well

[Arvinderpal]

A rollout command for clusterctl would be great. Perhaps we can also add support for the various sub-commands like undo/status/history as with kubectl rollout.

We can wait for 0.4.0

[vincepri]

/milestone v0.4.0

[Arvinderpal]

After a looking into this a bit further, MachineDeployment rollout is already supported. In fact, it follows the same conventions as a regular Deployment rollout. -- a Deployment's rollout is triggered if and only if the Deployment's Pod template (that is, .spec.template) is changed, so a restartedAt annotation is added to deployments to trigger an immediate restart.

Taking the same approach, when an (arbitrary) annotation is added to the MD.spec.template, a rollout is triggered:

kubectl patch machinedeployment test-md-0 --type merge -p '{"spec":{"template":{"metadata":{"annotations":{"cluster.x-k8s.io/restartedAt": "2020-09-23T09:47:07-07:00"}}}}}'

Changes to the MD.spec.template result in a new MachineSet. We can see the revision number annotation in the MD updated to the latest MS. We can also see our restartedAt annotation in the new MS.spec.template as well.

@vincepri Should we document the above approach? The rollout process for KCP is a bit different, but if that's not documented, we should add that as well.

[fabriziopandini]

Am I wrong or the approach described above triggers an immediate upgrade, while in this issue we are seeking fo triggering a deferred upgrade?

[Arvinderpal]

Yes, it triggers an immediate rollout, which IMO is fine. This approach follows closely with the general Deployment/RelicaSet model.

KCP handles things a bit differently, it has a specific field for "forcing" a rollout to occur immediately or sometime in the future.

[fabriziopandini]

Thanks, if this is the case we already have this doc https://cluster-api.sigs.k8s.io/tasks/change-machine-template.html#changing-infrastructure-machine-templates; I'm +1 to improve it if necessary

[Arvinderpal]

Yes, IMO the doc should be improved. That specific page talks about changing machine templates. However, the use case here is to issue an immediate rollout, irrespective of if anything has changed in the MD.