Cluster-api: MachineSet is looking at the wrong annotation for Machine deletion

[detiber]

What steps did you take and what happened:
Apply the documented annotation to a Machine that belongs to a MachineSet to select it for deletion (cluster.x-k8s.io/delete-machine) and then scale down the MachineSet. The MachineSet will not honor the documented annotation.

What did you expect to happen:
The MachineSet honor the documented annotation

Anything else you would like to add:
The MachineSet code is looking for cluster.k8s.io/delete-machine for the annotation instead of the documented cluster.x-k8s.io/delete-machine

It looks like at a minimum, the cluster-autoscaler is relying on the current behavior as well, so we might need to fix this with backwards compatibility in mind.

/kind bug

[ncdc]

Let's add code in 0.3.x to support both versions, and in 0.4.x only respect the x-k8s format?

[vincepri]

/milestone v0.3.4
/priority important-soon

@wfernandes did you want to try to tackle this issue?

[wfernandes]

/assign

[wfernandes]

/lifecycle active

[wfernandes]

So posing one of the hardest problems in CS.

If the existing annotation cluster.k8s.io/delete-machine is called DeleteNodeAnnotation what do we call the other one cluster.x-k8s.io/delete-machine?

Suggestions:

• MachineDeleteAnnotation
  derived from the autoscaler code
• DeleteMachineAnnotation
  Don't like the ones below anymore.
  ~- DeleteNodexAnnotation~
  ~- DeleteNodeAnnotationv2~
  ~- This might not be API compatible, but change the value of DeleteNodeAnnotation=cluster.x-k8s.io/delete-machine and put the older one in a variable called DeprecatedDeleteNodeAnnotation=cluster.k8s.io/delete-machine or something we don't care since we are planning on removing it in v0.4.x.~