Aks-engine: VMSS disk attach/detach issues w/ v1.13.12, v1.14.8, v1.15.5, v1.16.2

[jackfrancis]

As described in the issue below, a regression to VMSS disk attach/detach was introduced in Azure cloudprovider starting with the following versions:

• v1.13.12
• v1.14.8
• v1.15.5
• v1.16.2

Upstream issue:

https://github.com/kubernetes/kubernetes/pull/85158

If you are running stateful workloads backed by VMSS node pools experiencing the above symptoms, we recommend you downgrade any clusters on the above versions to the patch version prior. E.g. for a cluster running v1.16.2 built w/ the service principal auth model:

$ bin/aks-engine upgrade --subscription-id <sub id> --api-model <path to apimodel>/apimodel.json --location <location> --resource-group <resource group> --upgrade-version 1.16.1 --force --auth-method client_secret --client-id <appId of service principal> --client-secret <password of service principal>

For new clusters, please use the following versions that aren't subject to this regression:

• "orchestratorVersion": "1.13.11"
• "orchestratorVersion": "1.14.7"
• "orchestratorVersion": "1.15.4"
• "orchestratorVersion": "1.16.1"

[jackfrancis]

We expect the kubernetes/kubernetes#85158 (merged) to land in v1.13.14, v1.14.10, v1.15.7, v1.16.4 (ETA early December)

[cpuguy83]

If you have already been hit by this issue, you'll need to manually detach disks from the vmss nodes.
You can use https://gist.github.com/cpuguy83/75f2f1c6556aa99118eef2830952c844 to help with that.

It is difficult to (safely) automate detection of which disks disks need to be detached, however a good indicator may be if you have PV's you are trying to delete and they are stuck in "Terminating" state.

The other option is to compare the list of running pods on a node with the disks attached to the vmss instance (using azure cli).

While performing these actions it is bessed to cordon the node so that k8s will not schedule new workloads on it while you are working on it.

[palmerabollo]

We still see some issues with disk attach/detach operations with Kubernetes 1.14.7 and the latest aks-engine v0.43.2

[jackfrancis]

@palmerabollo This issue isn't representative of every possible disk attach/detach edge case (unfortunately). However, our internal testing strongly supports the conclusion that the versions listed in this issue are more likely to encounter disk attach/detach errors needing manual intervention.

[jackfrancis]

addressed in 1.15 and 1.16 w/ recent releases.

1.13 and 1.14 are still affected, we expect those versions will not get the needed fixes