Aks-engine: Slow disk attach / detach in Kubernetes on Azure

[jsafrane]

Describe the bug
I'm running OpenShift CI on Azure. When it comes to storage, OpenShift = vanilla Kubernetes, there is no code change.

On Azure, we noticed that Azure performance degrades gradually with number of tests we run parallel and the tests start to time out due to slow attach / detach. It is not uncommon for volume attach / detach operation to take more than 3 minutes!

Stats taken between 01/23/2020 - 01/31/2020:
Operations taking longer than 60 seconds: 254 (5.54%)
Operations taking longer than 120 seconds: 36 (0.79%)

Steps To Reproduce
Run many* times in parallel in a single Azure account:

1. Install Kubernetes on Azure
2. Run Kubernetes e2e storage tests (~ --ginkgo.Focus=azure)
3. Destroy the cluster
4. Go to 1.

*) it's actually hard to tell what is the right number, as the tests are managed by prow. I'd start with 20 three-node clusters in the same account.

Kubernetes version
1.16.0

Additional context
I've contacted Azure support and complained about slow volume attach there, they redirected me here. The were not able to tell me what is volume attach/detach SLA on Azure!?

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[andyzhangx]

hello @jsafrane this is a known issue for azure managed disk, Azure team are working on improving the disk attach/detach perf, timeline is in the middle of this year, which could make disk attach/detach costs < 20s

[jsafrane]

@andyzhangx, thanks for your response. It there an issue / ticket that I could track?

[andyzhangx]

@andyzhangx, thanks for your response. It there an issue / ticket that I could track?

this is internal, we have not announced that since the work is in-progress.
We have another project to fix this issue: using unmanaged vhd disk on azure file

[jackfrancis]

Closing as this is not an AKS Engine issue

[jsafrane]

Azure team are working on improving the disk attach/detach perf, timeline is in the middle of this year, which could make disk attach/detach costs < 20s

It's 3/4 of this year, was there any progress? Unfortunately, we must support both in-tree and CSI driver until the migration is finished.

[andyzhangx]

Azure team are working on improving the disk attach/detach perf, timeline is in the middle of this year, which could make disk attach/detach costs < 20s

It's 3/4 of this year, was there any progress? Unfortunately, we must support both in-tree and CSI driver until the migration is finished.

@jsafrane there is fast attach/detach feature deployment in Azure canary regions, single disk attach/detach costs ~10s as I tested. May still need some time to validate an then roll out to other public regions

[gsxmax]

@andyzhangx , hi Andy, could you please let us know the current status of the fast attach/detach feature?

[andyzhangx]

@andyzhangx , hi Andy, could you please let us know the current status of the fast attach/detach feature?

@gsxmax fast attach/detach of single disk feature would be public preview in next few months, there is a new feature(batch operation for disk attach/detach) which is already available in Azure Disk CSI driver v0.10.0 release

• batch operation for disk attach/detach
  This feature would improve disk attach/detach operation speed quite a lot, e.g. if we attach 20 disks to one vmss instance at one time, with this PR, only two disk attach operations would be triggered, that's 10x faster than original.