Aks-engine: I'm not able to create a cluster using Ultra SSD

[pierluigilenoci]

Describe the bug

I tried to create a cluster using Ultra SSD and I failed miserably.

Steps To Reproduce

1. Tried to spawn a cluster with Standard LB, masters with AvailabilitySet, minion with VirtualMachineScaleSets, with the right availabilityZones in a custom VNET. UltraSSD works only with VirtualMachineScaleSets 😞
2. Tried to change masters to VirtualMachineScaleSets but I received this message back Error: loading API model in generateCmd: error parsing the api model: standard loadBalancerSku should exclude master nodes. Please set KubernetesConfig "ExcludeMasterFromStandardLB" to "true"
3. Tried to add "ExcludeMasterFromStandardLB" to "true". I get Error: validating API model after populating values: when master profile is using VirtualMachineScaleSets and is custom vnet, set "vnetsubnetid" and "agentVnetSubnetID" for master profile.
4. Added "agentVnetSubnetID". I get WARN[0000] Clusters with VMSS masters are not yet upgradable! You will not be able to upgrade your cluster until a future version of aks-engine!

AKS Engine version

v0.43.0

Kubernetes version

1.15.5

[pierluigilenoci]

@mboersma could you please brief me give a short feedback?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pierluigilenoci]

@mboersma could you please brief me give a short feedback?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pierluigilenoci]

@mboersma could you please brief me give a short feedback?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pierluigilenoci]

@mboersma could you please brief me give a short feedback?

[pierluigilenoci]

Maybe one of you?
@brendandburns
@CecileRobertMichon
@devigned
@jackfrancis
@ksubrmnn
@marosset
@PatrickLang
@ritazh

[jackfrancis]

Hi @pierluigilenoci. The validation warning message in your step 4 is correct. You may use a VMSS-backed control plane with aks-engine, but aks-engine upgrade will not work against that cluster. So long as you have an alternate way of maintaining the cluster lifecycle, or plan to use these clusters as ephemeral cattle (and not pets), you should be O.K.

[pierluigilenoci]

@jackfrancis so you confirm that it is still not possible. Is an aks-engine release for Ultra SSD disks expected in the near future? Or at least is it a planned feature? Or should I forget it?

Thanks a lot for your time and your answers.

[jackfrancis]

I can confirm that we have no plans to make VMSS masters work w/ aks-engine upgrade.

Where does it say that ultrassd only works with VMSS? I can't find that documented anywhere.

[pierluigilenoci]

Here it says: "Doesn't currently support disk snapshots, VM images, availability sets, Azure Dedicated Hosts, or Azure disk encryption".
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/disks-enable-ultra-ssd#ga-scope-and-limitations
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/disks-enable-ultra-ssd#ga-scope-and-limitations

Here you can find also in the comments below:
https://azure.microsoft.com/en-us/blog/announcing-ultra-ssd-the-next-generation-of-azure-disks-technology-preview/

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pierluigilenoci]

@jackfrancis could you please take a look?

[CecileRobertMichon]

@pierluigilenoci Jack is away this week. To be clear, is the ask here to support VMSS master upgrade? Or are there additional changes required in AKS Engine to support Ultra SSD?

As Jack mentioned above their is no short term plan to support upgrade for VMSS masters in AKS Engine.

[CecileRobertMichon]

After taking a look at https://github.com/Azure/aks-engine/pull/2905, it looks like UltraSSD is already supported for both agent and master profiles. So the main limitation is that it only supports VMSS, and aks-engine upgrade does not work for VMSS masters. Based on this, my recommendation would be to either only enable UltraSSD for agent pools or, if enabling it for masters as well, to plan on having an alternate way of maintaining the cluster lifecycle, or to not have long running clusters.

Perhaps the documentation could be updated to be more clear about this limitation. I don't see any AKS Engine docs that mention UltraSSD only works with VMSS.

[jackfrancis]

This will be addressed in #3872

[pierluigilenoci]

@jackfrancis great news for the first birthday of this issue! 🎂 🥳
Thank you! ❤️