Aks-engine: Scale command terminates with error: Changing property 'platformFaultDomainCount' is not allowed

[chreichert]

Describe the bug
Scaling up or down VMSS Agentpools terminates with the following error:

INFO[0376] Finished ARM Deployment (qa_004_QKNOWS_K8s-1321548680). Error: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\r\n \"error\": {\r\n \"code\": \"PropertyChangeNotAllowed\",\r\n \"message\": \"Changing property 'platformFaultDomainCount' is not allowed.\",\r\n \"target\": \"platformFaultDomainCount\"\r\n }\r\n}"}]
Error: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\r\n \"error\": {\r\n \"code\": \"PropertyChangeNotAllowed\",\r\n \"message\": \"Changing property 'platformFaultDomainCount' is not allowed.\",\r\n \"target\": \"platformFaultDomainCount\"\r\n }\r\n}"}]

The scaling itself is working, nodes are added or deleted, but the apimodel is not updated to reflect updated cluster state after scaling.

Steps To Reproduce

• Set up Cluster with attached apimodel
• Upgrade to 1.14.3
• Scale agentpool with command:

aks-engine scale --subscription-id --resource-group --location northeurope --api-model deployment-20190625_084134/arm-deploy/apimodel.json --new-node-count=7 --node-pool static --auth-method client_secret --client-id --client-secret --debug

api-model

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.10",
      "kubernetesConfig": {
        "addons": [
          {
            "name": "blobfuse-flexvolume",
            "enabled": false
          },
          {
            "name": "smb-flexvolume",
            "enabled": false
          },
          {
            "name": "keyvault-flexvolume",
            "enabled": false
          },
          {
            "name": "cluster-autoscaler",
            "enabled": false,
            "containers": [
              {
                "name": "cluster-autoscaler",
                "cpuRequests": "100m",
                "memoryRequests": "300Mi",
                "cpuLimits": "100m",
                "memoryLimits": "300Mi"
              }
            ],
            "config": {
              "maxNodes": "5",
              "minNodes": "1"
            }
          }
        ],
        "enableRbac": true,
        "privateCluster": {
          "enabled": true
        },
        "networkPlugin": "kubenet",
        "networkPolicy": "calico",
        "cloudProviderBackoff": true,
        "cloudProviderBackoffRetries": 6,
        "cloudProviderBackoffJitter": 1,
        "cloudProviderBackoffDuration": 5,
        "cloudProviderBackoffExponent": 1.5,
        "cloudProviderRateLimit": false,
        "cloudProviderRateLimitQPS": 3,
        "cloudProviderRateLimitBucket": 10
      }
    },
    "aadProfile": {
      "serverAppID": "***",
      "clientAppID": "***",
      "tenantID": "***"
    },
    "masterProfile": {
      "count": 3,
      "dnsPrefix": "***",
      "vmSize": "Standard_D2s_v3",
      "OSDiskSizeGB": 128,
      "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet",
      "firstConsecutiveStaticIP": "10.239.255.10",
      "vnetCidr": "10.239.0.0/16"
    },
    "agentPoolProfiles": [
      {
        "name": "dynamic",
        "count": 4,
        "vmSize": "Standard_D16s_v3",
        "OSDiskSizeGB": 128,
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "VirtualMachineScaleSets",
        "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet"
      },
      {
        "name": "graph",
        "count": 1,
        "vmSize": "Standard_E32s_v3",
        "OSDiskSizeGB": 128,
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "VirtualMachineScaleSets",
        "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet"
      },
      {
        "name": "static",
        "count": 8,
        "vmSize": "Standard_D16s_v3",
        "OSDiskSizeGB": 128,
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "VirtualMachineScaleSets",
        "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet"
      },
      {
        "name": "elastic",
        "count": 5,
        "vmSize": "Standard_D32s_v3",
        "OSDiskSizeGB": 128,
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "VirtualMachineScaleSets",
        "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "***"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "***",
      "secret": "***"
    }
  }
}

Expected behavior
Cluster scales without any error

AKS Engine version
0.37.3

Kubernetes version
1.14.3

Additional context

[mboersma]

@chreichert I'm trying to reproduce this but haven't seen it yet. The platformFaultDomain count property only exists for a VM Availability Set, not a VM Scale Set. So presumably the error originates with the 3 masters which are in a VMAS.

Did you create the cluster with aks-engine v0.37.3 or an earlier version?

Did you upgrade one minor release at a time, or straight from 1.10.13 to 1.14.3 with --force?

Can you look in the portal to see what the current FD count is for the master VMAS?

[chreichert]

@mboersma The cluster was initially built a while ago with ACS-Engine 0.21.2.
Upgrade path was:

• Upgrade to 1.11.5 with ACS-Engine 0.26.2
• Upgrade to 1.11.6 with AKS-Engine 0.29.1
• Upgrade to 1.12.8 with AKS-Engine 0.37.3
• Upgrade to 1.13.7 with AKS-Engine 0.37.3
• Upgrade to 1.14.3 with AKS-Engine 0.37.2

If FD count means the number of Fault domains in the current master-availabilityset, its 2 Fault domains.

[chreichert]

I could also reproduce the issue by setting up a brand new cluster as described above.
It was not necessary to execute all upgrades up to 1.14.3. After upgrading to k8s 1.12.8 with AKS-Engine 0.37.3, scaling any agentpool with AKS-Engine 0.37.3 fails as described above.

[CecileRobertMichon]

This is fixed in v0.37.5 and v0.38.1.

[tapanhalani]

Just upgraded AKS from azure portal, from v1.11.8 -> 1.11.10 . When I tried scaling from 1 to 2 nodes (or for that matter, any number) I got this error "Changing property 'platformFaultDomainCount' is not allowed" .

[CecileRobertMichon]

@tapanhalani please open a support ticket.

[mboersma]

This issue was specific to aks-engine scale and isn't actually related to az aks upgrade.