Aks-engine: Upgrade older cluster fails with error "Changing property 'platformFaultDomainCount' is not allowed"

[chreichert]

Describe the bug
Upgrading an older cluster, that was initially created with ACS-Engine 0.21.2, from 1.13.5 to 1.14.7 stops while deploying agentscalesets with error: "Changing property 'platformFaultDomainCount' is not allowed"

Steps To Reproduce

• Set up Cluster with attached apimodel using ACS-Engine 0.21.1
• Upgrade to 1.11.5 using ACS-Engine 0.26.2
• Upgrade to 1.11.6 using AKS-Engine 0.29.1
• Upgrade to 1.12.8 using AKS-Engine 0.36.4
  some manual tweaking necessary due to Calico upgrade but finally working fine (first master became ready after adding tolerations to calico-node pod, Agent nodes needed Re-Imaging to become ready)

• Upgrade to 1.13.5 using AKS-Engine 0.36.4
  Up to here, all upgrades ran more or less without problems.

• Upgrade to 1.14.7 using AKS-Engine 0.44.0 with command:

  aks-engine upgrade --subscription-id --resource-group --location northeurope --api-model deployment-20191115_131752/arm-deploy/apimodel.json --upgrade-version 1.14.7 --auth-method client_secret --client-id --client-secret --debug

Produces the following error:

INFO[3302] Finished ARM Deployment (agentscaleset-19-11-21T08.43.12-2133258811). Error: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\rn \"error\": {\rn \"code\": \"PropertyChangeNotAllowed\",\rn \"message\": \"Changing property 'platformFaultDomainCount' is not allowed.\",\rn \"target\": \"platformFaultDomainCount\"\rn }\rn}"}]
ERRO[3302] error applying upgrade template in upgradeAgentScaleSets: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\rn \"error\": {\rn \"code\": \"PropertyChangeNotAllowed\",\rn \"message\": \"Changing property 'platformFaultDomainCount' is not allowed.\",\rn \"target\": \"platformFaultDomainCount\"\rn }\rn}"}]
Error: upgrading cluster: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\rn \"error\": {\rn \"code\": \"PropertyChangeNotAllowed\",\rn \"message\": \"Changing property 'platformFaultDomainCount' is not allowed.\",\rn \"target\": \"platformFaultDomainCount\"\rn }\rn}"}]

api-model

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.10",
      "kubernetesConfig": {
        "addons": [
          {
            "name": "blobfuse-flexvolume",
            "enabled": false
          },
          {
            "name": "smb-flexvolume",
            "enabled": false
          },
          {
            "name": "keyvault-flexvolume",
            "enabled": false
          },
          {
            "name": "cluster-autoscaler",
            "enabled": false,
            "containers": [
              {
                "name": "cluster-autoscaler",
                "cpuRequests": "100m",
                "memoryRequests": "300Mi",
                "cpuLimits": "100m",
                "memoryLimits": "300Mi"
              }
            ],
            "config": {
              "maxNodes": "5",
              "minNodes": "1"
            }
          }
        ],
        "enableRbac": true,
        "privateCluster": {
          "enabled": true
        },
        "networkPlugin": "kubenet",
        "networkPolicy": "calico",
        "cloudProviderBackoff": true,
        "cloudProviderBackoffRetries": 6,
        "cloudProviderBackoffJitter": 1,
        "cloudProviderBackoffDuration": 5,
        "cloudProviderBackoffExponent": 1.5,
        "cloudProviderRateLimit": false,
        "cloudProviderRateLimitQPS": 3,
        "cloudProviderRateLimitBucket": 10
      }
    },
    "aadProfile": {
      "serverAppID": "***",
      "clientAppID": "***",
      "tenantID": "***"
    },
    "masterProfile": {
      "count": 3,
      "dnsPrefix": "***",
      "vmSize": "Standard_D2s_v3",
      "OSDiskSizeGB": 128,
      "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet",
      "firstConsecutiveStaticIP": "10.239.255.10",
      "vnetCidr": "10.239.0.0/16"
    },
    "agentPoolProfiles": [
      {
        "name": "dynamic",
        "count": 1,
        "vmSize": "Standard_D16s_v3",
        "OSDiskSizeGB": 128,
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "VirtualMachineScaleSets",
        "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet"
      },
      {
        "name": "graph",
        "count": 1,
        "vmSize": "Standard_E32s_v3",
        "OSDiskSizeGB": 128,
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "VirtualMachineScaleSets",
        "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet"
      },
      {
        "name": "static",
        "count": 1,
        "vmSize": "Standard_D16s_v3",
        "OSDiskSizeGB": 128,
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "VirtualMachineScaleSets",
        "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet"
      },
      {
        "name": "elastic",
        "count": 1,
        "vmSize": "Standard_D32s_v3",
        "OSDiskSizeGB": 128,
        "storageProfile": "ManagedDisks",
        "availabilityProfile": "VirtualMachineScaleSets",
        "vnetSubnetId": "/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/kubernetes-vnet/subnets/kubernetes-subnet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "***"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "***",
      "secret": "***"
    }
  }
}

Expected behavior
Cluster can be upgraded to 1.14.7 without any error

AKS Engine version
0.44.0

Kubernetes version
1.13.5

Additional context

[chreichert]

We once had a similar problem scaling up this cluster wich resulted in the same error: #1530

[chreichert]

Has anybody an idea how to workaround this, or when it will be fixed?

[jackfrancis]

I will attempt to repro

[jackfrancis]

@chreichert what is the fault domain value when you do this against the vmss that is throwing the error during upgrade?

$ az vmss show -n <name of VMSS> -g <name of resource group> | grep 'platformFaultDomainCount'
  "platformFaultDomainCount": null,

[jackfrancis]

I was unable to repro this. There should be a way to manually update the api model so that this error does not occur during upgrade. (let's find the platformFaultDomainCount values on the existing cluster(s) + node pools)

[chreichert]

@jackfrancis Thanks for trying to repro, which I know is very time consuming.
I ran the command against all 4 vmss with the following results:

➜ .ssh az vmss show -n k8s-dynamic-11480702-vmss -g qa_004_QKNOWS_K8s | grep 'platformFaultDomainCount'
"platformFaultDomainCount": null,
➜ .ssh az vmss show -n k8s-elastic-11480702-vmss -g qa_004_QKNOWS_K8s | grep 'platformFaultDomainCount'
"platformFaultDomainCount": null,
➜ .ssh az vmss show -n k8s-static-11480702-vmss -g qa_004_QKNOWS_K8s | grep 'platformFaultDomainCount'
"platformFaultDomainCount": null,
➜ .ssh az vmss show -n k8s-graph-11480702-vmss -g qa_004_QKNOWS_K8s | grep 'platformFaultDomainCount'
"platformFaultDomainCount": null,
➜ .ssh

Looking at the failed deployment in the portal, it looks like the error occurs updating the master-availabilityset (during the agentpool deployment):

I downloaded the deployment template and searched it for platformDomainCount. I found one occurence, where it obviously is trying to update platformDomainCount on the master availabilityset:

    {
        "type": "Microsoft.Compute/availabilitySets",
        "apiVersion": "[variables('apiVersionCompute')]",
        "name": "[variables('masterAvailabilitySet')]",
        "location": "[variables('location')]",
        "sku": {
            "name": "Aligned"
        },
        "properties": {
            "platformFaultDomainCount": "[if(contains(split('canadacentral,centralus,eastus,eastus2,northcentralus,northeurope,southcentralus,westeurope,westus',','),variables('location')),3,if(equals('centraluseuap',variables('location')),1,2))]",
            "platformUpdateDomainCount": 3
        }
    },

Hope this helps.

If needed, i can provide the full deployment template, just tell me where and how to upload.

Could this probably relate to a problem we once had while scaling this cluster see #1530 and #1581 ?
There it had something to do with a cluster having masters in an availabilityset and agents as VMSS.

[jackfrancis]

Masters as AS w/ VMSS is fine.

What is the actual value of platformFaultDomainCount on the master vm before upgrade?

[chreichert]

➜ .ssh az vm availability-set show -n master-availabilityset-11480702 -g qa_004_QKNOWS_K8s
{
"id": "/subscriptions/XXX/resourceGroups/qa_004_QKNOWS_K8s/providers/Microsoft.Compute/availabilitySets/master-availabilityset-11480702",
"location": "northeurope",
"name": "master-availabilityset-11480702",
"platformFaultDomainCount": 2,
"platformUpdateDomainCount": 3,
"proximityPlacementGroup": null,
"resourceGroup": "qa_004_QKNOWS_K8s",
"sku": {
"capacity": null,
"name": "Aligned",
"tier": null
},
"statuses": null,
"tags": null,
"type": "Microsoft.Compute/availabilitySets",
"virtualMachines": [
{
"id": "/subscriptions/XXX/resourceGroups/QA_004_QKNOWS_K8S/providers/Microsoft.Compute/virtualMachines/K8S-MASTER-11480702-0",
"resourceGroup": "QA_004_QKNOWS_K8S"
},
{
"id": "/subscriptions/XXX/resourceGroups/QA_004_QKNOWS_K8S/providers/Microsoft.Compute/virtualMachines/K8S-MASTER-11480702-1",
"resourceGroup": "QA_004_QKNOWS_K8S"
},
{
"id": "/subscriptions/XXX/resourceGroups/QA_004_QKNOWS_K8S/providers/Microsoft.Compute/virtualMachines/K8S-MASTER-11480702-2",
"resourceGroup": "QA_004_QKNOWS_K8S"
}
]
}

Looks like it is 2 currently, and it wants to upgrade it to 3. But why does it do that on the upgrade of the agentscalesets? The upgrade of the masters before, all three were running fine.

[jackfrancis]

It looks like the ARM expression in the template you pasted above will change it to 3. Could you try manually adding this to the api model (under the “masterProfile” config object) before running upgrade using v0.44 of aks-engine:

“platformFaultDomainCount”: 2

[chreichert]

As it is already late evening here, i will try that tomorrow morning and get back to you here.

[chreichert]

With setting this parameter in the current apimodel, I could successfully continue the aborted upgrade.
I could also upgrade a newly set up cluster (as described in the initial comment), from 1.13.5 to 1.14.7 using aks-engine 0.44.2 and inserting the parameter “platformFaultDomainCount”: 2 before doing the final upgrade.
I still need to test, if we can still use the scale functionality on the upgraded cluster, and if we are able to further upgrade to at least 1.15.x.

[jackfrancis]

@serbrech @yangl900 FYI in case you're seeing any of this in AKS?

[chreichert]

Further tests were also successful. Scaling the cluster is still working, and a further upgrade to 1.15.4 was also possible.

I consider the workaround as functioning.

There is still the question, why AKS-Engine wants to manipulate the platformFaultDomainCount of the master AvailabilitySet during an upgrade step of the AgentScaleSets?

[jackfrancis]

@chreichert It shouldn't. I 'd like to keep this issue open in case someone else runs into something similar, which might help us to more accurately triage.

@serbrech recently implemented the functionality that allows platformFaultDomainCount setting to be passed in w/ the api model, so it's possible there are side effects from that change that we haven't anticipated.

So, very happy you have a proven workaround, let's keep this issue open and call root cause as yet unknown.

[serbrech]

@chreichert Hi, I only got hold of this thread recently, I'll try to figure this out.
My implementation is supposed to completely remove the master availability set from the template ( ref https://github.com/Azure/aks-engine/pull/2207/commits/7722d194385c846c6febd633f23ff226c5c2b1a5#r342318781)
We upgrade each VM resource individually during upgrade, so the VMAS is not relevant in this operation and should always be left untouched.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.