Aks-engine: [BUG-0.37.4] Scaling up/down fails with Changing property 'platformFaultDomainCount' is not allowed

[mo-saeed]

Describe the bug

• Install kubernetes cluster 1.13.5 with version 0.33.2
• Upgrade to 1.14.3 with version 0.37.4
• Scale up one node
  Got this error:
  {"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.","details":[{"code":"Conflict","message":"{\r\n \"error\": {\r\n \"code\": \"PropertyChangeNotAllowed\",\r\n \"message\": \"Changing property 'platformFaultDomainCount' is not allowed.\",\r\n \"target\": \"platformFaultDomainCount\"\r\n }\r\n}"}]}
  It seems it's related to this change https://github.com/Azure/aks-engine/pull/1509
  Steps To Reproduce
• Install kubernetes cluster 1.13.5 with version 0.33.2
• Upgrade to 1.14.3 with version 0.37.4
• Scale up one node

Expected behavior
Be able to scale up/down nodes with version 0.37.4

AKS Engine version
0.37.4
Kubernetes version

1.14.3 or 1.13.7

[CecileRobertMichon]

related to #1530

[mo-saeed]

@CecileRobertMichon are you able to reproduce this now ?

[CecileRobertMichon]

I haven't been able to look at this, @mboersma will take a look later.

[mo-saeed]

Thanks !

[mboersma]

I tried to reproduce this but ran into some CSE error I couldn't get past in this step Upgrade to 1.14.3 with version 0.37.4. So I upgraded using master, which worked, but then wasn't able to cause any problems with scaling. I'm trying again now.

@mo-saeed did you upgrade with --force? And do the logs show anything interesting if you try to scale with --debug?

[mo-saeed]

Hi @mboersma, No I didn't use --force as i upgraded from 1.13.5 to 1.14.3.
Yes there's a known issue now with 0.37.4 (invalid key). I also tried with 0.38.0 and I was able to reproduce the same scaling error.

So may be you can try to install 1.13.5 with aks-engine 0.33.2 and then upgrade to 1.14.3 with aks-engine 0.38.0 then scale and that should reproduce the issue.

[mboersma]

I haven't succeeded at reproducing this bug yet:

$ cat ../kubernetes.json 
{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes"
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "",
      "vmSize": "Standard_D2_v3"
    },
    "agentPoolProfiles": [
      {
        "name": "agentpool1",
        "count": 2,
        "vmSize": "Standard_D2_v3",
        "availabilityProfile": "AvailabilitySet"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": ""
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "",
      "secret": ""
    }
  }
}
$ aks-engine-v0.33.2 version
Version: v0.33.2
GitCommit: 016594238
GitTreeState: clean
$ aks-engine-v0.33.2 deploy --debug \
    --dns-prefix mycluster -l eastus -f \
    -m ../kubernetes.json \
    --client-id=${CLIENT_ID} --client-secret=${CLIENT_SECRET} \
    --set orchestratorProfile.orchestratorVersion=1.13.5
$ aks-engine-v0.38.0 version
Version: v0.38.0
GitCommit: d0b1b77af
GitTreeState: clean
$ aks-engine-v0.38.0 upgrade --debug \
    -l eastus -g mycluster \
    -m _output/apimodel.json \
    --client-id=${CLIENT_ID} --client-secret=${CLIENT_SECRET} \
    -k 1.14.3
$ aks-engine-v0.38.0 scale --debug \
    -l eastus -g mycluster \
    -m _output/apimodel.json \
    --client-id=${CLIENT_ID} --client-secret=${CLIENT_SECRET} \
    --node-pool=agentpool1 -c 3

@mo-saeed is your cluster configuration similar? Any more ideas on how to reproduce this?

[mo-saeed]

Here’s my config
{ "apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorType": "Kubernetes", "orchestratorVersion": "1.13.5", "kubernetesConfig": { "networkPlugin": "azure", "networkPolicy": "azure", "loadBalancerSku": "basic", "enablePodSecurityPolicy": false, "enableRbac": true, "addons": [{ "name": "tiller", "enabled": false }, { "name": "kubernetes-dashboard", "enabled": false }, { "name": "heapster", "enabled": false }, { "name": "keyvault-flexvolume", "enabled": false }, { "name": "metrics-server", "enabled": false }, { "name": "blobfuse-flexvolume", "enabled": false } ], "kubeletConfig": { "--max-pods": "110" }, "apiServerConfig": { }, "controllerManagerConfig": {}, "schedulerConfig": {} } }, "masterProfile": { "count": 1, "dnsPrefix": "aks-dev", "vmSize": "Standard_B2s", "vnetSubnetId": "****", "availabilityProfile": "AvailabilitySet", "firstConsecutiveStaticIP": "****" }, "agentPoolProfiles": [{ "name": "minion", "count": 2, "dnsPrefix": "", "vnetSubnetId": "****", "vmSize": "Standard_D2s_v3", "availabilityProfile": "VirtualMachineScaleSets" }], "linuxProfile": { "adminUsername": "admin", "ssh": { "publicKeys": [{ "keyData": "****" }] } }, "servicePrincipalProfile": { "clientId": "****", "secret": "****" } } }

[mo-saeed]

Also may be try to use westeurope region, may be it’s only reproducible there

[mboersma]

I can see the bug now–it required having the agent pool be VMSS actually. I'm working on a fix. Thanks for your help and patience @mo-saeed.

[tapanhalani]

I just upgrade AKS cluster from 1.11.8 -> 1.11.10 . Now I am trying to scale up/down the cluster, and it is resulting in this error "Changing property 'platformFaultDomainCount' is not allowed" . My production cluster is down due to this, and I followed the documentation for upgrades and scaling. Also, my cluster is in west-europe region.

[mo-saeed]

@tapanhalani which aks-engine version you use ?

[tapanhalani]

@mo-saeed How can I find that? I am just trying this all from the portal, not the cli.

[mo-saeed]

Are you using hosted aks service ? or aks-engine ?

[tapanhalani]

I am using hosted AKS service.

[mo-saeed]

So open azure support ticket and link this issue there, and ask them to include the fix :)

[tapanhalani]

We have just created an issue in Azure support. But I thought maybe I can get some insight of the issue/solution here. So this is completely unrelated to aks-engine ?

[mo-saeed]

Not completely as far as i know, It's sometimes aks team are behind the latest updates from the open-source aks-engine.

If you want the current aks-engine version in your aks cluster, may be check the tags on the agent pool nodes. I believe it should be there.

[tapanhalani]

Thanks for the info. But the problem is that there are no pool nodes in my resource group ... all VMs seem to have vanished (disks and NICs are still there). The AKS cluster is still showing the latest scale number that I tried to scale to.