Aks-engine: Standard LB-enabled cluster fails to scale up. - Episode 1

Created on 18 Sep 2019 · 19Comments · Source: Azure/aks-engine

Describe the bug
We have the same bug described here https://github.com/Azure/aks-engine/issues/1190 and here https://github.com/Azure/acs-engine/issues/4328
Probably this PR don't solve the problem: https://github.com/Azure/aks-engine/pull/1622

The story continues on this https://github.com/Azure/aks-engine/issues/2033

Steps To Reproduce
I created a cluster with this cluster definition:

{
    "apiVersion": "vlabs",
    "properties": {
        "orchestratorProfile": {
            "orchestratorType": "Kubernetes",
            "orchestratorVersion": "1.15.2",
            "kubernetesConfig": {
                "networkPlugin": "azure",
                "networkPolicy": "azure",
                "loadBalancerSku": "standard",
                "enablePodSecurityPolicy": true,
                "enableRbac": true,
                "addons": [
                    {
                        "name": "tiller",
                        "enabled": false
                    },
                    {
                        "name": "kubernetes-dashboard",
                        "enabled": false
                    },
                    {
                        "name": "heapster",
                        "enabled": false
                    },
                    {
                        "name": "keyvault-flexvolume",
                        "enabled": false
                    },
                    {
                        "name": "metrics-server",
                        "enabled": false
                    },
                    {
                        "name": "blobfuse-flexvolume",
                        "enabled": false
                    }
                ],
                "kubeletConfig": {
                    "--max-pods": "110"
                },
                "apiServerConfig": {
                    "--oidc-issuer-url": "xxxx",
                    "--oidc-client-id": "xxxx",
                    "--oidc-username-claim": "email",
                    "--oidc-groups-claim": "xxxx"
                },
                "controllerManagerConfig": {},
                "schedulerConfig": {}
            }
        },
        "masterProfile": {
            "count": 3,
            "dnsPrefix": "xxxx",
            "vmSize": "Standard_D2s_v3",
            "vnetSubnetId": "xxxx",
            "availabilityProfile": "AvailabilitySet",
            "firstConsecutiveStaticIP": "xxxx"
        },
        "agentPoolProfiles": [
            {
                "name": "minionb2s",
                "count": 2,
                "dnsPrefix": "",
                "vnetSubnetId": "xxxx",
                "vmSize": "Standard_B2s",
                "availabilityProfile": "VirtualMachineScaleSets"
            },
            {
                "name": "minionb4ms",
                "count": 1,
                "dnsPrefix": "",
                "vnetSubnetId": "xxxx",
                "vmSize": "Standard_B4ms",
                "availabilityProfile": "VirtualMachineScaleSets"
            }
        ],
        "linuxProfile": {
            "adminUsername": "xxxx",
            "ssh": {
                "publicKeys": [
                    {
                        "keyData": "ssh-rsa xxxx"
                    }
                ]
            }
        },
        "servicePrincipalProfile": {
            "clientId": "xxxx",
            "secret": "xxxx"
        }
    }
}

and used this command to scale up

aks-engine scale --location westeurope --subscription-id "xxxx" --resource-group "xxx" --api-model ./apimodel.json --node-pool minionb4ms --new-node-count 4 --auth-method client_secret --client-id "xxxx" --client-secret "xxxx"

and I get this error in deployment (Azure Portal)

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.","details":[{"code":"Conflict","message":"{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"VMExtensionProvisioningError\",\r\n \"message\": \"VM has reported a failure when processing extension 'vmssCSE'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=50\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n }\r\n ]\r\n }\r\n}"}]}

and this on CLI

INFO[0000] validating...
INFO[0005] Name suffix: 47670981
INFO[0005] Found no resources with type Microsoft.Network/routeTables in the template.  source="scaling command line"
INFO[0005] Starting ARM Deployment (development-1649939717). This will take some time...
INFO[0412] Finished ARM Deployment (development-1649939717). Error: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'vmssCSE'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=50\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"}]
Error: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'vmssCSE'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=50\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"}]

Expected behavior
The cluster scale-up

AKS Engine version
0.39.1 (6fff62731)

Kubernetes version
1.15.2

Additional context

bug

Source

pierluigilenoci

Most helpful comment

Like in Star Wars I renamed this with the right number 😆

pierluigilenoci on 1 Oct 2019

🚀2

All 19 comments

@ritazh @mboersma could you please take a look

pierluigilenoci on 18 Sep 2019

Can you possibly share, via the CLI, the output of az show on a NIC for a machine which is in the failed state (with subscription/sensitive details scrubbed) and the same for the load balancer containing worker nodes?

I would assume all of the following are true for this scenario to function:

prior to upgrade, your cluster had all worker nodes attached to a standard load balancer.
this cluster has never been converted from not having LB or using basic LB to using SLB
there is a configured outbound pool defined by aks-engine which points to a backend pool containing all your machines

Are both worker pools failed or only one? are they both in the proper backend pools?

alexeldeib on 18 Sep 2019

@alexeldeib this is the output of az network nic show for all 3 NIC and az network lb show for load balancer.

To answer to your request: yes, the cluster has all worker nodes attached to an SLB, I created the cluster with the attached cluster definition so no conversion in the process, the LB has a backend pool with all minions (of both pool) as you can see from the log.

pierluigilenoci on 19 Sep 2019

However, to recreate the problem is simple. Just use the JSON attached to the issue and launch the commands in the sequence I indicated. Obviously adding the correct data of your Azure account.

pierluigilenoci on 19 Sep 2019

@mboersma If you approve also this https://github.com/Azure/homebrew-aks-engine/pull/35 I will really happy to check if the problem is still there with v0.41.0 of aks-engine

pierluigilenoci on 19 Sep 2019

👍1

@alexeldeib I noticed that inside the LB (https://pastebin.com/ddzgymm6, line 73) there is this line "outboundRule": null. In this way, the VM can't reach the internet or I'm wrong?

pierluigilenoci on 19 Sep 2019

👀1

I believe line 194 is the actual issue (same field, empty array, but at the top level). But yes, you’re missing outbound rules on the LB.

@mboersma @ritazh FYI — seems like the outbound rule is getting dropped? That seems likely to be the culprit here. Ill poke around the arm template a bit to see if I can figure out why

alexeldeib on 19 Sep 2019

❤1 👍1

@mboersma with 0.41.0 I have another problem. Could you please approve this PR https://github.com/Azure/homebrew-aks-engine/pull/37 ?

pierluigilenoci on 23 Sep 2019

👍1

@alexeldeib have you poked around the arm template?

pierluigilenoci on 23 Sep 2019

I have not figured this out yet. @mboersma given the above re: outbound rule, does that point you to anything obvious?

alexeldeib on 23 Sep 2019

@mboersma with the latest version of aks-engine I don't have anymore this issue. It has been replaced by another one. https://github.com/Azure/aks-engine/issues/2033

pierluigilenoci on 26 Sep 2019

😕1

Like in Star Wars I renamed this with the right number 😆

pierluigilenoci on 1 Oct 2019

🚀2

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] on 30 Nov 2019

😢

pierluigilenoci on 2 Dec 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] on 30 Mar 2020

Thank you @jackfrancis

pierluigilenoci on 31 Mar 2020

@pierluigilenoci Are you no longer experiencing these issues w/ Standard LB?