Aks-engine: Upgrade fails for VMSS nodes when Azure Standard Load Balancer is provisioned

[ankrause]

Describe the bug
Running aks-engine upgrade on an Azure-based cluster with VMSS nodes fails to upgrade the VMSS nodes if a (standard) loadbalancer service exists.

Output:

time="2019-08-10T19:58:00-07:00" level=error msg="Failure to set capacity for VMSS k8s-linuxpool1-<snip>-vmss"
Error: upgrading cluster: Code="VMExtensionProvisioningError" Message="VM has reported a failure when processing extension 'vmssCSE'. Error message: \"Enable failed:
failed to execute command: command terminated with exit status=50\n[stdout]\n\n[stderr]\n\"."

The error suggests this a connection issue. I've tried SSH'ing into a failed node post-scale operation and I'm unable to query some internet URLs. bing.com returned a response but google.com did not. Both URLs returned a DNS record.

I've tried reproducing this issue in a fresh cluster (using the below template) which hasn't yet had any workloads deployed and the upgrade operation succeeds without error. This only appears to occur if a loadbalancer is provisioned.

I'll also note that scale out which occurs via the Cluster Autoscaler doesn't appear to have this issue. Nodes that get added during scale out scenarios appear to have internet and function as expected. This only occurs after the upgrade operation has started on VMSS nodes. Disabling the CA and manually scaling out the VMSS post-failure results in the same behavior (a failed VMSS instance).

Steps To Reproduce

This is the template used to deploy the original cluster prior to upgrading. The secrets in most cases are in Key Vault, but can likely be inlined. There are some customized/specific versions of addons deployed (CA, KVFlex) but the key appears to be the LB.

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.14.1",
      "kubernetesConfig": {
        "addons": [
          {
            "name": "aad-pod-identity",
            "enabled": false
          },
          {
            "name": "dns-autoscaler",
            "enabled": true
          },
          {
            "name": "container-monitoring",
            "enabled": true,
            "config": {
              "workspaceGuid": "__LogAnalyticsWorkspaceId__",
              "workspaceKey": "__LogAnalyticsWorkspaceKey__"
            }
          },
          {
            "name": "keyvault-flexvolume",
            "enabled": false
          },
          {
            "name": "tiller",
            "enabled": true,
            "containers": [
              {
                "name": "tiller",
                "image": "gcr.io/kubernetes-helm/tiller:v2.14.1",
                "cpuRequests": "50m",
                "memoryRequests": "150Mi",
                "cpuLimits": "50m",
                "memoryLimits": "150Mi"
              }
            ],
            "config": {
              "max-history": "0"
            }
          }
        ],
        "enableRbac": true,
        "loadBalancerSku": "standard",
        "useManagedIdentity": false,
        "azureCNIVersion": "v1.0.23",
        "azureCNIURLLinux": "https://acs-mirror.azureedge.net/cni/azure-vnet-cni-linux-amd64-v1.0.23.tgz"
      }
    },
    "masterProfile": {
      "count": 3,
      "dnsPrefix": "__DnsPrefix__",
      "vmSize": "Standard_D4s_v3"
    },
    "agentPoolProfiles": [
      {
        "name": "linuxpool1",
        "count": 2,
        "vmSize": "Standard_D8s_v3",
        "availabilityProfile": "VirtualMachineScaleSets",
        "osType": "Linux",
        "osDiskSizeGB": 1023,
        "acceleratedNetworkingEnabled": true,
        "singlePlacementGroup": false
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "__SshPublicKey__"
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "__ClusterServicePrincipalClientId__",
      "secret": "",
      "keyvaultSecretRef": {
        "vaultID": "__ClusterServicePrincipalKeyVaultId__",
        "secretName": "ClusterServicePrincipalSecret"
      }
    },
    "certificateProfile": {
      "apiServerCertificate": "__ApiServerCertificate__",
      "apiServerPrivateKey": "__ApiServerPrivateKey__",
      "caCertificate": "__CACertificate__",
      "caPrivateKey": "__CAPrivateKey__",
      "clientCertificate": "__ClientCertificate__",
      "clientPrivateKey": "__ClientPrivateKey__",
      "etcdClientCertificate": "__EtcdClientCertificate__",
      "etcdClientPrivateKey": "__EtcdClientPrivateKey__",
      "etcdServerCertificate": "__EtcdServerCertificate__",
      "etcdServerPrivateKey": "__EtcdServerPrivateKey__",
      "etcdPeerCertificates": [
        "__EtcdPeerCertificate0__",
        "__EtcdPeerCertificate1__",
        "__EtcdPeerCertificate2__"
      ],
      "etcdPeerPrivateKeys": [
        "__EtcdPeerPrivateKey0__",
        "__EtcdPeerPrivateKey1__",
        "__EtcdPeerPrivateKey2__"
      ],
      "kubeConfigCertificate": "__KubeConfigCertificate__",
      "kubeConfigPrivateKey": "__KubeConfigPrivateKey__"
    }
  }
}

Once deployed, create a service that will provision a loadbalancer in Azure. It can be in any namespace and doesn't need an actual workload behind it.

apiVersion: v1
kind: Service
metadata:
  name: test-service
spec:
  ports:
    - port: 443
      protocol: TCP
      targetPort: 443
  type: LoadBalancer

Once the LB has been provisioned, attempt to upgrade the cluster from 1.14.1 to 1.14.5. I have also attempted this with 1.14.5 upgrading to 1.15.2.

aks-engine upgrade --api-model <path to model> --subscription-id <subid> --resource-group <rg name> --location <location> --upgrade-version "1.14.5" --auth-method "cli"

Expected behavior
The cluster upgrade succeeds even if there is a loadbalancer provisioned and in use by the cluster.

AKS Engine version
v0.38.6

Kubernetes version
1.14.1 -> 1.14.5
1.14.5 -> 1.15.2

[ankrause]

Looks like this may not be an issue with upgrading specifically, but rather with adding nodes to a VMSS post-deployment in K8S 1.14.5. It may also occur in other versions higher than 1.14.1 but I haven't tried those.

I just received the same error when using the Cluster Autoscaler to scale out on an otherwise stable 1.14.5 cluster. The initial 2 instances of my node pool were running correctly and their CSE status was succeeded. The 3rd instance created by increasing scale capacity from 2->3 resulted in a "Failed" CSE provisioning status with the same error.

[jackfrancis]

@ankrause can you use the aks-engine v0.39.1 release to re-try your operational scenario? This change went in w/ v0.39.0:

https://github.com/Azure/aks-engine/pull/1622

[ankrause]

Sorry about that - I forgot there was a new minor version available as I had only grabbed patch for fear of breaking something else. Should have re-checked the version notes for anything relevant.

I just tested this again by re-deploying a cluster with 1.14.4 and attempting to upgrade to 1.14.5 using AKS Engine v0.39.1 with a SLB provisioned, same error.

[daguilarv]

We are facing this issue even using aks-engine 0.38.8 with standard loadBalancers, vmss agentpools and k8s 1.13.10.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.