Aks-engine: Incomplete upgrade of vmss agent pool

[mhyllander]

Is this a request for help?: Yes

---

Is this an ISSUE or FEATURE REQUEST? (choose one): ISSUE

---

What version of acs-engine?:
Version: v0.21.1
GitCommit: 132dab3a5
GitTreeState: clean

---

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.10.6 -> 1.11.2

What happened:
While upgrading a cluster with a vmss agent pool, the upgrade command failed.

It seems that there are two problems:

1. After upgrading the master, acs-engine has an authentication problem and exits before upgrading the vmss agent pool.
2. The upgrade of the vmss agent pool updates the vmss model and tags, but does not actually upgrade the existing vmss instances.

What you expected to happen:
I expected the upgrade command to upgrade all nodes in the cluster.

How to reproduce it (as minimally and precisely as possible):
I created a cluster with custom vnet, one master and a vmss agent pool with two nodes, running k8s 1.10.6. Then I ran the upgrade command to upgrade to k8s 1.11.2:

$ acs-engine upgrade --subscription-id xxxx --resource-group mhy-test-k8s-1 --deployment-dir _output/mhy-test-k8s-1 --location westeurope --upgrade-version 1.11.2 --auth-method client_secret --client-id xxxx --client-secret xxxx --debug
INFO[0000] validating...
DEBU[0000] Resolving tenantID for subscriptionID: xxxx
DEBU[0003] Already registered for "Microsoft.Compute"
DEBU[0003] Already registered for "Microsoft.Storage"
DEBU[0003] Already registered for "Microsoft.Network"
INFO[0003] Name suffix: 42855177
INFO[0003] Gathering agent pool names...
INFO[0003] VM k8s-agentpool1-42855177-vmss_0 in VMSS k8s-agentpool1-42855177-vmss has a current tag of Kubernetes:1.10.6 and a desired tag of Kubernetes:1.11.2. Upgrading this node.

INFO[0003] VM k8s-agentpool1-42855177-vmss_1 in VMSS k8s-agentpool1-42855177-vmss has a current tag of Kubernetes:1.10.6 and a desired tag of Kubernetes:1.11.2. Upgrading this node.

INFO[0003] Master VM name: k8s-master-42855177-0, orchestrator: Kubernetes:1.10.6 (MasterVMs)

INFO[0003] Upgrading to Kubernetes version 1.11.2

INFO[0003] Master nodes StorageProfile: ManagedDisks
INFO[0003] Prepping master nodes for upgrade...
INFO[0003] Resource count before running NormalizeResourcesForK8sMasterUpgrade: 10
INFO[0003] Evaluating if agent pool: master, resource: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')))] needs to be removed
INFO[0003] Evaluating if extension: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')),'/cse', '-master-', copyIndex(variables('masterOffset')))] needs to be removed
INFO[0003] Evaluating if extension: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')), '/computeAksLinuxBilling')] needs to be removed
INFO[0003] Resource count after running NormalizeResourcesForK8sMasterUpgrade: 10
INFO[0003] Total expected master count: 1
INFO[0003] Master nodes that need to be upgraded: 1
INFO[0003] Master nodes that have been upgraded: 0
INFO[0003] Starting upgrade of master nodes...
INFO[0003] masterNodesInCluster: 1
INFO[0003] Upgrading Master VM: k8s-master-42855177-0
INFO[0003] fetching VM: mhy-test-k8s-1/k8s-master-42855177-0
INFO[0003] found nic name for VM (mhy-test-k8s-1/k8s-master-42855177-0): k8s-master-42855177-nic-0
INFO[0003] deleting VM: mhy-test-k8s-1/k8s-master-42855177-0
INFO[0003] waiting for vm deletion: mhy-test-k8s-1/k8s-master-42855177-0
INFO[0166] deleting nic: mhy-test-k8s-1/k8s-master-42855177-nic-0
INFO[0166] waiting for nic deletion: mhy-test-k8s-1/k8s-master-42855177-nic-0
INFO[0192] deleting managed disk: mhy-test-k8s-1/k8s-master-42855177-0_OsDisk_1_d2e861bd33ae47a4a85afecabba802af
INFO[0252] Master offset: 0

INFO[0252] Master pool set count to: 1 temporarily during upgrade...

INFO[0252] Starting ARM Deployment (master-18-09-05T14.52.24-1865577621). This will take some time...
INFO[1036] Finished ARM Deployment (master-18-09-05T14.52.24-1865577621). Succeeded
INFO[1036] Error validating upgraded master VM: k8s-master-42855177-0
FATA[1036] Error upgrading cluster: No Auth Provider found for name "azure"

The current state:

$ kubectl get nodes
NAME                                 STATUS    ROLES     AGE       VERSION
k8s-agentpool1-42855177-vmss000000   Ready     agent     21m       v1.10.6
k8s-agentpool1-42855177-vmss000001   Ready     agent     23m       v1.10.6
k8s-master-42855177-0                Ready     master    27m       v1.11.2

I restarted the upgrade command:

$ acs-engine upgrade --subscription-id xxxx --resource-group mhy-test-k8s-1 --deployment-dir _output/mhy-test-k8s-1 --location westeurope --upgrade-version 1.11.2 --auth-method client_secret --client-id xxxx --client-secret xxxx --debug
INFO[0000] validating...
DEBU[0000] Resolving tenantID for subscriptionID: xxxx
DEBU[0002] Already registered for "Microsoft.Compute"
DEBU[0002] Already registered for "Microsoft.Storage"
DEBU[0002] Already registered for "Microsoft.Network"
INFO[0003] Name suffix: 42855177
INFO[0003] Gathering agent pool names...
INFO[0003] Master VM name: k8s-master-42855177-0, orchestrator: Kubernetes:1.11.2 (UpgradedMasterVMs)

INFO[0003] Upgrading to Kubernetes version 1.11.2

INFO[0003] Master nodes StorageProfile: ManagedDisks
INFO[0003] Prepping master nodes for upgrade...
INFO[0003] Resource count before running NormalizeResourcesForK8sMasterUpgrade: 10
INFO[0003] Evaluating if agent pool: master, resource: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')))] needs to be removed
INFO[0003] Evaluating if extension: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')),'/cse', '-master-', copyIndex(variables('masterOffset')))] needs to be removed
INFO[0003] Evaluating if extension: [concat(variables('masterVMNamePrefix'), copyIndex(variables('masterOffset')), '/computeAksLinuxBilling')] needs to be removed
INFO[0003] Resource count after running NormalizeResourcesForK8sMasterUpgrade: 10
INFO[0003] Total expected master count: 1
INFO[0003] Master nodes that need to be upgraded: 0
INFO[0003] Master nodes that have been upgraded: 1
INFO[0003] Starting upgrade of master nodes...
INFO[0003] masterNodesInCluster: 1
INFO[0003] Master VM: k8s-master-42855177-0 is upgraded to expected orchestrator version
INFO[0003] Expected master count: 1, Creating 0 more master VMs
INFO[0003] Deploying the agent scale sets ARM template...
INFO[0003] Starting ARM Deployment (agentscaleset-18-09-05T15.07.07-954170151). This will take some time...
INFO[0055] Finished ARM Deployment (agentscaleset-18-09-05T15.07.07-954170151). Succeeded
INFO[0055] Upgrading VMSS k8s-agentpool1-42855177-vmss
INFO[0055] No VMs to upgrade for VMSS k8s-agentpool1-42855177-vmss, skipping
INFO[0055] Completed upgrading all VMSS
INFO[0055] Cluster upgraded successfully to Kubernetes version 1.11.2

DEBU[0055] output: wrote _output/mhy-test-k8s-1/apimodel.json

The current state:

$ kubectl get nodes
NAME                                 STATUS    ROLES     AGE       VERSION
k8s-agentpool1-42855177-vmss000000   Ready     agent     23m       v1.10.6
k8s-agentpool1-42855177-vmss000001   Ready     agent     25m       v1.10.6
k8s-master-42855177-0                Ready     master    29m       v1.11.2

In the Azure portal I can see that the two vmss nodes are now not the latest model:

I could manually scale up the vmss:

Then cordon, drain and delete the original two nodes with kubectl, and finally get an upgraded cluster:

$ kubectl get nodes
NAME                                 STATUS    ROLES     AGE       VERSION
k8s-agentpool1-42855177-vmss000002   Ready     agent     21m       v1.11.2
k8s-agentpool1-42855177-vmss000003   Ready     agent     21m       v1.11.2
k8s-master-42855177-0                Ready     master    1h        v1.11.2

Delete the original vmss instances from the Azure portal:

Anythinge else we need to know:

[jackfrancis]

Hi @mhyllander, is this reproducible? We're happy to attempt to repro ourselves if you can provide the original api model (creds redacted) that you used to build the cluster. We haven't seen this in our E2E tests but would love to know if you've stumbled upon an edge case!

[mhyllander]

Hi @jackfrancis, yes I actually did the upgrade twice with the same result. First time was with one node in the vmss, but then I repeated it with two nodes. (Actually I had a similar problem when I tested an upgrade back in January. That time I was using an AvailabilitySet for the agent pool, and the upgrade command failed after each node in the cluster, also with an authentication error. But a lot has happened since then, so maybe it's not relevant.)

I think the main "special" features of the cluster I set up are:

1. Use of a pre-created resource group and custom vnet.
2. The servicePrincipalProfile SP is scoped to the resource group,
3. Use aadProfile.adminGroupID to grant admin privileges to members of an AD security group. The aadProfile.serverAppID SP's manifest therefore specifies "groupMembershipClaims": "SecurityGroup".

I have a script that creates the cluster, here are some excerpts that outline the steps:

    # create resource group
    az group create --name="${RES_GROUP}" --location="$LOCATION"

    # create a service principal for the Kubernetes cluster
    az ad sp create-for-rbac --name="${SP_NAME}" --role='Contributor' --scopes="/subscriptions/${SBSCR}/resourceGroups/${RES_GROUP}" > "$SPJSON"

    # create the custom vnet
    az network vnet create \
       --name "${VNET}" \
       --resource-group "${RES_GROUP}" \
       --address-prefixes "$VNET_CIDR" \
       --subnet-name "${SUBNET}" \
       --subnet-prefix "$VNET_CIDR"

# generate the ARM deployment templates using acs-engine (the apimodel file has been
# generated from a template with actual values inserted for this cluster)
acs-engine generate "${DEPLOYD}/${KUBENAME}.json"

# deploy kubernetes cluster
az group deployment create \
   --name "$DEPLOY" \
   --resource-group "${RES_GROUP}" \
   --template-file ./_output/${KUBENAME}/azuredeploy.json \
   --parameters ./_output/${KUBENAME}/azuredeploy.parameters.json

This is the apimodel used by the acs-engine generate command above:

{
    "apiVersion": "vlabs",
    "properties": {
        "orchestratorProfile": {
            "orchestratorType": "Kubernetes",
            "orchestratorVersion": "1.10.6",
            "kubernetesConfig": {
                "serviceCidr": "10.128.0.0/16",
                "dnsServiceIP": "10.128.0.10",
                "clusterSubnet": "10.129.24.0/21",
                "enableRbac": true,
                "enablePodSecurityPolicy": true,
                "etcdDiskSizeGB": "512",
                "dockerEngineVersion": "17.03.*",
                "kubeletConfig": {
                    "--max-pods": "64",
                    "--authentication-token-webhook": "true"
                },
                "addons": [
                    {
                        "name": "cluster-autoscaler",
                        "enabled": true,
                        "config": {
                            "minNodes": "2",
                            "maxNodes": "5"
                        }
                    }
                ]
            }
        },
        "masterProfile": {
            "count": 1,
            "vmSize": "Standard_D2_v3",
            "dnsPrefix": "mhy-test-k8s-1",
            "vnetSubnetId": "/subscriptions/xxxxxxxx/resourceGroups/mhy-test-k8s-1/providers/Microsoft.Network/virtualNetworks/mhy-test-k8s-1-vnet/subnets/mhy-test-k8s-1-subnet",
            "vnetCidr": "10.0.0.0/8",
            "firstConsecutiveStaticIP": "10.129.31.239",
            "storageProfile" : "ManagedDisks"
        },
        "agentPoolProfiles": [
            {
                "name": "agentpool1",
                "count": 2,
                "vmSize": "Standard_D2_v3",
                "vnetSubnetId": "/subscriptions/xxxxxxxx/resourceGroups/mhy-test-k8s-1/providers/Microsoft.Network/virtualNetworks/mhy-test-k8s-1-vnet/subnets/mhy-test-k8s-1-subnet",
                "storageProfile" : "ManagedDisks",
                "OSDiskSizeGB": 50,
                "availabilityProfile": "VirtualMachineScaleSets"
            }
        ],
        "linuxProfile": {
            "adminUsername": "devop",
            "ssh": {
                "publicKeys": [
                    {
                        "keyData": "kkkkkkkk"
                    }
                ]
            }
        },
        "servicePrincipalProfile": {
            "clientId": "xxxxxxxx",
            "secret": "yyyyyyyy"
        },
        "aadProfile": {
            "serverAppID": "xxxxxxxx",
            "clientAppID": "yyyyyyyy",
            "tenantID": "zzzzzzzz",
            "adminGroupID": "gggggggg"
        }
    }
}

[mhyllander]

Also maybe I should mention that when I ran the acs-engine upgrade command, I used the servicePrincipalProfile SP for authentication (--client-id and --client-secret).

[mhyllander]

Related issue: https://github.com/Azure/acs-engine/issues/2560

[rocketraman]

I ran into a similar problem with a different cause: in my case the upgrade failed due to a drain timeout on one of the nodes:

ERRO[0264] There are pending pods when an error occurred: Drain did not complete within 1m0s

ERRO[0264] pod/xxx

ERRO[0264] Error draining VM in VMSS: Drain did not complete within 1m0s 
FATA[0264] Error upgrading cluster: Drain did not complete within 1m0s

But the tags on the VMSS were updated to the new version, so when retrying the update:

INFO[0132] No VMs to upgrade for VMSS k8s-agentpool2-36147952-vmss, skipping 
INFO[0132] Completed upgrading all VMSS                 
INFO[0132] Cluster upgraded successfully to Kubernetes version 1.12.2

even though the nodes are still on 1.12.1:

$ kubectl get nodes
NAME                                 STATUS                     ROLES    AGE   VERSION
...
k8s-agentpool2-36147952-vmss000000   Ready,SchedulingDisabled   agent    26d   v1.12.1
k8s-agentpool2-36147952-vmss000001   Ready                      agent    25d   v1.12.1

Updating the orchestrator tag on the vmss manually back to 1.12.1 and retrying the upgrade seems to work, but clearly the tag is being updated prematurely here.

[chreichert]

Same problem here, like mhyllander described. Upgrade fails after the upgrade of any master node during validation phase:

INFO[0163] Starting ARM Deployment (master-18-12-05T11.27.18-1386552021). This will take some time...
INFO[0538] Finished ARM Deployment (master-18-12-05T11.27.18-1386552021). Succeeded
INFO[0538] Error validating upgraded master VM: k8s-master-12658618-0
FATA[0538] Error upgrading cluster: No Auth Provider found for name "azure"

After repeating upgrade for every master (every time same error for the master upgrades), acs-engine reports upgrade done, but all VMs in all VMSS are not upgraded yet.

INFO[0002] Deploying the agent scale sets ARM template...
INFO[0002] Starting ARM Deployment (agentscaleset-18-12-05T11.51.25-1847771691). This will take some time...
INFO[0025] Finished ARM Deployment (agentscaleset-18-12-05T11.51.25-1847771691). Succeeded
INFO[0025] Upgrading VMSS k8s-dynamic-12658618-vmss
INFO[0025] No VMs to upgrade for VMSS k8s-dynamic-12658618-vmss, skipping
INFO[0025] Upgrading VMSS k8s-graph-12658618-vmss
INFO[0025] No VMs to upgrade for VMSS k8s-graph-12658618-vmss, skipping
INFO[0025] Upgrading VMSS k8s-static-12658618-vmss
INFO[0025] No VMs to upgrade for VMSS k8s-static-12658618-vmss, skipping
INFO[0025] Completed upgrading all VMSS
INFO[0025] Cluster upgraded successfully to Kubernetes version 1.11.5

The VMSS overview in the portal shows the new K8s version, but all vmss instances still run on previous k8s version.

The upgrade can manually be finalized by using the upgrade functionality on the VMSS instances in the Azure portal, but I think this should be done by acs-engine upgrade itself.

[navicore]

same here, "No VMs to upgrade for VMSS" after successful master upgrade for acs-engine 0.26.1 upgrading from v1.10.9 to v1.11.5

[jackfrancis]

Are any of the vmss upgrade failures here due to vmss master node pool? (<-- that is currently not functional)

[mhyllander]

Not in my case, we only use vmss for the agent pool.

[navicore]

masters here are not vmss either. but some more info:

I just scaled up the VMSS with an az vmss scale command and the new nodes all are of the upgraded version... the VMs that were running at upgrade time are still the old version.

[chreichert]

Our master pool is also not VMSS but AvailabilitySet.
Specialities are: Private cluster, custom VNET, K8s AAD integration.

@navicore: We were able to "finalize" the upgrade of the"old nodes" in Azure portal. If you have critical workload on these nodes consider to drain the nodes in k8s first. Then in Azure portal browse to the VMSS Instances page, select all affected Instances (tick checkboxes) and then click on Upgrade. This will reboot and upgrade these VMs to the new version that was previously set with acs-engine upgrade.

[navicore]

@chreichert that worked. I was able to finish the upgrade for each instance via the portal. Thanks!

[jackfrancis]

Thanks all on this thread. @mboersma / @tariq1890 let's add some brief documentation based on @chreichert's guidance to help folks who have vmss agent pools that end up in incomplete upgrade states.

[chreichert]

@jackfrancis but this still leaves us with the problem, that the upgrade terminates after upgrading each master node with the following error:

INFO[0538] Error validating upgraded master VM: k8s-master-12658618-0
FATA[0538] Error upgrading cluster: No Auth Provider found for name "azure"

Could this be an auth problem in AAD integration enabled clusters?

[tariq1890]

@feiskyer Do you know when this error would be thrown? No Auth Provider found for name "azure"

[feiskyer]

@feiskyer Do you know when this error would be thrown? No Auth Provider found for name "azure"

where's the error comes from? acs-engine or other places? It seems a dependency error which the required package "k8s.io/client-go/plugin/pkg/client/auth" is not imported.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[chreichert]

Can the stale flag please be removed?
The problem with the incomplete upgrade of agent pools and the auth error after upgrading each master node is not solved yet.
Thanks.

[mo-saeed]

Would it possible to add a parameter to aks-engine upgrade, to define the drain timeout ? right now it's only 1m which is not really good for nodes with big number of pods.

[jackfrancis]

I'm going to close this issue for now as there are a lot of distinct upgrade-related issues. Sorry if that annoys folks, I'd certainly encourage folks to re-open if the initial issue is still valid with current aks-engine (v0.45).

We're certainly aware that there are a legion of environmental effects that can cause aks-engine upgrade to fail, and that every edge case isn't accommodated in the idempotent aspirations of the upgrade operation.