Aks-engine: Aks-engine upgrade command fails on cluster that was deployed with older acs-engine.

[OldSchooled]

Is this a request for help?:

Yes.

Is this an ISSUE or FEATURE REQUEST?

Issue.

What version of aks-engine?:

acs-engine 0.18.8 & aks-engine 0.29.1

The template used is here:

{
    "apiVersion": "vlabs",
    "properties": {
        "orchestratorProfile": {
            "orchestratorType": "Kubernetes",
            "orchestratorVersion": "1.10.4"
        },
        "agentPoolProfiles": [
            {
                "storageProfile": "ManagedDisks",
                "name": "foo0206a",
                "count": 2,
                "osType": "Linux",
                "vmSize": "Standard_A2m_v2",
                "availabilityProfile": "VirtualMachineScaleSets"
            }
        ],
        "servicePrincipalProfile": {
            "clientId": "XXX",
            "secret": "XXX"
        },
        "linuxProfile": {
            "adminUsername": "fooUser",
            "ssh": {
                "publicKeys": [
                    {
                        "keyData": "XXX"
                    }
                ]
            }
        },
        "masterProfile": {
            "storageProfile": "ManagedDisks",
            "count": 1,
            "dnsPrefix": "k8s-master-foo0206a",
            "vmSize": "Standard_F2s"
        }
    }
}

Kubernetes version:
1.10.4 -> 1.11.6

What happened:
I have clusters that were provisioned with acs-engine version 0.18.8. When upgrading the acs-engine to aks-engine 0.29.1 and then attempting the aks-engine upgrade command I receive the following error:

INFO[1129] Finished ARM Deployment (master-19-02-11T13.45.56-1306699829). Error: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-master-0'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=30\\n[stdout]\\n\\n[stderr]\\nConnection to k8s.gcr.io 443 port [tcp/https] succeeded!\\nConnection to gcr.io 443 port [tcp/https] succeeded!\\nConnection to docker.io 443 port [tcp/https] succeeded!\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"}]
INFO[1129] Error creating upgraded master VM: k8s-master-40723034-0
FATA[1129] Error upgrading cluster: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[{"code":"Conflict","message":"{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-master-0'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=30\\n[stdout]\\n\\n[stderr]\\nConnection to k8s.gcr.io 443 port [tcp/https] succeeded!\\nConnection to gcr.io 443 port [tcp/https] succeeded!\\nConnection to docker.io 443 port [tcp/https] succeeded!\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"}]

What you expected to happen:
I expect the upgrade to complete successfully.

How to reproduce it (as minimally and precisely as possible):

• Using the provided template, run the acs-engine generate command using acs-engine version 0.18.8
• Run az group create to create a clean resource group.
• Run az group deployment create using the the output folder created in step one.
• Using aks-engine version 0.29.1, Run aks-engine upgrade command using the output folder from above as the deployment-dir and 1.11.6 as the target version.
  (note: I've also tried this with acs-engine version 0.30.1 with the same result).

Anything else we need to know:
I have included the cloud-init-output.log and the cluster-provision.log as attachments. Thank you for your time.

cloud-init-output.log
cluster-provision.log

[welcome[bot]]

👋 Thanks for opening your first issue here! If you're reporting a 🐞 bug, please make sure you include steps to reproduce it.

[denniszielke]

We reproduced the issue and have identified the underlying problem.
In our cluster-provisionlog we also get the message that ensureK8sControlPlane returns false - meaning the api-server is not coming up after the upgrade. The deployment keeps waiting until it eventually times out.
The reason for the api server not coming up is because it is trying to load a configuration file that is being referenced for the --audit-policy-file as part of the mount operation that is loaded into the api-server container as defined in /etc/kubernetes/manifests/kube-apiserver.yaml. The issue here being that this file used to be located at /etc/kubernetes/manifests/audit-policy.yaml while in newer aks-engine and kubernetes version it is now located in /etc/kubernetes/addons/audit-policy.yaml. The file is put there during the arm deployment that is part of the provisioning of a new master vm that is defined by the custom-data.
We are working on fixing this. In the meantime you can check if this is in fact the issue that you are also experiencing by doing the following:

• Start the upgrade (using the newest aks-engine release 0.32.3) on your existing cluster
• The master vm will be deleted
• Watch for the new master vm to come up behind the public ip
• When the new master is reachable log into it via ssh and copy the policy file from /etc/kubernetes/addons/audit-policy.yaml to /etc/kubernetes/manifests/audit-policy.yaml
• This should ensure that the api-server will eventually be able to start

By doing this my upgrade from 1.x to 11.8 worked successfully (it took about an hour).
In the meantime we are working on the fix - assuming you are experiencing the same.

[denniszielke]

It seems the value is being set here: https://github.com/Azure/aks-engine/blob/737ffe56be0b6972deb10c0fe0f547e174cbfc3a/pkg/api/defaults-apiserver.go#L91 which needs a switch when upgrading from old acs-engine to aks-engine to ensure that the file is referenced from the new location.

[denniszielke]

Another alternative for a successful upgrade:

• Go into the output folder of the initial cluster deployment
• In the apimodel.json find the entry apiServerConfig
• Change the value for the audit policy file to "--audit-policy-file": "/etc/kubernetes/addons/audit-policy.yaml"
• Upgrade should work normally

[OldSchooled]

Dennis, you rock my socks. Your team also rocks my socks. There's a collective rocking of socks happening over here.

Using the methods described, we are able to upgrade our clusters from 1.10.x to 1.11.8. I will continue testing the rest of the upgrade path but so far, so good.

Update: I was able to successfully upgrade the test cluster all the way to 1.13.4.