Aks-engine: Installing Cluster fails with exist code 99

[mo-saeed]

Describe the bug
Installation of kubernetes 1.13.6 with aks-engine 0.37.4 fails with the following error:

"code": "VMExtensionProvisioningError",
        "message": "VM has reported a failure when processing extension 'cse-master-0'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=99\n[stdout]\n\n[stderr]\nConnection to k8s.gcr.io 443 port [tcp/https] succeeded!\nConnection to gcr.io 443 port [tcp/https] succeeded!\nConnection to docker.io 443 port [tcp/https] succeeded!\n\"."

ERR_APT_UPDATE_TIMEOUT=99 # Timeout waiting for apt-get update to complete

It seems that it's related to this change https://github.com/Azure/aks-engine/pull/1538

Steps To Reproduce
Install a new k8s cluster (1.13.6) with aks-engine 0.37.4

Expected behavior
To be able to install cluster with 0.37.4

AKS Engine version
0.37.4
Kubernetes version
1.13.6

[tumluliu]

Have met the same issue with aks-engine 0.37.4 and k8s 1.15.0

[CecileRobertMichon]

Looking

[CecileRobertMichon]

This is due to the clear-containers key expiring:

+ tee /tmp/apt-get-update.out
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
Err:8 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release.gpg
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release: The following signatures were invalid: KEYEXPIRED 1562449942
W: Failed to fetch http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/Release.gpg  The following signatures were invalid: KEYEXPIRED 1562449942
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu xenial-security InRelease
Ign:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:6 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Err:8 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release.gpg
  The following signatures were invalid: KEYEXPIRED 1562449942
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release: The following signatures were invalid: KEYEXPIRED 1562449942
W: Failed to fetch http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/Release.gpg  The following signatures were invalid: KEYEXPIRED 1562449942
W: Some index files failed to download. They have been ignored, or old ones used instead.

[CecileRobertMichon]

Kata / clear containers regenerated the keys so this issue should be fixed now, @mo-saeed @tumluliu can you please confirm?

[mo-saeed]

@CecileRobertMichon I never used kata/clear containers, but anyway i will try to deploy the cluster and will update you

[CecileRobertMichon]

@mo-saeed the clear containers apt source is pre-baked in the base Ubuntu image the VMs are built with, that's why you would see this error even without specifying clear containers.

[mo-saeed]

@CecileRobertMichon I still have the same issue } Deployment failed. Correlation ID: 7c9d0dc0-5138-4bad-ae61-5791caec4908. { "status": "Failed", "error": { "code": "ResourceDeploymentFailure", "message": "The resource operation completed with terminal provisioning state 'Failed'.", "details": [ { "code": "VMExtensionProvisioningError", "message": "VM has reported a failure when processing extension 'cse-master-0'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=99\n[stdout]\n\n[stderr]\nConnection to k8s.gcr.io 443 port [tcp/https] succeeded!\nConnection to gcr.io 443 port [tcp/https] succeeded!\nConnection to docker.io 443 port [tcp/https] succeeded!\n\"." } ] } }

And it only happens with 0.37.4 when using 0.36.1 it passes

[CecileRobertMichon]

@mo-saeed could you please ssh into the master VM and send me the output of tail /var/log/azure/cluster-provision.log?

Also what apimodel are you using and what region? I'll try to repro as well.

I'm also working on v0.38.0 right now so I'd be interested in knowing if it repros there if you're able to build from source. If not, I'll upload the binaries soon.

[mo-saeed]

Here's the log :

```
Err:7 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04 Release.gpg
The following signatures were invalid: KEYEXPIRED 1562449942
Fetched 1,163 B in 0s (1,462 B/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04 Release: The following signatures were invalid: KEYEXPIRED 1562449942
W: Failed to fetch http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/Release.gpg The following signatures were invalid: KEYEXPIRED 1562449942
W: Some index files failed to download. They have been ignored, or old ones used instead.

• '[' 10 -eq 10 ']'
• return 1
• exit 99```

It's in west-europe.

[CecileRobertMichon]

@mo-saeed can please you share your apimodel? I still can't repro...

[mo-saeed]

@CecileRobertMichon
{ "apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorType": "Kubernetes", "orchestratorVersion": "1.14.3", "kubernetesConfig": { "networkPlugin": "azure", "networkPolicy": "azure", "loadBalancerSku": "basic", "enablePodSecurityPolicy": false, "enableRbac": true, "addons": [{ "name": "tiller", "enabled": false }, { "name": "kubernetes-dashboard", "enabled": false }, { "name": "heapster", "enabled": false }, { "name": "keyvault-flexvolume", "enabled": false }, { "name": "metrics-server", "enabled": false }, { "name": "blobfuse-flexvolume", "enabled": false } ], "kubeletConfig": { "--max-pods": "110" }, "apiServerConfig": { }, "controllerManagerConfig": {}, "schedulerConfig": {} } }, "masterProfile": { "count": 1, "dnsPrefix": "aks-dev", "vmSize": "Standard_B2s", "vnetSubnetId": "****", "availabilityProfile": "AvailabilitySet", "firstConsecutiveStaticIP": "****" }, "agentPoolProfiles": [{ "name": "minion", "count": 2, "dnsPrefix": "", "vnetSubnetId": "****", "vmSize": "Standard_D2s_v3", "availabilityProfile": "VirtualMachineScaleSets" }], "linuxProfile": { "adminUsername": "admin", "ssh": { "publicKeys": [{ "keyData": "****" }] } }, "servicePrincipalProfile": { "clientId": "****", "secret": "****" } } }

[CecileRobertMichon]

Thanks. If you have time, can you please give https://github.com/Azure/aks-engine/releases/tag/v0.38.0 a try?

[mo-saeed]

Sure, let me try.

[mo-saeed]

@CecileRobertMichon Now it works with 0.38.0 👍 What was the change ?

[CecileRobertMichon]

@mo-saeed I'm not sure I still haven't been able to repro. I might just be that the expired key was regenerated by the Kata folks since you last tried. Or it could be that the base image used by v0.37.4 has the expired key but not the one used by v0.38.0. I still need to get to the bottom of this.

[CecileRobertMichon]

I still couldn't repro with 0.37.4 using the same apimodel. I'm going to assume this was fixed by the regenerated key.

opened #1587 as a secondary fix.

[PacoGuy]

Just FYI @CecileRobertMichon I just used v0.38.0 to launch an AvailabilitySet node to create a new image from it and the key has not been updated in the image:
*******vmss000000:~$ sudo apt-get update Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease Hit:2 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64 InRelease Hit:3 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64 InRelease Hit:4 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64 InRelease Ign:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04 InRelease Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease Get:6 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04 Release [682 B] Hit:8 http://azure.archive.ubuntu.com/ubuntu xenial InRelease Get:9 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB] Get:10 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB] Get:11 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04 Release.gpg [481 B] Err:11 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04 Release.gpg The following signatures were invalid: KEYEXPIRED 1562449942 Fetched 217 kB in 1s (206 kB/s) Reading package lists... Done W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04 Release: The following signatures were invalid: KEYEXPIRED 1562449942 W: Failed to fetch http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/Release.gpg The following signatures were invalid: KEYEXPIRED 1562449942 W: Some index files failed to download. They have been ignored, or old ones used instead.
To make matters worse, when I use that image to launch a VMSS the VMSS does not register with the master nodes despite the deployment showing a success state. I'll open a ticket for that now.

[PacoGuy]

For reference I ended up here because of the exact same issue as the original poster with v0.34.0 and 1.13.5 which had been previously working.
Also the log above is from:
vmss000000:~$ uname -a Linux k8s-agentpool1-*********-vmss000000 4.15.0-1050-azure #55-Ubuntu SMP Sat Jun 29 00:27:54 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux111

[tumluliu]

@CecileRobertMichon many thanks for your investigating. But I still got the same error from my side with aks-engine 0.38.0, k8s 1.15.0 and distro aks-ubuntu-18.04. The logs from /var/log/azure/cluster-provision.log:

Err:11 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release.gpg
  The following signatures were invalid: EXPKEYSIG CCC19FA1383BC8D1 home:clearcontainers OBS Project <home:[email protected]>
Fetched 1163 B in 1s (1366 B/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release: The following signatures were invalid: EXPKEYSIG CCC19FA1383BC8D1 home:clearcontainers OBS Project <home:[email protected]>
W: Failed to fetch http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/Release.gpg  The following signatures were invalid: EXPKEYSIG CCC19FA1383BC8D1 home:clearcontainers OBS Project <home:[email protected]>
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 10 -eq 10 ']'
+ return 1
+ exit 99

my region is also west-eu

[mo-saeed]

It seems it's really can't be reproduced all the time, sometimes this key works and sometimes not.

[davis-x]

Just started getting this too ubuntu 16.04 Version: v0.37.4

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release: The following signatures were invalid: KEYEXPIRED 1562449942
W: Failed to fetch http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/Release.gpg  The following signatures were invalid: KEYEXPIRED 1562449942
W: Some index files failed to download. They have been ignored, or old ones used instead.

[tumluliu]

[tumluliu]

I can confirm that 0.38.0 works with no such issue for me

[CecileRobertMichon]

The mirrors have not yet been updated: https://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/ still shows a different key from http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04/

According to sig-azure folks, the mirrors can take up to 24 hours to get updated.

[CecileRobertMichon]

We have patch coming in a few hours that will completely remove clear containers from the base image that will fix this issue. It will still be available as an option for download during deployment.

[CecileRobertMichon]

Fixed in v0.37.5 and v0.38.1.

[next-sergioboluda]

Hello @CecileRobertMichon

We still use

"aksEngineVersion": "v0.35.6-aks"

Did you think about the update for that version?

Thanks a lot, BR.-

[CecileRobertMichon]

If you are using AKS (looks like you are), the AKS Engine version used by AKS has been updated with the latest AKS release

[vijaygos]

Hi @CecileRobertMichon , we use aks-engine so bootstraping a new cluster can take sometime. Do you have any recommendations on how we can mitigate this issue instead of creating a new cluster for the time being ?

[CecileRobertMichon]

@vijaygos not sure I understand your comment... If you are running into this issue while creating a cluster, then I recommend retrying with a newer version of aks-engine. If you are running into this issue while scaling (or running apt operations on) an existing cluster, you will need to upgrade to aks-engine 0.38.1 or above. See https://github.com/Azure/aks-engine/blob/master/docs/topics/upgrade.md.

If you want to fix this manually on each VM, these commands will do the trick:

apt-get purge -y cc-runtime
apt-get autoremove -y
rm -f /etc/apt/sources.list.d/cc-runtime.list