Azure-cli: aks-preview extension breaks most az aks commands

Created on 13 Nov 2019  Â·  25Comments  Â·  Source: Azure/azure-cli

Describe the bug
aks-preview extension broke az aks get-credentials. This was working perfectly prior to Nov 13 2019.

To Reproduce

bash-4.2# az --version
azure-cli                         2.0.75

command-modules-nspkg              2.0.3
core                              2.0.75
nspkg                              3.0.4
telemetry                          1.0.4

Python location '/usr/bin/python'
Extensions directory '/home/jenkins/.azure/cliextensions'

Python (Linux) 2.7.5 (default, Jun 11 2019, 14:33:56)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

Legal docs and information: aka.ms/AzureCliLegal

bash-4.2# ls -l /home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py
ls: cannot access /home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py: No such file or directory

bash-4.2# az aks get-credentials -n aks-testing -g rsg-testing -a -f admin.kubeconfig
Merged "aks-testing" as current context in admin.kubeconfig

bash-4.2# az extension add --name aks-preview

bash-4.2# ls -l /home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py
-rw-r--r--. 1 root root 107080 Nov 13 07:17 /home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py

bash-4.2# az aks get-credentials -n aks-testing -g rsg-testing -a -f admin.kubeconfig
The command failed with an unexpected error. Here is the traceback:

invalid syntax (custom.py, line 1155)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/knack/cli.py", line 206, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 523, in execute
    self.commands_loader.load_arguments(command)
  File "/usr/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 291, in load_arguments
    self.command_table[command].load_arguments()  # this loads the arguments via reflection
  File "/usr/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 291, in load_arguments
    super(AzCliCommand, self).load_arguments()
  File "/usr/lib/python2.7/site-packages/knack/commands.py", line 97, in load_arguments
    cmd_args = self.arguments_loader()
  File "/usr/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 488, in default_arguments_loader
    op = handler or self.get_op_handler(operation, operation_group=kwargs.get('operation_group'))
  File "/usr/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 528, in get_op_handler
    op = import_module(mod_to_import)
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py", line 1155
    f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as '
    ^
SyntaxError: invalid syntax

To open an issue, please run: 'az feedback'

Expected behavior
az aks get-credentials will not fail

Environment summary
install method is via pip: pip install --no-cache-dir --upgrade azure-cli
running on a docker image with base openshift3/jenkins-slave-base-rhel7
tested in bash shell

Additional context
This was working perfectly prior to Nov 13 2019.

AKS Service Attention

Most helpful comment

dirty workaround: install an older version with something like:
az extension remove --name aks-preview
az extension add --source https://azurecliaks.blob.core.windows.net/azure-cli-extension/aks_preview-0.4.21-py2.py3-none-any.whl

All 25 comments

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/aks-pm

dirty workaround: install an older version with something like:
az extension remove --name aks-preview
az extension add --source https://azurecliaks.blob.core.windows.net/azure-cli-extension/aks_preview-0.4.21-py2.py3-none-any.whl

Seeing the same error on preview extension aks-preview-0.4.22

This is with az aks create on aks 0.14.8

ERROR: The command failed with an unexpected error. Here is the traceback:",
  ",
  ERROR: invalid syntax (custom.py, line 1155)",
  Traceback (most recent call last):",
    File \"/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py\", line 206, in invoke",
      cmd_result = self.invocation.execute(args)",
    File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 523, in execute",
      self.commands_loader.load_arguments(command)",
    File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py\", line 291, in load_arguments",
      self.command_table[command].load_arguments()  # this loads the arguments via reflection",
    File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 291, in load_arguments",
      super(AzCliCommand, self).load_arguments()",
    File \"/usr/lib64/az/lib/python2.7/site-packages/knack/commands.py\", line 97, in load_arguments",
      cmd_args = self.arguments_loader()",
    File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py\", line 488, in default_arguments_loader",
      op = handler or self.get_op_handler(operation, operation_group=kwargs.get('operation_group'))",
    File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py\", line 528, in get_op_handler",
      op = import_module(mod_to_import)",
    File \"/usr/lib64/python2.7/importlib/__init__.py\", line 37, in import_module",
      __import__(name)",
    File \"/home/deployer/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py\", line 1155",
      f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as '",
      ^",

The command is successful if we roll back to a previous version of the extension.

Thanks for the reports, I'm not able to reproduce with the same aks and extension builds but I do have python3 installed. @zikalino any insights into this error by chance?

This line:
f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as '",

is using an f-string which is valid syntax only in Python 3.6 and higher.

On Centos, if you install the azure cli via yum using this package: azure-cli-2.0.76-1.el7.x86_64
az will use Python 2.7

@jluk you need to run Python 2.7 to reproduce this bug.

@jluk see: https://github.com/Azure/azure-cli/issues/11234

Right now, if you do yum install azure-cli, the Azure cli is hardcoded to use Python 2.7

@marcpascual You can work around this problem by doing:

  1. Uninstall azure-cli
  2. Re-install azure-cli using pip3:
pip3 install azure-cli

I've also submitted this fix: https://github.com/Azure/azure-cli/pull/11235

az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
invalid syntax (custom.py, line 1158)
Traceback (most recent call last):
File "/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py", line 197, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 262, in execute
self.commands_loader.load_arguments(command)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 253, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 141, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/usr/lib64/az/lib/python2.7/site-packages/knack/commands.py", line 76, in load_arguments
cmd_args = self.arguments_loader()
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 444, in default_arguments_loader
op = handler or self.get_op_handler(operation)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 489, in get_op_handler
op = import_module(mod_to_import)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/jenkinsappuser/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py", line 1158
f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as'
^
SyntaxError: invalid syntax

Rotate certificates in Azure Kubernetes Service (AKS)
https://docs.microsoft.com/en-us/azure/aks/certificate-rotation

please help.

@sandeep1987 can you uninstall the azure-cli, and re-install azure-cli 2.0.77 which is built for Python 3?

https://docs.microsoft.com/en-us/cli/azure/release-notes-azure-cli?view=azure-cli-latest#november-26-2019

The extensions do not work with Python 2

@rodrigc I uninstall the azure-cli and install Python 3, still I am not able to install azure-cli, see blow log for more details, please help.

[jenkinsappuser@vmcicdjen02 ~]$ sudo yum remove azure-cli
[sudo] password for jenkinsappuser:
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package azure-cli.x86_64 0:2.0.50-1.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================

Package Arch Version Repository Size

Removing:
azure-cli x86_64 2.0.50-1.el7 @azure-cli 171 M

Transaction Summary

Remove 1 Package

Installed size: 171 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : azure-cli-2.0.50-1.el7.x86_64 1/1
Verifying : azure-cli-2.0.50-1.el7.x86_64 1/1

Removed:
azure-cli.x86_64 0:2.0.50-1.el7

Complete!
[jenkinsappuser@vmcicdjen02 ~]$ sudo rm /etc/yum.repos.d/azure-cli.repo

[jenkinsappuser@vmcicdjen02 ~]$ sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc

[jenkinsappuser@vmcicdjen02 ~]$ sudo sh -c 'echo -e "[azure-cli]

name=Azure CLI
baseurl=https://packages.microsoft.com/yumrepos/azure-cli
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/azure-cli.repo'

[jenkinsappuser@vmcicdjen02 ~]$ sudo yum install azure-cli
Loaded plugins: fastestmirror
azure-cli | 2.9 kB 00:00:00
Loading mirror speeds from cached hostfile

  • epel: mirror.vcu.edu
    Resolving Dependencies
    --> Running transaction check
    ---> Package azure-cli.x86_64 0:2.0.77-1.el7 will be installed
    --> Processing Dependency: python3 for package: azure-cli-2.0.77-1.el7.x86_64
    --> Finished Dependency Resolution
    _### Error: Package: azure-cli-2.0.77-1.el7.x86_64 (azure-cli)
    Requires: python3
    You could try using --skip-broken to work around the problem
    You could try running: rpm -Va --nofiles --nodigest_

[jenkinsappuser@vmcicdjen02 ~]$ sudo yum install azure-cli --skip-broken
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile

  • epel: mirror.vcu.edu
    Resolving Dependencies
    --> Running transaction check
    ---> Package azure-cli.x86_64 0:2.0.77-1.el7 will be installed
    --> Processing Dependency: python3 for package: azure-cli-2.0.77-1.el7.x86_64
    azure-cli/filelists_db | 6.2 MB 00:00:00

Packages skipped because of dependency problems:
azure-cli-2.0.77-1.el7.x86_64 from azure-cli

*****Python version*******
[jenkinsappuser@vmcicdjen02 ~]$ python3 --version
Python 3.6.8

  1. What version of Centos or RedHat Linux are you using? What is the content of /etc/os-release?

  2. Can you yum install python3

@rodrigc
[jenkinsappuser@vmcicdjen02 ~]$ lsb_release -d
Description: CentOS Linux release 7.4.1708 (Core)

[jenkinsappuser@vmcicdjen02 ~]$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[jenkinsappuser@vmcicdjen02 ~]$

[jenkinsappuser@vmcicdjen02 ~]$ sudo yum install python3
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile

Linux vmcicdjen02 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

@sandep1987 If you can update your Centos to something newer than Centos 7.4, especially Centos 7.7, yum install python3 will work.
There is a Python3 package here: http://mirror.centos.org/centos/7/os/x86_64/Packages/

@rodrigc I was trying to update, it didn't help …

[jenkinsappuser@vmcicdjen02 ~]$ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[jenkinsappuser@vmcicdjen02 ~]$ sudo su -
Last login: Sat Dec 14 14:04:35 UTC 2019 on pts/0
[root@vmcicdjen02 ~]# yum check-update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile

@rodrigc
[root@vmcicdjen02 ~]# yum clean all
Loaded plugins: fastestmirror
Cleaning repos: azure-cli base epel extras ius mysql-connectors-community mysql-tools-community mysql56-community openlogic
: updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors
[root@vmcicdjen02 ~]# rm -rf /var/cache/yum/*
[root@vmcicdjen02 ~]# yum clean all; yum makecache
Loaded plugins: fastestmirror
Cleaning repos: azure-cli base epel extras ius mysql-connectors-community mysql-tools-community mysql56-community openlogic
: updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Loaded plugins: fastestmirror
azure-cli | 2.9 kB 00:00:00
base | 3.6 kB 00:00:00
epel/x86_64/metalink | 17 kB 00:00:00
epel | 5.3 kB 00:00:00
extras | 2.9 kB 00:00:00
ius | 1.3 kB 00:00:00
mysql-connectors-community | 2.5 kB 00:00:00
mysql-tools-community | 2.5 kB 00:00:00
mysql56-community | 2.5 kB 00:00:00
openlogic | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/35): azure-cli/primary_db | 49 kB 00:00:00
(2/35): azure-cli/other_db | 4.1 kB 00:00:00
(3/35): azure-cli/filelists_db | 6.2 MB 00:00:00
(4/35): base/x86_64/group_gz | 156 kB 00:00:00
base/x86_64/primary_db FAILED
http://olcentgbl.trafficmanager.net/centos/7.4.1708/os/x86_64/repodata/b24800ea3136d17e328f2f06cb2169fc42ae3ed97bf8dd6ee86e503ce7ac4cc3-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.

base/x86_64/other_db FAILED
http://olcentgbl.trafficmanager.net/centos/7.4.1708/os/x86_64/repodata/cf0cc856d46b3095106da78256fb28f9d8defea4118d0e75eab07dc53b7d3f0d-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
(5/35): epel/x86_64/group_gz | 90 kB 00:00:00
(6/35): base/x86_64/filelists_db | 6.7 MB 00:00:00
(7/35): epel/x86_64/updateinfo | 1.0 MB 00:00:00
(8/35): epel/x86_64/prestodelta | 568 B 00:00:00
(9/35): epel/x86_64/primary_db | 6.9 MB 00:00:01
(10/35): epel/x86_64/other_db | 3.3 MB 00:00:00
(11/35): extras/x86_64/primary_db | 185 kB 00:00:00
(12/35): epel/x86_64/updateinfo_zck | 1.5 MB 00:00:00
(13/35): extras/x86_64/other_db | 121 kB 00:00:00
(14/35): extras/x86_64/filelists_db | 709 kB 00:00:00
(15/35): ius/x86_64/filelists | 757 kB 00:00:00
(16/35): ius/x86_64/primary | 149 kB 00:00:00
(17/35): ius/x86_64/other | 75 kB 00:00:00
(18/35): mysql-connectors-community/x86_64/primary_db | 49 kB 00:00:00
(19/35): mysql-connectors-community/x86_64/filelists_db | 66 kB 00:00:00
(20/35): mysql-connectors-community/x86_64/other_db | 13 kB 00:00:00
(21/35): mysql-tools-community/x86_64/primary_db | 66 kB 00:00:00
(22/35): mysql-tools-community/x86_64/filelists_db | 192 kB 00:00:00
(23/35): mysql-tools-community/x86_64/other_db | 15 kB 00:00:00
(24/35): mysql56-community/x86_64/primary_db | 241 kB 00:00:00
(25/35): openlogic/x86_64/filelists_db | 33 kB 00:00:00
(26/35): mysql56-community/x86_64/other_db | 63 kB 00:00:00
(27/35): mysql56-community/x86_64/filelists_db | 783 kB 00:00:00
(28/35): openlogic/x86_64/other_db | 12 kB 00:00:00
(29/35): updates/x86_64/primary_db | 6.9 MB 00:00:00
(30/35): openlogic/x86_64/primary_db | 81 kB 00:00:00
(31/35): updates/x86_64/other_db | 734 kB 00:00:00
(32/35): updates/x86_64/filelists_db | 4.2 MB 00:00:00
epel/x86_64/filelists_db FAILED
http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/a20678c3bc61d0823f5484cb410a3df23e57e2cf55e8b34e4fdd47424f9f570b-filelists.sqlite.bz2: [Errno 12] Timeout on http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/a20678c3bc61d0823f5484cb410a3df23e57e2cf55e8b34e4fdd47424f9f570b-filelists.sqlite.bz2: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
(33/35): epel/x86_64/filelists_db | 12 MB 00:00:00
Determining fastest mirrors

One of the configured repositories failed (CentOS-7 - Base),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:

 1. Contact the upstream for the repository and get them to fix the problem.

 2. Reconfigure the baseurl/etc. for the repository, to point to a working
    upstream. This is most often useful if you are using a newer
    distribution release than is supported by the repository (and the
    packages for the previous distribution release still work).

 3. Run the command with the repository temporarily disabled
        yum --disablerepo=base ...

 4. Disable the repository permanently, so yum won't use it by default. Yum
    will then just ignore the repository until you permanently enable it
    again or use --enablerepo for temporary usage:

        yum-config-manager --disable base
    or
        subscription-manager repos --disable=base

 5. Configure the failing repository to be skipped, if it is unavailable.
    Note that yum will try to contact the repo. when it runs most commands,
    so will have to try and fail each time (and thus. yum will be be much
    slower). If it is a very temporary problem though, this is often a nice
    compromise:

        yum-config-manager --save --setopt=base.skip_if_unavailable=true

failure: repodata/b24800ea3136d17e328f2f06cb2169fc42ae3ed97bf8dd6ee86e503ce7ac4cc3-primary.sqlite.bz2 from base: [Errno 256] No more mirrors to try.
http://olcentgbl.trafficmanager.net/centos/7.4.1708/os/x86_64/repodata/b24800ea3136d17e328f2f06cb2169fc42ae3ed97bf8dd6ee86e503ce7ac4cc3-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
[root@vmcicdjen02 ~]#

@rodrigc
[root@vmcicdjen02 ~]# ll /etc/yum.repos.d
total 60
-rw-r--r-- 1 root root 163 Dec 13 21:17 azure-cli.repo
-rw-r--r--. 1 root root 1243 Jan 12 2018 CentOS-Base.repo
-rw-r--r--. 1 root root 1309 Aug 30 2017 CentOS-CR.repo
-rw-r--r--. 1 root root 649 Aug 30 2017 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root 314 Aug 30 2017 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 630 Aug 30 2017 CentOS-Media.repo
-rw-r--r--. 1 root root 1331 Aug 30 2017 CentOS-Sources.repo
-rw-r--r--. 1 root root 3830 Aug 30 2017 CentOS-Vault.repo
-rw-r--r-- 1 root root 1050 Sep 17 23:25 epel.repo
-rw-r--r-- 1 root root 1149 Sep 17 23:25 epel-testing.repo
-rw-r--r-- 1 root root 669 May 1 2019 ius-archive.repo
-rw-r--r-- 1 root root 591 May 1 2019 ius.repo
-rw-r--r-- 1 root root 669 May 1 2019 ius-testing.repo
-rw-r--r-- 1 root root 1209 Jan 29 2014 mysql-community.repo
-rw-r--r-- 1 root root 1060 Jan 29 2014 mysql-community-source.repo

@sandep1987 you might need to update your Centos-Base.repo file. See: https://unix.stackexchange.com/questions/210448/centos-7-update-using-yum-update-does-not-work

@rodrigc let me try. thank you for your help ….

@rodrigc thanks a bunch for the troubleshooting and the PR! Your help is much appreciated, @sandep1987 please let us know if your issue is resolved with the provided advice, otherwise I'll close this issue soon in favor of latest CLI builds and using python3.

@jluk I also have a PR to drop Python 2.7 builds for azure-cli: https://github.com/Azure/azure-cli/pull/11363 if you want to help review that

I think this should be done to avoid the maintenance burden of running on Python 2

@rodrigc it worked now, thank you so much for your help.
@jluk thank you for follow-up.

This line:
f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as '",

is using an f-string which is valid syntax only in Python 3.6 and higher.

On Centos, if you install the azure cli via yum using this package: azure-cli-2.0.76-1.el7.x86_64
az will use Python 2.7

Just to say, we had the same issue with python 3.5 so this comment helped a lot. Upgrading to python 3.6 solved the issue.

@sandep1987 glad you solved it!

Was this page helpful?
0 / 5 - 0 ratings