Describe the bug
aks-preview extension broke az aks get-credentials. This was working perfectly prior to Nov 13 2019.
To Reproduce
bash-4.2# az --version
azure-cli 2.0.75
command-modules-nspkg 2.0.3
core 2.0.75
nspkg 3.0.4
telemetry 1.0.4
Python location '/usr/bin/python'
Extensions directory '/home/jenkins/.azure/cliextensions'
Python (Linux) 2.7.5 (default, Jun 11 2019, 14:33:56)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
Legal docs and information: aka.ms/AzureCliLegal
bash-4.2# ls -l /home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py
ls: cannot access /home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py: No such file or directory
bash-4.2# az aks get-credentials -n aks-testing -g rsg-testing -a -f admin.kubeconfig
Merged "aks-testing" as current context in admin.kubeconfig
bash-4.2# az extension add --name aks-preview
bash-4.2# ls -l /home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py
-rw-r--r--. 1 root root 107080 Nov 13 07:17 /home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py
bash-4.2# az aks get-credentials -n aks-testing -g rsg-testing -a -f admin.kubeconfig
The command failed with an unexpected error. Here is the traceback:
invalid syntax (custom.py, line 1155)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 523, in execute
self.commands_loader.load_arguments(command)
File "/usr/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 291, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/usr/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 291, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/usr/lib/python2.7/site-packages/knack/commands.py", line 97, in load_arguments
cmd_args = self.arguments_loader()
File "/usr/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 488, in default_arguments_loader
op = handler or self.get_op_handler(operation, operation_group=kwargs.get('operation_group'))
File "/usr/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 528, in get_op_handler
op = import_module(mod_to_import)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/jenkins/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py", line 1155
f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as '
^
SyntaxError: invalid syntax
To open an issue, please run: 'az feedback'
Expected behavior
az aks get-credentials will not fail
Environment summary
install method is via pip: pip install --no-cache-dir --upgrade azure-cli
running on a docker image with base openshift3/jenkins-slave-base-rhel7
tested in bash shell
Additional context
This was working perfectly prior to Nov 13 2019.
Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/aks-pm
dirty workaround: install an older version with something like:
az extension remove --name aks-preview
az extension add --source https://azurecliaks.blob.core.windows.net/azure-cli-extension/aks_preview-0.4.21-py2.py3-none-any.whl
Seeing the same error on preview extension aks-preview-0.4.22
This is with az aks create on aks 0.14.8
ERROR: The command failed with an unexpected error. Here is the traceback:",
",
ERROR: invalid syntax (custom.py, line 1155)",
Traceback (most recent call last):",
File \"/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py\", line 206, in invoke",
cmd_result = self.invocation.execute(args)",
File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 523, in execute",
self.commands_loader.load_arguments(command)",
File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py\", line 291, in load_arguments",
self.command_table[command].load_arguments() # this loads the arguments via reflection",
File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py\", line 291, in load_arguments",
super(AzCliCommand, self).load_arguments()",
File \"/usr/lib64/az/lib/python2.7/site-packages/knack/commands.py\", line 97, in load_arguments",
cmd_args = self.arguments_loader()",
File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py\", line 488, in default_arguments_loader",
op = handler or self.get_op_handler(operation, operation_group=kwargs.get('operation_group'))",
File \"/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py\", line 528, in get_op_handler",
op = import_module(mod_to_import)",
File \"/usr/lib64/python2.7/importlib/__init__.py\", line 37, in import_module",
__import__(name)",
File \"/home/deployer/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py\", line 1155",
f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as '",
^",
The command is successful if we roll back to a previous version of the extension.
Thanks for the reports, I'm not able to reproduce with the same aks and extension builds but I do have python3 installed. @zikalino any insights into this error by chance?
This line:
f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as '",
is using an f-string which is valid syntax only in Python 3.6 and higher.
On Centos, if you install the azure cli via yum using this package: azure-cli-2.0.76-1.el7.x86_64
az will use Python 2.7
@jluk you need to run Python 2.7 to reproduce this bug.
@jluk see: https://github.com/Azure/azure-cli/issues/11234
Right now, if you do yum install azure-cli, the Azure cli is hardcoded to use Python 2.7
@marcpascual You can work around this problem by doing:
pip3 install azure-cli
I've also submitted this fix: https://github.com/Azure/azure-cli/pull/11235
az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
invalid syntax (custom.py, line 1158)
Traceback (most recent call last):
File "/usr/lib64/az/lib/python2.7/site-packages/knack/cli.py", line 197, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 262, in execute
self.commands_loader.load_arguments(command)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 253, in load_arguments
self.command_table[command].load_arguments() # this loads the arguments via reflection
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/commands/__init__.py", line 141, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/usr/lib64/az/lib/python2.7/site-packages/knack/commands.py", line 76, in load_arguments
cmd_args = self.arguments_loader()
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 444, in default_arguments_loader
op = handler or self.get_op_handler(operation)
File "/usr/lib64/az/lib/python2.7/site-packages/azure/cli/core/__init__.py", line 489, in get_op_handler
op = import_module(mod_to_import)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/jenkinsappuser/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py", line 1158
f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as'
^
SyntaxError: invalid syntax
Rotate certificates in Azure Kubernetes Service (AKS)
https://docs.microsoft.com/en-us/azure/aks/certificate-rotation
please help.
@sandeep1987 can you uninstall the azure-cli, and re-install azure-cli 2.0.77 which is built for Python 3?
https://docs.microsoft.com/en-us/cli/azure/release-notes-azure-cli?view=azure-cli-latest#november-26-2019
The extensions do not work with Python 2
@rodrigc I uninstall the azure-cli and install Python 3, still I am not able to install azure-cli, see blow log for more details, please help.
[jenkinsappuser@vmcicdjen02 ~]$ sudo yum remove azure-cli
[sudo] password for jenkinsappuser:
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package azure-cli.x86_64 0:2.0.50-1.el7 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================================================================
Removing:
azure-cli x86_64 2.0.50-1.el7 @azure-cli 171 M
Remove 1 Package
Installed size: 171 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : azure-cli-2.0.50-1.el7.x86_64 1/1
Verifying : azure-cli-2.0.50-1.el7.x86_64 1/1
Removed:
azure-cli.x86_64 0:2.0.50-1.el7
Complete!
[jenkinsappuser@vmcicdjen02 ~]$ sudo rm /etc/yum.repos.d/azure-cli.repo
[jenkinsappuser@vmcicdjen02 ~]$ sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
[jenkinsappuser@vmcicdjen02 ~]$ sudo sh -c 'echo -e "[azure-cli]
name=Azure CLI
baseurl=https://packages.microsoft.com/yumrepos/azure-cli
enabled=1
gpgcheck=1
gpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/azure-cli.repo'
[jenkinsappuser@vmcicdjen02 ~]$ sudo yum install azure-cli
Loaded plugins: fastestmirror
azure-cli | 2.9 kB 00:00:00
Loading mirror speeds from cached hostfile
[jenkinsappuser@vmcicdjen02 ~]$ sudo yum install azure-cli --skip-broken
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Packages skipped because of dependency problems:
azure-cli-2.0.77-1.el7.x86_64 from azure-cli
*****Python version*******
[jenkinsappuser@vmcicdjen02 ~]$ python3 --version
Python 3.6.8
What version of Centos or RedHat Linux are you using? What is the content of /etc/os-release?
Can you yum install python3
@rodrigc
[jenkinsappuser@vmcicdjen02 ~]$ lsb_release -d
Description: CentOS Linux release 7.4.1708 (Core)
[jenkinsappuser@vmcicdjen02 ~]$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
[jenkinsappuser@vmcicdjen02 ~]$
[jenkinsappuser@vmcicdjen02 ~]$ sudo yum install python3
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Linux vmcicdjen02 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
@sandep1987 If you can update your Centos to something newer than Centos 7.4, especially Centos 7.7, yum install python3 will work.
There is a Python3 package here: http://mirror.centos.org/centos/7/os/x86_64/Packages/
@rodrigc I was trying to update, it didn't help …
[jenkinsappuser@vmcicdjen02 ~]$ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[jenkinsappuser@vmcicdjen02 ~]$ sudo su -
Last login: Sat Dec 14 14:04:35 UTC 2019 on pts/0
[root@vmcicdjen02 ~]# yum check-update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
@rodrigc
[root@vmcicdjen02 ~]# yum clean all
Loaded plugins: fastestmirror
Cleaning repos: azure-cli base epel extras ius mysql-connectors-community mysql-tools-community mysql56-community openlogic
: updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors
[root@vmcicdjen02 ~]# rm -rf /var/cache/yum/*
[root@vmcicdjen02 ~]# yum clean all; yum makecache
Loaded plugins: fastestmirror
Cleaning repos: azure-cli base epel extras ius mysql-connectors-community mysql-tools-community mysql56-community openlogic
: updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Loaded plugins: fastestmirror
azure-cli | 2.9 kB 00:00:00
base | 3.6 kB 00:00:00
epel/x86_64/metalink | 17 kB 00:00:00
epel | 5.3 kB 00:00:00
extras | 2.9 kB 00:00:00
ius | 1.3 kB 00:00:00
mysql-connectors-community | 2.5 kB 00:00:00
mysql-tools-community | 2.5 kB 00:00:00
mysql56-community | 2.5 kB 00:00:00
openlogic | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/35): azure-cli/primary_db | 49 kB 00:00:00
(2/35): azure-cli/other_db | 4.1 kB 00:00:00
(3/35): azure-cli/filelists_db | 6.2 MB 00:00:00
(4/35): base/x86_64/group_gz | 156 kB 00:00:00
base/x86_64/primary_db FAILED
http://olcentgbl.trafficmanager.net/centos/7.4.1708/os/x86_64/repodata/b24800ea3136d17e328f2f06cb2169fc42ae3ed97bf8dd6ee86e503ce7ac4cc3-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below wiki article
https://wiki.centos.org/yum-errors
If above article doesn't help to resolve this issue please use https://bugs.centos.org/.
base/x86_64/other_db FAILED
http://olcentgbl.trafficmanager.net/centos/7.4.1708/os/x86_64/repodata/cf0cc856d46b3095106da78256fb28f9d8defea4118d0e75eab07dc53b7d3f0d-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
(5/35): epel/x86_64/group_gz | 90 kB 00:00:00
(6/35): base/x86_64/filelists_db | 6.7 MB 00:00:00
(7/35): epel/x86_64/updateinfo | 1.0 MB 00:00:00
(8/35): epel/x86_64/prestodelta | 568 B 00:00:00
(9/35): epel/x86_64/primary_db | 6.9 MB 00:00:01
(10/35): epel/x86_64/other_db | 3.3 MB 00:00:00
(11/35): extras/x86_64/primary_db | 185 kB 00:00:00
(12/35): epel/x86_64/updateinfo_zck | 1.5 MB 00:00:00
(13/35): extras/x86_64/other_db | 121 kB 00:00:00
(14/35): extras/x86_64/filelists_db | 709 kB 00:00:00
(15/35): ius/x86_64/filelists | 757 kB 00:00:00
(16/35): ius/x86_64/primary | 149 kB 00:00:00
(17/35): ius/x86_64/other | 75 kB 00:00:00
(18/35): mysql-connectors-community/x86_64/primary_db | 49 kB 00:00:00
(19/35): mysql-connectors-community/x86_64/filelists_db | 66 kB 00:00:00
(20/35): mysql-connectors-community/x86_64/other_db | 13 kB 00:00:00
(21/35): mysql-tools-community/x86_64/primary_db | 66 kB 00:00:00
(22/35): mysql-tools-community/x86_64/filelists_db | 192 kB 00:00:00
(23/35): mysql-tools-community/x86_64/other_db | 15 kB 00:00:00
(24/35): mysql56-community/x86_64/primary_db | 241 kB 00:00:00
(25/35): openlogic/x86_64/filelists_db | 33 kB 00:00:00
(26/35): mysql56-community/x86_64/other_db | 63 kB 00:00:00
(27/35): mysql56-community/x86_64/filelists_db | 783 kB 00:00:00
(28/35): openlogic/x86_64/other_db | 12 kB 00:00:00
(29/35): updates/x86_64/primary_db | 6.9 MB 00:00:00
(30/35): openlogic/x86_64/primary_db | 81 kB 00:00:00
(31/35): updates/x86_64/other_db | 734 kB 00:00:00
(32/35): updates/x86_64/filelists_db | 4.2 MB 00:00:00
epel/x86_64/filelists_db FAILED
http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/a20678c3bc61d0823f5484cb410a3df23e57e2cf55e8b34e4fdd47424f9f570b-filelists.sqlite.bz2: [Errno 12] Timeout on http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/a20678c3bc61d0823f5484cb410a3df23e57e2cf55e8b34e4fdd47424f9f570b-filelists.sqlite.bz2: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
(33/35): epel/x86_64/filelists_db | 12 MB 00:00:00
Determining fastest mirrors
One of the configured repositories failed (CentOS-7 - Base),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=base ...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable base
or
subscription-manager repos --disable=base
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=base.skip_if_unavailable=true
failure: repodata/b24800ea3136d17e328f2f06cb2169fc42ae3ed97bf8dd6ee86e503ce7ac4cc3-primary.sqlite.bz2 from base: [Errno 256] No more mirrors to try.
http://olcentgbl.trafficmanager.net/centos/7.4.1708/os/x86_64/repodata/b24800ea3136d17e328f2f06cb2169fc42ae3ed97bf8dd6ee86e503ce7ac4cc3-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
[root@vmcicdjen02 ~]#
@rodrigc
[root@vmcicdjen02 ~]# ll /etc/yum.repos.d
total 60
-rw-r--r-- 1 root root 163 Dec 13 21:17 azure-cli.repo
-rw-r--r--. 1 root root 1243 Jan 12 2018 CentOS-Base.repo
-rw-r--r--. 1 root root 1309 Aug 30 2017 CentOS-CR.repo
-rw-r--r--. 1 root root 649 Aug 30 2017 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root 314 Aug 30 2017 CentOS-fasttrack.repo
-rw-r--r--. 1 root root 630 Aug 30 2017 CentOS-Media.repo
-rw-r--r--. 1 root root 1331 Aug 30 2017 CentOS-Sources.repo
-rw-r--r--. 1 root root 3830 Aug 30 2017 CentOS-Vault.repo
-rw-r--r-- 1 root root 1050 Sep 17 23:25 epel.repo
-rw-r--r-- 1 root root 1149 Sep 17 23:25 epel-testing.repo
-rw-r--r-- 1 root root 669 May 1 2019 ius-archive.repo
-rw-r--r-- 1 root root 591 May 1 2019 ius.repo
-rw-r--r-- 1 root root 669 May 1 2019 ius-testing.repo
-rw-r--r-- 1 root root 1209 Jan 29 2014 mysql-community.repo
-rw-r--r-- 1 root root 1060 Jan 29 2014 mysql-community-source.repo
@sandep1987 you might need to update your Centos-Base.repo file. See: https://unix.stackexchange.com/questions/210448/centos-7-update-using-yum-update-does-not-work
@rodrigc let me try. thank you for your help ….
@rodrigc thanks a bunch for the troubleshooting and the PR! Your help is much appreciated, @sandep1987 please let us know if your issue is resolved with the provided advice, otherwise I'll close this issue soon in favor of latest CLI builds and using python3.
@jluk I also have a PR to drop Python 2.7 builds for azure-cli: https://github.com/Azure/azure-cli/pull/11363 if you want to help review that
I think this should be done to avoid the maintenance burden of running on Python 2
@rodrigc it worked now, thank you so much for your help.
@jluk thank you for follow-up.
This line:
f'save them to the storage account {colorama.Style.BRIGHT}{colorama.Fore.GREEN}{storage_account_name}{colorama.Style.RESET_ALL} as '",is using an f-string which is valid syntax only in Python 3.6 and higher.
On Centos, if you install the azure cli via yum using this package:
azure-cli-2.0.76-1.el7.x86_64
azwill use Python 2.7
Just to say, we had the same issue with python 3.5 so this comment helped a lot. Upgrading to python 3.6 solved the issue.
@sandep1987 glad you solved it!
Most helpful comment
dirty workaround: install an older version with something like:
az extension remove --name aks-previewaz extension add --source https://azurecliaks.blob.core.windows.net/azure-cli-extension/aks_preview-0.4.21-py2.py3-none-any.whl