Azure-sdk-for-python: msrest and msrestazure dependencies not installed

[freefood89]

Is it just me, or does this package not install dependencies?

1. installed via pip3 install azure
from azure.common.credentials import UserPassCredentials resulted in ImportError: No module named 'msrest'
2. installed msrest via pip3 install msrest
from azure.common.credentials import UserPassCredentials resulted in ImportError: No module named 'msrestazure'
3. installed msrest azure via pip3 install msrestazure
from azure.common.credentials import UserPassCredentials finally worked

[lmazuel]

Hi @freefood89 ,

I saw rare situations where msrest(azure) dependencies are not installed. I think it's related to this pip issue:
https://github.com/pypa/pip/issues/988

Because msrest(azure) in a sub-dependencies of an azure-common extension called autorest. But since some other package does not need the extension, it seems that if pip sees an azure-common dependency before it sees azure-common[autorest] dependency, it does not install the extension.

Thank you for the feedback, I will try to find a smart way to make this more predictable.

[elemantalcode]

I seem to have the same problem. I initially did: pip install azure==2.0.0rc5 and tried to run https://github.com/ansible/ansible/blob/devel/contrib/inventory/azure_rm.py but get:

[terraform@pc001 ~]$ ./azure_rm.py
The Azure python sdk is not installed (try 'pip install azure==2.0.0rc5') - No module named msrestazure.azure_exceptions

[terraform@pc001 ~]$ pip install azure==2.0.0rc5
Requirement already satisfied (use --upgrade to upgrade): azure==2.0.0rc5 in /usr/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): azure-storage==0.32.0 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-servicebus==0.20.2 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-servicemanagement-legacy==0.20.3 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-batch==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-graphrbac==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-nspkg in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): futures in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-common in /usr/lib/python2.7/site-packages (from azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-scheduler==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-compute==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-keyvault==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-notificationhubs==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-cdn==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-powerbiembedded==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-redis==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-commerce==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-cognitiveservices==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-resource==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-storage==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-authorization==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-batch==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-network==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-logic==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-web==0.30.0rc5 in /usr/lib/python2.7/site-packages (from azure-mgmt==0.30.0rc5->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in /usr/lib/python2.7/site-packages (from python-dateutil->azure-storage==0.32.0->azure==2.0.0rc5)
Requirement already satisfied (use --upgrade to upgrade): azure-mgmt-nspkg in /usr/lib/python2.7/site-packages (from azure-mgmt-scheduler==0.30.0rc5->azure-mgmt==0.30.0rc5->azure==2.0.0rc5)

[terraform@pc001 ~]$ pip install msrestazure
Requirement already satisfied (use --upgrade to upgrade): msrestazure in /usr/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): msrest>=0.4.0 in /usr/lib/python2.7/site-packages (from msrestazure)
Requirement already satisfied (use --upgrade to upgrade): chardet>=2.3.0 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): certifi>=2015.9.6.2 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): enum34>=1.0.4 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): keyring>=5.6 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): isodate>=0.5.4 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): requests>=2.7.0 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): requests-oauthlib>=0.5.0 in /usr/lib/python2.7/site-packages (from msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): secretstorage; sys_platform == "linux2" or sys_platform == "linux" in /usr/lib/python2.7/site-packages (from keyring>=5.6->msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): oauthlib>=0.6.2 in /usr/lib/python2.7/site-packages (from requests-oauthlib>=0.5.0->msrest>=0.4.0->msrestazure)
Requirement already satisfied (use --upgrade to upgrade): pycrypto in /usr/lib64/python2.7/site-packages (from secretstorage; sys_platform == "linux2" or sys_platform == "linux"->keyring>=5.6->msrest>=0.4.0->msrestazure)

mrestazure was not installed as a dependency during the initial install of pip install azure

Some versioning:
OS: Red Hat Enterprise Linux Server release 6.8 (Santiago)
PIP: pip 8.1.2 from /usr/lib/python2.7/site-packages (python 2.7)

Please note @lmazuel The above post is simply to showcase and inform that installing azure does not include all dependencies in specific situations as you've previously observed; the contrib script is out of scope for this section. (The fix was to point the contrib script to python 2.7 instead of 2.6).

[lmazuel]

Thanks for the feedback @elemantalcode !

[grindthemall]

Just ran into this same issue.

"pip install msrestazure" corrected the problem for me.

OS: Centos 7.2
PIP: 8.1.2
Python 2.7.5

[xingzhougmu]

@lmazuel
I install Azure Python SDK from the git source, but running into the same issue when "import azure.mgmt.redis"

I use the following commands to install SDK for Python3.5
git clone git://github.com/Azure/azure-sdk-for-python.git
cd azure-sdk-for-python
python3 setup.py install

My OS is: Ubuntu 16.04LTS

[trondhindenes]

This issue is now 6 months old, and there's still not a clean way to install the azure sdk for python. Any updates on this?

[squasta]

Hi, I have also problems with dependencies with Ansible (running Ubuntu 16.04 LTS) with Azure SDK for python. I tried everything described on this page. nothing works. Still get the message Try pip install azure==2.0.0rc5- No module named common.credentials

[lmazuel]

@trondhindenes I started to change the way I install msrestazure, replacing azure-common[autorest] that creates the problem by an explicit msrestazure dependency, see for instance this setup.py file. Each time I release a new package, I take this opportunity to update it. I thought it was too soon to answer here, but I didn't forget and eventually all packages will be fixed. This is NOT a fix in the azure meta-package, since as a meta-package it does not have a dependency on msrestazure.

@squasta if you have a message "no module named common.credentials", then you're not into this issue. This issue installs correctly azure.common.credentials BUT doesn't install the underlying package to make it work, so you got a msrestazure error that can be fixed installing manually msrestazure. If you still think you have the issue mentioned here, please provide a stackstrace. If not, please create a new issue with your specific stacktrace.

[trondhindenes]

Thanks for the update @lmazuel , good to know that the issue is being worked on.

[lmazuel]

@trondhindenes I pushed a commit and changed them all:
https://github.com/Azure/azure-sdk-for-python/commit/3dc2d4f8047ce5b8421572eda9278a5c1a72ea3c

GH has closed the issue automatically, I guess it's fair, but the fix will be published package per package only each time I publish new release. I will not publish new version of everything just for this. However, an update of azure-mgmt-compute and azure-mgmt-resource is planned for early February, so since they are the most used packages, problem will disappear quickly.

Please feel free to still comment here, even closed, and I re-open it if necessary.

[scmdavid]

I am using Mac (10.12.3) and python 2.7.13
I manually run
sudo pip install msrest (0.4.5)
sudo pip install msrestazure (0.4.7)
sudo pip install "azure==2.0.0rc5"

All succeed but I still got :
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Do you have azure==2.0.0rc5 installed? Try pip install azure==2.0.0rc5- No module named msrest.serialization"}

[rkumar357]

I have the same issue , any update on this ?

[lmazuel]

Hi @rkumar357
As I wrote in my latest message, new packages are now released with the fix. I will not release a new package just for that issue, but it will be fixed by all new package and eventually in every packages.
Thanks,

[rkumar357]

I have problem with Python 2.7.10 , Python 2.7.13 got succeeded.

[lmazuel]

It's a random issue, depending on the order pip process the dependencies. This is not related to Python version (unless Python is shipped with pip and pip is newer with 2.7.13, it's possible).
New packages released since 01/2017 does not have the issue. I cannot fix the old ones, where you might have to pip install msrestazure manually.
thanks,

[rkumar357]

Cheers @lmazuel , working fine , thanks for the help.

[ghost]

So I just installed azure az new installation on fedora 24, got the same error and cannot use module at all. Any advise ?

Same in docker ubuntu after : pip install "azure==2.0.0rc5"
python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from azure.common.credentials import UserPassCredentials
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named azure.common.credentials

[lmazuel]

Hi @ment0s
You don't have the same error, since the missing module is azure.common.credentials, not msrestazure. Could you provide a more detailled description of the messages you got from pip during the installation?

[ghost]

pip output : http://pastebin.com/RgQDNF0s

With ansible from git I get these errors :

"Do you have azure==2.0.0rc5 installed? Try pip install azure==2.0.0rc5- cannot import name integer_types"

pip install rc5 : http://pastebin.com/QNrsFk6Y

[rkumar357]

@ment0s , ansible using old version of python, just check in verbose mode, to execute any pthyon script ansible will use /usr/bin/python or something esle ?

/usr/bin/python --version
python --version
/usr/bin/lib/python --version

if you have done brew install then Cellar/bin/python --version

Compare which python is used while executing ansible python script, i have soft link 2.7 python library to /usr/bin/python, since ansible is using that location for me.

[mkesselaers]

I have the same problem with a fresh install on MacOS
TASK [Create a resource group] ************************************************* fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Do you have msrestazure installed? Trypip install msrestazure- No module named msrest.serialization"}

Python version is 2.7.13.
My pip list is the following (only relevant packages).
azure (1.0.3) azure-batch (0.30.0rc5) azure-common (1.1.4) azure-graphrbac (0.30.0rc5) azure-mgmt (0.20.2) azure-mgmt-authorization (0.30.0rc5) azure-mgmt-batch (0.30.0rc5) azure-mgmt-cdn (0.30.0rc5) azure-mgmt-cognitiveservices (0.30.0rc5) azure-mgmt-commerce (0.30.0rc5) azure-mgmt-common (0.20.0) azure-mgmt-compute (0.20.1) azure-mgmt-keyvault (0.30.0rc5) azure-mgmt-logic (0.30.0rc5) azure-mgmt-network (0.20.1) azure-mgmt-notificationhubs (0.30.0rc5) azure-mgmt-nspkg (2.0.0) azure-mgmt-powerbiembedded (0.30.0rc5) azure-mgmt-redis (0.30.0rc5) azure-mgmt-resource (0.20.1) azure-mgmt-scheduler (0.30.0rc5) azure-mgmt-storage (0.20.0) azure-mgmt-web (0.30.0rc5) azure-nspkg (2.0.0) azure-servicebus (0.20.1) azure-servicemanagement-legacy (0.20.2) azure-storage (0.20.3) msrest (0.4.7) msrestazure (0.4.7)

I tried uninstalling and re-installing azure, azure-mgmt, msrest and msrestazure, but this didn't work.
Any idea about what is wrong?

[lmazuel]

Hi @mkesselaers

Can you open a python interpreter and type "import msrest"?

Thanks,

[mkesselaers]

Hi @lmazuel,

This doesn't give any error, which seems like everything is fine.

Best regards,
Maarten

[lmazuel]

If you are able to import msrest from a Python interpreter but Ansible can't, might be a problem with Ansible (it's Ansible right?)

[mkesselaers]

Yes, you're right. But since the error that I've got was found on this thread, I thought that it would be best to post it here.
How did the other OPs solved it?

[rkumar357]

Try running,

ansible-playbook test.yml -e ansible_python_interpreter=/usr/local/opt/python/bin/python2.7

[mkesselaers]

Hi @rkumar357 ,

This works indeed.
I presume that Ansible now defaults to python 3 as interpreter?

Thanks for your help!
Best regards,
Maarten

[rkumar357]

Cheers !!! for ansible i belive python 2 is deafult.

[lmazuel]

@rkumar357 @mkesselaers Ansible does not support Python 3 at all for now (see on PyPI). So only option here is Python 2.7. I don't know how Ansible is defining his shebang, but if for some reason it starts with Py3 by default you will be in trouble.
I didn't know Ansible has a ansible_python_interpreter (thanks @rkumar357 :)). I will keep this in mind and that's indeed the correct solution.

[aaronjones]

I am running Ansible (2.2.2.0 or 2.3.0.0) inside a docker container and am receiving the same msrestazure.azure_exceptions error despite applying the various fixes noted above.

ansible-playbook -vvvv  -i /home/ansible/ansible/contrib/inventory/azure_rm.py generatevm.yml -e ansible_python_interpreter=/usr/local/opt/python/bin/python2.7
...
ERROR! Attempted to execute "/home/ansible/ansible/contrib/inventory/azure_rm.py" as inventory script: Inventory script (/home/ansible/ansible/contrib/inventory/azure_rm.py) had an execution error: The Azure python sdk is not installed (try `pip install 'azure>=2.0.0rc5' --upgrade`) - No module named msrestazure.azure_exceptions

pip show azure
Name: azure
Version: 2.0.0rc6
Summary: Microsoft Azure Client Libraries for Python
Home-page: https://github.com/Azure/azure-sdk-for-python
Author: Microsoft Corporation
Author-email: [email protected]
License: MIT License
Location: /usr/local/lib/python2.7/dist-packages

Looking at the azure_rm.py code, it looks like it is just failing upon import (I'm not a python dev... yet). From what I have read, providing the reference to the path where azure_rm.py lives should be sufficient for python to pick up the remaining libraries listed in the import... though I'm not certain.
I cloned https://github.com/Azure/azure-sdk-for-python.git on Friday, so that shouldn't be out of date, and I've tried both rc5 & rc6 pip installs of azure .
Any tips would be appreciated.

[lmazuel]

Hi @aaronjones

Could you try:

/usr/local/opt/python/bin/python2.7 -c "import msrestazure; print(msrestazure.__version__)"

[aaronjones]

slightly embarrassing, but delivers the same result and same problem. (I didn't realize python wasn't installed in that same location, so it must have defaulted to what was installed)
Using the following command:
/usr/bin/python2.7 -c "import msrestazure; print(msrestazure.__version__)"
results in:

Traceback (most recent call last):
File "", line 1, in
ImportError: No module named msrestazure

[lmazuel]

So, you can install msrestazure this way (assuming you have sudo access):
sudo /usr/bin/python2.7 -m pip install msrestazure

[aaronjones]

Sorry for the delay in response. It has at least moved me forward to a credentials error, so that is progress. I guess the ansible documentation or possibly the sdk readme may need updating to reflect the requirement of installing mrestazure separately?
https://docs.ansible.com/ansible/guide_azure.html
I'm guessing this is just an issue running ansible inside of docker given other positive responses above?

[lmazuel]

msrestazure is not supposed to be installed separately, but we got issues in the past with pip. We fixed that in recent packages, but likely you got it with old version :(
Installing the future azure 2.0.0rc7 should definitely fix the issue by forcing all packages to be in a recent version where the issue is fixed.

[dselans]

I just ran into this on OS X 10.11.6. Just in case someone else runs into this, my (seemingly) underlying issue was that I installed ansible via brew, which installs python 2.7.13 as a dep. Running pip install azure* via local python 2.7.10 ended with the ImportError: No module named 'msrestazure' issue.

The fix was to ensure that I'm using python 2.7.13 - I went ahead and updated my $PATH to include /usr/local/Cellar/python/2.7.13/bin. After this, running sudo pip install azure==2.0.0rc5 succeeded.

[redice]

My solution is a workable way to solve No module named msrest.serialization issue.
In OS X 10.13.6, run brew instabll ansible to install the latest ansbile package (my case is 2.6.2)
Ansible 2.6.2 has its own python 2.7.15 in _/usr/local/Cellar/ansible/2.6.2/libexec/bin_ directory

Need to use its pip script to install azure packages
/usr/local/Cellar/ansible/2.6.2/libexec/bin/pip install azure to fix the azure dependences.