Azure-cli: Packages conflict for azure.storage.blob

[daxianji007]

Describe the bug
I installed the code in azure-cli repo. The code dependent on package azure-storage-blob and it is installed. But I cannot import azure.storage.blob in python.

To Reproduce

1. Clone the repo
2. run python scripts/dev_setup.py
3. run pip show azure-storage-blob, this package is installed.
4. run python -c "import azure.storage.blob", get error ModuleNotFoundError: No module named 'azure.storage'

From my investigation, it might be introduced by https://github.com/Azure/azure-cli/pull/8855

Expected behavior
azure.storage.blob can be imported.

Environment summary
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32

Additional context
This is blocking the Docs CI, please take this issue as high priority.

[tjprescott]

@daxianji007 why do you think it was introduced by #8855?

[tjprescott]

I am not able to reproduce this. I followed your directions and do not get the ModuleNotFoundError.

[tjprescott]

You might try the following:
pip install --upgrade azure.storage.blob

[tjprescott]

As I'm not able to repro this, and @dendeli-msft indicated the issue has been fixed on the docs side, I'm going to close this.

[daxianji007]

@tjprescott I find this PR because I run Docs CI locally for specified commit. The CI passed at commit 35f9aec6684fe36b1a08297e6b64b7a5b0f7f5b7 and failed at commit 9e360fc664c8acd0a1f1a9246fd58ffe5238acb5. The only difference between the two commits is #8855.

I have deeper investigation today, this could be related to pip version change in #8855. I think it is the reason why you did not repro it.

The difference comes from this command. When force-reinstall azure-nspkg==1.0.0 and azure-mgmt-nspkg==1.0.0 with pip==9.0.3 . It shows:

Successfully installed azure-nspkg-1.0.0
Successfully installed azure-mgmt-nspkg-1.0.0

When When force-reinstall with pip==19.0.3. It shows:

Successfully installed azure-nspkg-1.0.0
Successfully installed azure-mgmt-nspkg-1.0.0 azure-nspkg-3.0.2

Then the packages might be conflicted and I cannot import azure.storage.blob. If I manually force-reinstall azure-nspkg==1.0.0. This issue is gone.

The issue is temporarily mitigated but Docs CI keeps blocked. New changes will not be reflected in documents. It is possible to set the pip version to "pip~=9.0.1" like this Pull Request #9068?

[tjprescott]

@lmazuel do you know why azure.storage.blob would require azure-nspkg-1.0.0 and not work with azure-nspkg-3.0.2?

[tjprescott]

@daxianji007 I still cannot repro this. When I do a pip list in my environment, I show pip 19.0.3 and azure-nspkg 3.0.2. Yet all of our storage commands and tests run fine and I can run python -c "import azure.storage.blob" without issue.

[daxianji007]

@tjprescott Now I am confused. Could the virtualenv impact something? Here is the full script I/DocsCI use:

git clone https://github.com/Azure/azure-cli
cd azure-cli
virtualenv --no-site-packages venv
.\venv\Scripts\activate
python scripts/dev_setup.py
python -m pip install sphinx==1.5.6
cd doc\sphinx
make xmlwithversion

Get error ModuleNotFoundError: No module named 'azure.storage'

I tried pip install --upgrade azure.storage.blob it does not work. I will try to add pip install --force-reinstall azure-nspkg==1.0.0 before make xmlwithversion in Docs CI to work around. Do you have any better idea?

[tjprescott]

Is your Docs CI running python 2 or 3? It looks like Python 2 (because you are using vitualenv instead of venv). You should really use Python 3.

[daxianji007]

The Docs CI uses python 3, I am using python 3 too. I tried to use venv but got the same result....

[stevengum]

I ran into a similar problem (#9073) with the namespace package azure.mgmt. Installing pip at 9.0.1 has everything working again.

[tjprescott]

@stevengum just installing pip fixed your issue?

[tjprescott]

@daxianji007 the dev_setup.py script is no longer the recommended way to set up a dev environment. Can you try the following:

# create and activate your virtual environment
pip install azdev
azdev setup -c

[stevengum]

@tjprescott, I should clarify, I used pip at 9.0.1 and then ran the dev_setup.py script again. This solution worked.

@daxianji007 I followed Travis' instructions and these worked for me in a new environment.

[tjprescott]

Thanks @stevengum. @daxianji007 please try using azdev in your script. For what it is worth, we plan to migrate the doc generation script to azdev as well in the future.

[daxianji007]

@tjprescott Thanks for the solution! Now I can finnally go through the Docs CI. And I also tested that the generated document is almost the same with before.

[yonzhan]

Close it since using azdev unblocked you.