Azure-cli: Impossible to install on without conflicts

[wvxvw]

Describe the bug
When I try to install azure-cli and azure.batch together, no matter Python version, whether it is using pip or easy_install, tried on Debian and Arch Linux, there is never a version combination of these two packages that will not generate errors. For example, if you run this:

pip install azure-cli azure.batch

After a huge wall of diagnostic messages you will see this:

azure 4.0.0 has requirement azure-batch~=4.1, but you'll have azure-batch 5.1.1 which is incompatible.
azure-cli-batchai 0.4.3 has requirement azure-mgmt-storage==2.0.0rc4, but you'll have azure-mgmt-storage 2.0.0 which is incompatible.
azure-cli-batchai 0.4.3 has requirement azure-storage-blob==1.1.0, but you'll have azure-storage-blob 1.3.1 which is incompatible.
azure-cli-servicebus 0.3.0 has requirement azure-mgmt-servicebus==0.5.1, but you'll have azure-mgmt-servicebus 0.5.2 which is incompatible.
azure-cli-dms 0.1.1 has requirement azure-mgmt-datamigration==0.1.0, but you'll have azure-mgmt-datamigration 1.0.0 which is incompatible.
azure-cli-appservice 0.2.5 has requirement azure-mgmt-containerregistry==2.2.0, but you'll have azure-mgmt-containerregistry 2.3.0 which is incompatible.
azure-cli-appservice 0.2.5 has requirement azure-mgmt-storage==2.0.0rc4, but you'll have azure-mgmt-storage 2.0.0 which is incompatible.
azure-cli-appservice 0.2.5 has requirement azure-mgmt-web==0.40.0, but you'll have azure-mgmt-web 0.35.0 which is incompatible.
azure-cli-backup 1.2.1 has requirement azure-mgmt-recoveryservices==0.1.0, but you'll have azure-mgmt-recoveryservices 0.3.0 which is incompatible.
azure-cli-backup 1.2.1 has requirement azure-mgmt-recoveryservicesbackup==0.1.1, but you'll have azure-mgmt-recoveryservicesbackup 0.3.0 which is incompatible.
azure-cli-storage 2.2.2 has requirement azure-mgmt-storage==2.0.0rc4, but you'll have azure-mgmt-storage 2.0.0 which is incompatible.
azure-cli-cosmosdb 0.2.1 has requirement azure-mgmt-cosmosdb==0.4.0, but you'll have azure-mgmt-cosmosdb 0.4.1 which is incompatible.
azure-cli-sql 2.1.4 has requirement azure-mgmt-storage==2.0.0rc4, but you'll have azure-mgmt-storage 2.0.0 which is incompatible.
azure-cli-acr 2.1.6 has requirement azure-mgmt-containerregistry==2.2.0, but you'll have azure-mgmt-containerregistry 2.3.0 which is incompatible.
azure-cli-acr 2.1.6 has requirement azure-mgmt-storage==2.0.0rc4, but you'll have azure-mgmt-storage 2.0.0 which is incompatible.
azure-cli-acr 2.1.6 has requirement azure-storage-blob==1.1.0, but you'll have azure-storage-blob 1.3.1 which is incompatible.
azure-cli-batch 3.4.0 has requirement azure-batch==5.0.0, but you'll have azure-batch 5.1.1 which is incompatible.
azure-cli-iotcentral 0.1.2 has requirement azure-mgmt-iotcentral==0.2.0, but you'll have azure-mgmt-iotcentral 0.1.0 which is incompatible.
azure-cli-container 0.3.5 has requirement azure-mgmt-containerinstance==1.1.0, but you'll have azure-mgmt-containerinstance 1.2.1 which is incompatible.
azure-cli-eventgrid 0.2.0 has requirement azure-mgmt-eventgrid==0.4.0, but you'll have azure-mgmt-eventgrid 1.0.0 which is incompatible.
azure-cli-rdbms 0.3.2 has requirement azure-mgmt-rdbms==1.3.0, but you'll have azure-mgmt-rdbms 1.4.1 which is incompatible.
azure-cli-reservations 0.4.0 has requirement azure-mgmt-reservations==0.3.0, but you'll have azure-mgmt-reservations 0.2.1 which is incompatible.
azure-cli-dls 0.1.3 has requirement azure-datalake-store==0.0.31, but you'll have azure-datalake-store 0.0.34 which is incompatible.
azure-cli-ams 0.2.3 has requirement azure-mgmt-media==1.0.0rc1, but you'll have azure-mgmt-media 1.0.0 which is incompatible.
azure-cli-servicefabric 0.1.5 has requirement azure-mgmt-storage==2.0.0rc4, but you'll have azure-mgmt-storage 2.0.0 which is incompatible.
azure-cli-botservice 0.1.1 has requirement azure-mgmt-web==0.40.0, but you'll have azure-mgmt-web 0.35.0 which is incompatible.
azure-cli-iot 0.3.3 has requirement azure-mgmt-iothub==0.6.0, but you'll have azure-mgmt-iothub 0.5.0 which is incompatible.
azure-cli-dla 0.2.3 has requirement azure-mgmt-datalake-analytics==0.2.0, but you'll have azure-mgmt-datalake-analytics 0.6.0 which is incompatible.

To Reproduce
Use this Dockerfile

FROM python:3.7.0-stretch

RUN pip install --no-cache-dir azure azure-cli azure.batch

Expected behavior
Package dependencies are resolved properly

Environment summary
I tried with Python 3.7.0, Python 3.6.7, Python 3.6.4 (this is the one used by official Microsoft Docker image for azure-cli), I used both pip that comes bundled with corresponding Python and the latest version. The error messages differ depending on versions used, but it never goes through without errors. To make matters worse, the versions of packages installed in official Docker image are unavailable from PyPI (there's no such thing as azure=2.0.47 in that repository, for example, which is the version reported by the image).

[tjprescott]

Pip is not the recommended installation method for azure-cli on any platform. Please see https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest for the recommended method for your platform.

[tjprescott]

You can also do this:

pip install azure-cli 
pip install --upgrade azure-batch

It will still complain that azure-cli-batch requires 5.1.0 but it will update your azure-batch installation to 5.1.1.

This would not be recommended though, because we cannot guarantee the behavior of the CLI with mismatched SDKs.

The recommended solution would be to pip install the SDK and use the platform-specific installer for the CLI (which will then reference its own dependencies separately). If you want to pip install the CLI, you should so in a separate virtual environment.

cc/ @lmazuel @johanste

[wvxvw]

Wait, so why do you keep uploading azure-cli to pypi.org, if it's not installable with the tool that is specifically provided by PyPA? Simply remove it from there, and solve the problem. It doesn't work anyway, so it doesn't qualify for releasing it for general public...

Better yet, fix the dependency problem... I mean, telling me that it is broken by design doesn't make it less broken...

[HQJaTu]

If pip install is not the recommended way, why don't you upgrade the OpenSUSE-package. It's depending on Python 2 and stopped working months ago. So, a deadlock persists, the "recommended" way doesn't produce a working result and the non-recommended isn't supported.

[tigerkeeper5]

Can someone explain how to install Azure cli not using pip in a Dockerfile on Ubuntu? I am having the same issue where the pip azure cli install is overlapping with other pip azure libraries. There are no examples that Microsoft provides for dockerizing their manual install.

I also cannot use the Azure CLI docker image. It needs to be added to my own image.

[IRLeif]

I'm using azure-cli in an internal Python package. As such, I need to include azure-cli as a dependency in the internal package, so that azure-cli gets installed automatically whenever a developer in my organization installs the internal package. Right now, this approach results in dependency conflicts between Microsoft's Python packages.

I would have to agree with @wvxvw that, if the azure-cli package cannot be installed via pip without jumping through additional hoops, then it should not be available via PyPI, as this is confusing.

Otherwise, I would need to ask all developers to manually install azure-cli on their machine, which is undesirable as it requires special foreknowledge on the developer's part. How can the installation process be automated?

[yonzhan]

@fengzhou-msft to take a look

[fengzhou-msft]

Conflicts can be expected when you pip install multiple Python packages that share common dependnecies because pip only supports installing one verison of a dependency and does not have a dependency resolver. It has the "first found wins" behavior for dependency requirements/constraints. If you change the order to pip install azure-cli azure, azure-cli will win and the dependencies would not be compatible with azure.

On the other hand, we cannot take away the version constraints of dependencies since azure-cli could easily break with a non-compatible version of dependency.

Platform-specific azure-cli packages solve this issue by isolating the dependencies in an azure-cli-specific loction and not share with other Python applications. In most cases, you should first consider the rpm/deb/msi/homebrew azure-cli package.

The constraints of pip does not make azure-cli python package useless though, for instance, you can easily play with multiple versions of azure-cli in different virtualenvs which cannot be done with platform packages.

[fengzhou-msft]

I'm using azure-cli in an internal Python package. As such, I need to include azure-cli as a dependency in the internal package, so that azure-cli gets installed automatically whenever a developer in my organization installs the internal package. Right now, this approach results in dependency conflicts between Microsoft's Python packages.

I would have to agree with @wvxvw that, if the azure-cli package cannot be installed via pip without jumping through additional hoops, then it should not be available via PyPI, as this is confusing.

Otherwise, I would need to ask all developers to manually install azure-cli on their machine, which is undesirable as it requires special foreknowledge on the developer's part. How can the installation process be automated?

In this case, you can try to put azure-cli in a different location with --target/--prefix options of pip and include the az script in the path.

[HQJaTu]

Conflicts can be expected when you pip install multiple Python packages that share common dependnecies because pip only supports installing one verison of a dependency and does not have a dependency resolver. It has the "first found wins" behavior for dependency requirements/constraints. If you change the order to pip install azure-cli azure, azure-cli will win and the dependencies would not be compatible with azure.
...
The constraints of pip does not make azure-cli python package useless though, for instance, you can easily play with multiple versions of azure-cli in different virtualenvs which cannot be done with platform packages.

Ok. Make sure to mention all of this appropriately in documentation.

When Azure-CLI is installed from RPM-package, it does not work. All we users need is a working approach.

[fengzhou-msft]

Conflicts can be expected when you pip install multiple Python packages that share common dependnecies because pip only supports installing one verison of a dependency and does not have a dependency resolver. It has the "first found wins" behavior for dependency requirements/constraints. If you change the order to pip install azure-cli azure, azure-cli will win and the dependencies would not be compatible with azure.
...
The constraints of pip does not make azure-cli python package useless though, for instance, you can easily play with multiple versions of azure-cli in different virtualenvs which cannot be done with platform packages.

Ok. Make sure to mention all of this appropriately in documentation.

When Azure-CLI is installed from RPM-package, it does not work. All we users need is a working approach.

Can you submit an issue for the RPM-package with more details about your environment?

[HQJaTu]

Sure. On an OpenSuse machine go for a: zypper install azure-cli and see how everything gets tangled.