Azure-cli: msal and msal extensions pinned to old versions

[ianhelle]

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug
azure-cli-core has dependencies on msal (~= 1.0.0) and msal-extensions (0.1.3) that are incompatible with with recent versions of azure-identity (anything after 1.3.1).

azure-identity recently released support for use AzureCliCredential - pretty much the only way to obtain an AzureCredential from an existing Azure CLI logon. An AzureCredential is needed for recent versions of increasing parts of Azure SDK like Keyvault client, azure monitor and anything else migrating to msal-style creds. However, you cannot install a version of azure-identity that supports AzureCliCredential without a conflict with azure-cli-core.

New versions of pip are reporting this as an error and it breaks things like readthedocs builds.

To Reproduce
pip install azure-cli-core (or azure-cli)
pip install azure-identity

Expected behavior
azure-cli-core has more relaxed versioning requirements for msal and msal-extensions allowing it to be used alongside Azure SDK components that require later versions.

Environment summary
This is repro-able in any environment

Additional context

[yonzhan]

add to S176

[jiasli]

Yes, azure-cli-core has dependency on https://github.com/Azure/azure-cli/blob/573e66b75c1a6b102d32ef50e6740950e8c4fd4d/src/azure-cli-core/setup.py#L53-L54

However, installing Azure CLI with pip is not officially supported. If you must install with pip, we don't recommend installing Azure CLI and other Python SDKs in the same virtual environment.

azure.identity.AzureCliCredential only uses subprocess to call az account get-access-token, so it is fine to install Azure CLI and azure-identity under different virtual environments.

Moreover, installing Azure CLI and other Python SDKs in the same virtual environment will break other commands of Azure CLI as well, because Azure CLI still uses Track 1 SDKs and replacing them with Track 2 SDKs will break the underlying dependencies.

[ianhelle]

@jiasli - thanks for the update.
I understand the not supporting pip install for the whole of azure-cli and using separate environments, etc. but why is azure-cli-core even available on PyPI if you don't support installing with pip?

There's still the question that azure-identity now has AzureCliCredential support but in the current state of things you cannot use it in a supported way because it's impossible to have azure-cli-core and azure-identity in the same environment without this version conflict - although yes, you're right - I saw the way that azure-identity calls azure cli via subprocess.

BTW - I wasn't asking if it had a fixed dependency on 1.0.0 :-) - more why does it have this dependency on this quite old version? Do later msal/msal-extensions break functionality or is it just to constrain the test matrix (not that I'm saying the latter is unreasonable).

It's definitely complicated by the fact that newer versions of pip are much more vocal about conflicts like this. The really annoying thing is that we don't even use azure cli directly - one of our dependencies does. :-(

I'll play around with installing it into a separate environment but I'd assumed that a pip-installed azure-cli wouldn't even be visible to a different environment.

[jiasli]

I understand the not supporting pip install for the whole of azure-cli and using separate environments, etc. but why is azure-cli-core even available on PyPI if you don't support installing with pip?

I will leave this question to @fengzhou-msft who is working on CLI release pipeline.

The dependency of MSAL is for VM SSH. @arrownj will bump the MSAL version accordingly and test with the latest MSAL. Hope this will unblock you.

... one of our dependencies does

May I know how the dependency is using Azure CLI? Ideally it should also use sub-process like AzureCliCredential does.

I'd assumed that a pip-installed azure-cli wouldn't even be visible to a different environment.

az is supposed to be visible on the PATH. This will require you to edit your PATH environment variable to include az. For example, on Windows, if you install outside a virtual environment, it is at C:\Users\username\AppData\Local\Programs\Python\Python38\Scripts\az.bat. Inside a virtual environment, it is at <env>\Scripts\az.bat. But again, we recommend installing in officially supported ways, instead of using pip.

[ivanst0]

I believe this could be easily fixed by changing

'msal~=1.0.0',
'msal-extensions~=0.1.3',

into

'msal~=1.0',
'msal-extensions~=0.2',

@arrownj, would this work?

[jiasli]

@qwordy, could you help verify if msal-extensions~=0.2 works for VM SSH?