Azure-cli: Proposal: Stop Shipping Command Modules as Independent PyPI Packages

[marstr]

Today we ship a series of Python libraries on PyPI that match the pattern azure-cli-* which encapsulate the declaration of commands for interacting with individual services. A few examples:

• https://pypi.org/project/azure-cli-appservice/
• https://pypi.org/project/azure-cli-resource/
• https://pypi.org/project/azure-cli-vm/

We then stitch together the set of commands that we want to call the Azure CLI in another top-level PyPI package: https://pypi.org/project/azure-cli Fundamentally, there isn't any real code that ships as part of this package, it just aggregates the list of command modules that we want to include.

We are currently in the middle of discussing how we want to go about locking our dependencies and create a stronger reproducible build story. A lot of that conversation has hinged around just locking our command modules, so that we aren't too opinionated about dependencies when people are installing our package in a virtualenv that isn't exclusively dedicated to our tool. However, it's worth considering: If we only want to lock our command modules, and we will always want to lock to the version that is currently in the same branch in source control, should we be shipping these independently at all?

In a lot of ways, the easiest thing to do would be to ship all of the command modules together inside the "azure-cli" package itself. It would solve the problem we've seen customers run into while trying to install a particular version of the CLI using pip, where we need to distribute to them a requirements.txt file based on their platform and version of Python. It would also mean that there were exactly two well-defined install scenarios: 1. Installing into a foreign virtualenv where we don't have control of our dependencies. 2. Installing into a virtualenv we can call our own, and use modern tools like pipenv to lock our dependencies and have a truly reproducible build.

What's more, in some ways, we're essentially already doing this. Consider our .msi, .rpm and, .deb packages, which install a locally built wheel of the command modules instead of reaching out to PyPI. This means that taking this approach would just bring our pip package into parity as a release channel with our other more platform-specific release channels.

Now, there is at least one downside: Folks who want to install a dramatically minified version of the CLI would have a harder time than they do today. Instead of creating their own shell packages akin to azure-cli from their own repositories and taking dependencies on just the command module libraries they care about, they would have to create a fork of our repository. Truthfully, I don't _think_ there's anybody doing this today. However, even if they are, I'd be excited to see if it actually just opened up a doorway for us to merge those curated packages back into our repository to publish different SKUs of the CLI that have optimized command-sets for different tasks.

Now, all that said, if I've missed something that makes this untenable, I want to hear it! What have I missed? @yugangw-msft @tjprescott @williexu @adewaleo

[tjprescott]

Philosophically this sounds reasonable to me. I don't know how much work it would be technically.

[marstr]

The first step is probably an exploratory one that should be time-boxed to a day or less: In a branch, pick one representative module and try modifying that to the proposed shape. Once we know how much time it took for one package, we'll have a lot more information about how involved it would be to do the rest.

edit: In that exploratory action, we probably wouldn't try to fix build problems that were introduced in our release scripts/azdev either. We would strictly look at pip.

[tjprescott]

@lmazuel might have a good idea how to do this or if there are pitfalls.

[marstr]

Chatted with @lmazuel about this over the weekend offline. The summary is, it wouldn't work for the SDK because it's too large of a dependency to push on people (which, totally agreed.) But that we're already packaging all of these modules as the CLI anyway. He didn't see any trouble with doing this.

[marstr]

We just finished a meeting where the Azure CLI team discussed this, and we all agreed that ultimately this is something we'd like to do. However, given that there are changes to the way our testing infrastructure will work, we'd prefer to not pay those costs until after //build in May.

[tjprescott]

@marstr you can put it into milestone sprint 61, which the first post-Build milestone.

[tjprescott]

I think as part of this work we would need to prioritize #5969 as this would allow us to deliberately and more clearly identify things in preview without relying on a "module version" number.