Pip: Add ability to install a specific pre-release version of a package

[TheFriendlyCoder]

• Pip version: 10.0.0b2
• Python version: 3.6
• Operating system: Mac OSX

Description:

I wasn't sure if PIP would be the best place to report this "feature request" or not. I understand there is minimal support within setuptools, wheel and the other supporting packages for this so feel free to recommend where else I may want to post this request.

Basically what I'm looking to be able to do is build versions of a package that implement specific features, and then be able to install the builds for those specific feature builds for testing.

Consider this use case:
I'm working on project MyCoolProject. The next release is going to be v5.0.0. Further, suppose we have 2 developers working on this future release: Jane and John. Now, suppose both developers are working on their own specific features for this next release: Jane -> FeatureA, John -> FeatureB. Both developers have forked their codebases to work on their respective areas, and both are using continuous integration / deployment, testing, etc.

It would be helpful if Jane could then install specific builds of her fork for testing and prototyping and such, while John does the same. For example, Jane should be able to do something like the following to install the latest build of the MyCoolProject which includes her specific feature (ie: built from her personal fork / feature branch):

pip install MyCoolProject==5.0.0.FeatureA

Similarly, John should be able to install the latest build for his feature doing something like:

pip install MyCoolProject==5.0.0.FeatureB

What I've run:

Based on my current understanding, pip seems to look at the version specifier - 5.0.0 in these examples, and then simply sorts the resulting packages associated with that version alphabetically, returning the "latest" one it finds.... so in practice I think you'll always get the latest build or whichever package was published last installed... so if John published the latest build and Jane tried to install the package from her feature branch, she'd end up getting John's build instead....

I think part of this is behavior is based on the fact that the non-numeric characters in a version number have very specific / vague patterns / interpretations. For example, the setuptools docs suggest you should always use one of 'alpha', 'beta', 'dev', 'rc', 'a', 'b', 'c' for your version "labels" ... but there's nothing in any of the Python packaging tools that forces / enforces those specific labels, so it is perfectly reasonable to use 'MyFeature' or 'MyName' as labels instead ... although the expectations / behaviors resulting from using these patterns is undefined at best.

So maybe my feature request here is more of a request to document a more flexible naming convention for these non-numeric version identifiers / labels and to implement support for those patterns in pip... not certain. Any suggestions on how best to proceed with this would be appreciated.

[cjerdonek]

Are the releases already published, or is this theoretical? If the former, have you tried === in place of ==, e.g. pip install MyCoolProject===5.0.0.FeatureA? See this section of PEP 440. I'm not sure if that will work, but it's easy enough to try.

Some other comments:

Have you already read the pip install docs? A couple things these docs mention are that you also have the option of installing versions under development by specifying a VCS URL. That approach might work better for you. There's also the --pre option for installing pre-release and development versions.

Lastly, you might also want to try perusing PEP 440 in full for more info on the various naming rules (e.g. about development releases).

[TheFriendlyCoder]

So, I am finally getting back to this feature request. Thank you so much for your response.
I have skimmed the PEP 440 document quite a few times, but I have never really taken the time to read the spec in more depth... until now. So here's an update to what I have tried thus far.

Are the releases already published, or is this theoretical?

We publish a package for every commit using continuous integration. When built from a branch we typically treat them as pre-releases and attach a 'dev' suffix to the end (ie: 5.0.0.dev1234). When built from a tag we remove the dev suffix and publish a package with just the semver compatible release version (ie: 5.0.0).

What we want to do is to be able to specify a different descriptive identifier to indicate that a "dev" build contains a specific feature (ie: 5.0.0.featureA1234) and to be able to install the latest build of said feature using pip (ie: pip install MyCoolProject>=5.0.0.featureA -> installs latest build of 5.0.0 containing feature A).

There is nothing theoretical about my feature request. This is our actual production use case.

have you tried === in place of ==, e.g. pip install MyCoolProject===5.0.0.FeatureA?

I was unaware of this "arbitrary equality" operator until you pointed it out. I tried it using one of my sample projects and found the same behavior as when I used the equality operator (ie: Could not find a version that satisfies the requirement MyCoolProject===5.0.0.FeatureA)

Have you already read the pip install docs? A couple things these docs mention are that you also have the option of installing versions under development by specifying a VCS URL.

I am quite familiar with the pip docs. Honestly I hadn't given the VCS installation option a second thought as we typically pull packages from a package repository. I took the time tonight to give it a more thorough test. While it does seem to work fairly well when manually installing individual packages locally I couldn't get them to work as package dependencies in my setuptools scripts. (ie: setup(install_requires=["git+ssh://path/to/my/repo@branchname"],...)). Setuptools gives the error "'install_requires' must be a string or list of strings containing valid project/version requirement specifiers".

Also, even if setuptools did recognize that syntax there would probably be other nuances that would cause problems with that. For example, our git repositories require authentication so I'm not sure setuptools would be able to pull sources from git applying the relevant authentication protocol (ie: ssh key in my example) to get the sources to build...

There's also the --pre option for installing pre-release and development versions.

Yes, I am aware of the --pre option, however the behavior of the pip install command both with and without this option result in the same behavior (ie: pip install --pre MyCoolProject==5.0.0.FeatureA == pip install MyCoolProject==5.0.0.FeatureA)

Lastly, you might also want to try perusing PEP 440 in full for more info on the various naming rules (e.g. about development releases).

Unless I'm missing something, I don't see anything in the development releases section that indicates how to embed a feature-specific identifier in the version information anywhere. The closest thing I could find that seemed promising was the local version identifiers. By appending a "+" character it seems the spec shoud allow an alphanumeric label identifier to be added, followed in turn by a numeric identifier that could be used for sorting purposes (ie: 5.0.0+featurea.12345, 5.0.0+featurea.67890, 5.0.0+featureb.2468, 5.0.0+featureb.4321). However, based on my ad-hoc testing this numbering format still exhibits the same behavior as all my previous test cases. Lets say I deploy 2 packages for MyCoolProject with the following versions:

• 5.0.0+featurea.12345
• 5.0.0+featureb.45678

Then I perform a pip install MyCoolProject>=5.0.0+featurea. I'd expect to get version 5.0.0+featurea.12345 installed, but instead I get version 5.0.0+featureb.45678. It's like the version parser sorts the first portion of the local version identifier and says "featureb > featurea" (due to alphanumeric sorting) and then returns 5.0.0+featureb.45678, effectively ignoring the final dot-separated numeric token in the sequence. imo that's actually a bit worse than the way the .dev#### suffix works. Lets say you publish these two example builds in the opposite order, so you have:

• 5.0.0+featureb.12345
• 5.0.0+featurea.45678

Doing a pip install MyCoolProject>=5.0.0+featurea actually installs version 5.0.0+featureb.12345... which is non-intuitive at best and totally incorrect at worst.

---

Finally, after reading the version spec a bit more I came across the compatible release comparison operator which looked promising. I was hoping / thinking that doing something like pip install MyCoolProject~=5.0.0.featurea or pip install MyCoolProject~=5.0.0+featurea might work, but sadly this operator only seems to work with the basic semver / numeric portions of the version identifier. The docs explicitly state that local version identifiers are not permitted and that pre/post/dev/alpha/beta identifiers are ignored. In fact, if the alphanumeric identifier isn't one of the explicitly supported character strings mentioned in the spec (ie: 5.0.0.featureA1234) pip will fail outright. :(

So, after spending a few more hours trying different combination of options and reading the PEP440 spec more thoroughly I am still left with the understanding that there is currently no way to achieve the results I'm looking for with this feature request.

If I've still missed or overlooked something though feel free to let me know.