Pipenv: Add a pip-style `--upgrade-strategy` setting to `pipenv lock`?

[ncoghlan]

pip install these days offers an --upgrade-strategy option to let users choose between "only-if-needed" upgrades (which upgrades components only if needed to satisfy new dependencies) and "eager" upgrades, which upgrades everything which has a newer version available that still meets the dependency constraints.

In pip 10, the default strategy is changing from eager to only-if-needed: https://github.com/pypa/pip/pull/4500

pipenv lock currently follows the original pip install --upgrade policy of "upgrade everything".

This is actually fine for my own current use cases - most of my dependencies are stable enough that the combination of loose dependencies in Pipfile and eager upgrades in pipenv lock works well. I also think this remains the right default behaviour for pipenv.

However, I'm wondering if permitting pipenv lock --upgrade-strategy=only-if-needed may make pipenv applicable to more use cases.

[nateprewitt]

Thanks for bringing this to our attention, @ncoghlan! Do you know offhand what version of pip introduced --upgrade-strategy? My only concern adding this kind of functionality is that we may encounter conflicts with older versions of pip.

Our recent hiccup in Requests (requests/requests#4006) has me a bit paranoid about features that didn't exist back to pip 6.X, if not earlier. There are still at least 1.5 million people/instances using the 1.5.X series of pip, likely due to it coming from distribution repositories. I guess we could consider forcing a minimum version of pip though.

@kennethreitz, do you have any thoughts on supporting this. I do think we should maintain eager as the default after pip 10 is released.

[ncoghlan]

@nateprewitt It doesn't currently seem to be mentioned in the release notes [1], but https://github.com/pypa/pip/pull/3972 is the PR that added it. That means it would have first appeared in pip 9.0.0 in November 2016.

[1] I submitted an issue regarding that oversight: https://github.com/pypa/pip/issues/4568

[ncoghlan]

As far as the version compatibility question goes, I think it would be reasonable to have using --upgrade-strategy at the pipenv level simply fail if the underlying pip was too old (ideally with a nice error message explaining that at least pip 9.0.0 is needed, rather than letting the raw unknown argument error escape).

However, I also think you're right that the more important point here is to have pipenv pass --upgrade-strategy=eager on pip 10+ so that pipenv retains its current default behaviour. Allowing pipenv users to optionally pass --upgrade-strategy=only-if-needed would then just be a bonus.

[ncoghlan]

Adding @dstufft to the discussions, as he mentioned hoping to deprecate and remove the --upgrade-strategy option some day, and I think pipenv is an example of a use case where it makes sense to keep the old eager upgrade behaviour available indefinitely (just on an opt-in basis).

[stoggi]

This causes a problem for me. My Pipfile.lock specifies a specific version

"s3transfer": { "version": "==0.1.10"},

But when I run pipenv install a different version of s3transfer is installed 0.0.1. After a bit of investigation I figured out that pip was installing the correct version of s3transfer==0.1.10 but a subsequent package awscli was installing s3transfer==0.0.1 because it was specified as one of it's dependencies.

So, since the Pipfile.lock contains all dependencies of all your packages anyway, when running pip install from a Pipfile.lock we could be using the --no-deps option so that pip doesn't overwrite the dependencies at all.

[kennethreitz42]

I definitely want to avoid adding options if at all possible, and your statement of "This is actually fine for my own current use cases" is a telling one, in this instance :)

[nateprewitt]

@stoggi, the issue you're experiencing is actually unrelated to this. The issue tracking your problem is #298. We'll need to have some work done on how we both validate Pipfiles and handle dependency conflicts. Neither pip or pipenv handle this well currently but hopefully will in a future release.

[ncoghlan]

I'm going to close this one, as I think the pip-tools model is going to be a better fit for the "only-if-needed" case, and I'm fine with folks having to drop down to that lower level tooling if they want more control over how their requirements are managed.

However, the change in pip's default upgrade strategy still needs to be accounted for at the pipenv level, so I filed #438 to cover that.

[brettdh]

I'd like to push back here. only-if-needed is a much more sensible default in my view, particularly coming from other package managers such as npm, yarn, and gem. If packages are automatically upgraded when doing so is not required, then the lockfile is not a lockfile; it's just a suggestion.

The whole point of the lockfile is to pin dependencies to a known working version - to not have to assume that "most of my dependencies are stable enough"; to make all upgrades explicit (which, of course, is better than implicit).

I started this line of conversation in #966, but maybe this (or another issue) is a better place for it. Having --upgrade-strategy in pipenv would be a fine stopgap, but I maintain that only-if-needed is a more sensible default, as evidenced by pip 10 and the aforementioned package managers. The current situation doesn't allow either, which makes pipenv a less reliable tool for locking down dependencies.

[Telofy]

I agree with @brettdh. With pip, I used a constraints file to make very sure that versions don’t change, and I actually get deterministic builds. I hoped that the lock file would achieve the same purpose, but whenever I try to add a dependency (even one with no dependencies itself), Pipenv upgrades a number of unrelated packages and rewrites the lock file that I hoped would keep them at their exact version. Even if I do pipenv install --ignore-pipfile utilofies, the lock file gets changed and unrelated libraries are upgraded.

To get deterministic builds, Pipenv should add to the lock file but never change something that is already in it.

@ncoghlan: I’d like to drop down to this lower level now to make sure that my builds are really deterministic. What were you thinking of there? I tried out controlling pip’s behaviour with environment variables, but it didn’t work for me.

[ncoghlan]

@Telofy For the first project where I experimented with pipenv, we switched to using pip-compile to manage the actual test environment setup: https://github.com/jazzband/pip-tools

As far as version pinning goes, that works the same way as pipenv though: it expects you to be willing to upgrade all your existing dependencies whenever you add a new one.

I'm not aware of any current Python dependency pinning tools that offer a --keep-existing-versions mode, where they retain the pinned version for all current dependencies, and only add new deps, or remove no longer needed ones.

[Telofy]

@ncoghlan Thanks!

In all my projects save for the one where I’m experimenting with Pipenv, I pipe all of pip freeze into a constraints file (constraints.txt or versions.txt), and then specify that in my requirements.txt using the -c option. When I add a new dependency,

1. I add it to the requirements.txt,
2. rerun pip with the old version constraints, and then
3. pipe the new pip freeze into the versions.txt file.

If there are conflicts, I need to resolve them manually, but at least I can be sure that no transitive dependencies upgrade to versions that are subtly incompatible with dependencies of mine that I’d much rather use as black boxes.

What is tricky, is upgrading an existing dependency that has dependencies of its own without accidentally changing the versions of unrelated packages. I hoped Pipenv could do that.

It’s just imperative that nothing changes implicitly about any of the direct and transitive dependencies no matter whether the build is executed today or in a year.

[mimischi]

I agree with what @Telofy said. Deterministic builds would be really helpful and I actually thought pipenv was trying to do that with Pipfile.lock.

Let's say I write a Django app and use packageA==1.0.0, while there is a new version packageA==2.0.0 with breaking changes, that I do not want to include in my project. Upon upgrading to django==2.0.0, I add packageB via pipenv install packageB. Now pipenv will upgrade packageA to it's newest version, while I may wanted to keep the old version to keep my project from breaking.

In that case: what are my options? Pin down dependencies in Pipfile instead of using * as the version argument?

So far I've been using requirements.txt and pinned down version by hand / using services like https://pyup.io.