Pipenv: Pin `pipenv install X` to compatible version of X in Pipfile instead of *

[dlukes]

Is your feature request related to a problem? Please describe.

In my experience, pipenv seems to be a bit too eager to update to newer versions of dependencies (see e.g. #1219). Based on other tools like cargo and npm, I would expect pipenv install in a newly checked out project to get me a working environment for running the project. However, since pipenv install X pins to X = "*" in the Pipfile, it is possible that I'll get newer versions of the deps incompatible with the given project.

Describe the solution you'd like

See also https://github.com/pypa/pipenv/issues/1219#issuecomment-360690738:

Maybe Pipenv should do what npm does and make pipenv install pin to the major version instead of '*'.

I think it would be great if pipenv install X pinned to X = "~=<current_version>" instead of X = "*", ideally by default (but I admit my perspective may be limited by my particular use cases) or at least through an option that can be specified on the command line and/or in the Pipfile itself, something like --pin-to-current / pin_to_current = true.

Describe alternatives you've considered

What I'm doing currently is trying to remember to run pipenv install --ignore-pipfile instead of pipenv install when setting up a project for development on a new machine. However, this skips all updates, even compatible ones (bug / security fixes etc.), since the purpose of --ignore-pipfile is deterministic builds for production, not setting up a compatible environment for development.

[techalchemy]

well what's pinned in the pipfile isn't super relevant, or it shouldn't be... What should be relevant is what other packages you have installed right? Right now You're describing a change or proposing a solution, and I can kind of see where it might be coming from, but it'd be super useful if you could give us an example where "*" causes problems and "~=" is actually what is desired.

The reason I ask for this is because it is possible you are running into a resolver bug which is just fixed as of today, which is the primary reason the release has been delayed.

[uranusjr]

The general proposition of using ~= (or even ^=, which does not exist in pip) is still valid, and I have seen this raised quite a few times now. * means every re-lock and update could potentially break user code because a new major version is released and it is not compatible with the application.

You could argue the user should be aware of this, and use e.g. django~=2.1.0 in the first place. But still, there is no good way to say “I want the latest ~= possible” instead of manually looking it up. It would be immensely helpful to either a) set this as default, and require django==* to explicitly get *, or b) have a command flag as mentioned above (an option in Pipfile less so IMO).

While the problem described is probably fixed, the user experience side of this proposition is still worth discussing.

[dlukes]

Sorry, I should have given a more concrete user story in the OP to make the issue clearer :) For instance:

1. I start a website project, do pipenv install flask. This puts flask = "*" in my Pipfile and installs the latest version of flask, say 0.12.3.
2. I deploy a working version of the project (using pipenv install --ignore-pipfile to get the exact same dependencies as those used in development).
3. After some time, it turns out it's necessary to add a new feature to the project. In the meantime, I've either switched computers or simply deleted my local dev copy of the project, or perhaps a new team member will be implementing the new feature -- in all these cases, the dev environment has to be set up from scratch. These are the options:
   
   
   
   • pipenv install: this will install the latest flask 1.0.2, which may very well have some incompatibilities with 0.12.3, the version my project was originally developed with
   
   
   • pipenv install --ignore-pipfile (like when deploying): this will install the original version 0.12.3 of flask, but in the meantime, 0.12.4 has also been released, which should be compatible with the originally used version but probably contains bug and security fixes that I shouldn't ignore...
   
   
   • ... which means that what I should really do is manually look up the originally used version of flask in Pipfile.lock, change flask = "*" in the Pipfile to flask = "~=0.12.3", and onlye then do pipenv install, which will get me flask 0.12.4
   
   

I think the last behavior is the most useful one, yet it is the hardest one to achieve (the only one which involves manual operations). If the goal of the current default behavior of pipenv install (see first bullet above) is to nudge pipenv users towards ditching obsolete dependencies, I think that could be achieved in a less jarring way by just warning them that a newer version exists but won't be installed because of compatibility requirements.

As @uranusjr says, it's a user experience issue:

You could argue the user should be aware of this, and use e.g. django~=2.1.0 in the first place. But still, there is no good way to say “I want the latest ~= possible” instead of manually looking it up.

I couldn't agree more.

Another possibility is that I'm using pipenv completely wrong, in which case please point out the problems with the workflow above and I'll think about how the docs could be improved to better communicate pipenv's intended usage patterns :)

[techalchemy]

Right but I want the questions i asked answered. I am not super interested in talking about how pinning works because I know that already, I’m interested in UX. That’s why I was asking about the actual interaction.

For context, I don’t really want a bunch of random garbage in pipfiles that we arbitrarily decide on. I am currently of the opinion, and have always been, that these _default behaviors_ should be just that — behaviors performed during lock. So if we want a loose pin on major version to be the default, why do we need to put that in every entry? We can just lock with that behavior and use the old lockfile as a reference.

In the same vein, maybe the actual correct behavior of the resolver is something that isn’t expressed as a pin, but if I can see a use case I can understand how it might work. Because so far I see that we shouldn’t break things, but that’s super vague. Not everyone uses semver and it’s not necessarily sensible to force major version pins. I suspect that would cause a LOT of dependency conflicts unnecessarily.

I get the impulse. But understand that a resolver only works when its resolution space is unconstrained enough to find matches. That’s the whole point of having one. If you are going to upgrade one thing it makes a lot more sense to let the resolver sort everything out than to hand back 20 resolution errors; it also asks a lot more of the user if you go that route. Most importantly If you as a user make a series of upgrades, now you have either to go one by one unpinning things and resolving again or you have to update each major version pin by hand. That doesn’t seem like a good UC when we have lockfiles for pinning

I have yet to encounter a situation where it would be easier to do these unpinning steps than to see that your app dependency went up a major version and add just a single loose pin. So at the end of the day I feel like sure, we can argue about whatever, but I care a lot about what we are asking users to do and how much work that will take and how hard it will be and how intuitive and how smooth or pleasing. And if I think something makes the experience worse, I care about the alternatives and how good they are, and the impact of the problem.

If you know where else this has come up I’d love to get a list together to get a good understanding of shah it is that people need.

One last point: I want to reiterate that we need to drop the assumption that all code is semver and all semver only breaks with major releases. It simply isn’t the reality and I don’t think it’s fair to make the assumption on users behalf that all of these things are true of their dependencies.

[dlukes]

I care a lot about what we are asking users to do and how much work that will take and how hard it will be and how intuitive and how smooth or pleasing

Thanks for that :) And I totally get that I'm only a single data point on this, the fact that I happen to experience some UX friction in the current setup says nothing about other people.

I was just trying to convey my expectations coming to pipenv from other environment management tools I've used. npm install and cargo build both create an environment in which the project can be run. pipenv install not necessarily so, which came across as surprising and confusing the first time I encountered it. But maybe I'm part of a tiny minority.

If you as a user make a series of upgrades, now you have either to go one by one unpinning things and resolving again or you have to update each major version pin by hand. That doesn’t seem like a good UC when we have lockfiles for pinning

I would tend to think that in the age of GitHub, upgrading dependencies is something you do less often than setting up a project on a new machine, so you can afford for the former process to be slightly more involved if it streamlines the latter, but maybe I'm mistaken. In any case, the following looks to me like a perfectly acceptable upgrade experience:

1. Try to upgrade flask to the latest version.
2. If it can be done given other constraints in the Pipfile, then great.
3. If the resolver fails, then give the user an option to essentially unpin everything that's necessary in the Pipfile for the requested upgrade to happen, and upgrade those dependencies too (a.k.a what happens currently when all deps in the Pipfile are *).

I want to reiterate that we need to drop the assumption that all code is semver and all semver only breaks with major releases. It simply isn’t the reality and I don’t think it’s fair to make the assumption on users behalf that all of these things are true of their dependencies.

I agree and I'm far from being a semver zealot. I like the idea of signaling breaking changes, but I think it would be better achieved by attaching tags to version numbers, rather than hiding the information in version number comparison rules.

Still, it is used a lot out there in the wild, and relying on its semantics gives me a better chance of ending up with a working (w.r.t. the project) and up-to-date environment than * (which will be up-to-date but not necessarily working) or --ignore-pipfile (which will be working but not necessarily up-to-date).

[uranusjr]

Not strictly related, but I want to mention pipenv sync for deployment. It works like pipenv install --ignore-pipfile, but better.

[techalchemy]

pipenv install --deploy is probably the best option if you can afford to fail if your lockfile is out of sync with your pipfile.

Sorry for typos btw I was on mobile yesterday, which is also why I didn’t actually answer you when I responded — I hadnt seen your reply. The main difference for npm and cargo is that they enforce semantic versioning and python does not, so pinning is a lot more consistent.

[dlukes]

Sorry for typos btw I was on mobile yesterday

No worries :)

The main difference for npm and cargo is that they enforce semantic versioning and python does not, so pinning is a lot more consistent.

Thanks for pointing that out, I guess it's something I've vaguely been aware of in an informal way, but it helps to see it stated explicitly. With that in mind, the UX friction I'm describing may actually be just culture clash, to which the answer is of course, when in Rome, do as the Romans do. I still think the changes in behavior I suggested would be useful for the reasons detailed above, but if you think it's a bad fit for Python culture, I won't belabor the point :)

One last thing messaging-wise, though: pipenv explicitly bills itself as "a tool that aims to bring the best of all packaging worlds (bundler, composer, npm, cargo, yarn, etc.) to the Python world". It's the first sentence of the docs, and as such, it sets up expectations based on these other tools the reader might have used, including expectations about pipenv install. I think it would be a good idea to add a short discussion to the docs as to why this particular expectation won't be met (= the other tools assume and leverage semver, pipenv by design doesn't).

[techalchemy]

@dlukes thanks for the dialogue. Part of the goal of our messaging is to drive the vision for what the future of python packaging might need to be. I don't think anyone necessarily knows the answer to that question yet, and who knows, maybe pypi should enforce semver. It's worth discussing, and you're definitely not the first person to bring it up.

[techalchemy]

While I am still opposed to compatible version pins by default (especially as even more packages are moving to calver, making this even less meaningful of a strategy), this is probably worth discussing again. I am comfortable rejecting that specific idea for now but possibly proposing either an argument, a command, or a switch to toggle pins back and forth? Last time I talked with Kenneth we talked about possibly adding a config file, this might be something you could set there.

[uranusjr]

Maybe something like pipenv install <package> --pin-to-compatible would be enough. (And then we can have env car PIPENV_PIN_TO_COMPATIBLE to configure this globally.)

---

Edit: I view this as a usability problem. Since semantic versioning is common, it is common to want to have compatible pins. It is likely not a good idea to pin like that by default, but there is really not an easy way to do a compatible pin. So the point is not whether to do this by default, but we should provide a way to automate compatible pins when it is actually wanted. So my proposal above is to

1. Keep pipenv install <package> as is.
2. Add a flag to toggle compatible pin. If set, you’ll get something like package = "~=1.2.3" instead of * in Pipfile.

[dlukes]

I realize it's a fraught issue, what with some packages following calver, other packages doing dotted versions but without abiding by semver semantics...

Broadly speaking, I'm basically in favor of any mechanism that lets me pin dependencies in the Pipfile to more specific versions than * without having to manually hunt for the version numbers :) The details should probably be up to someone who can better assess what's the best fit for Python packaging in its current state (= you guys).

Re: "drive the vision for what the future of python packaging might need to be" -- I don't want to swerve too much off-topic, but I'm curious if there has been discussion of using the Pipfile for specifying the dependencies of libraries, i.e. instead of doing that in setup.py? Or is this something that's not even theoretically on the table?

[uranusjr]

I understand, the difficulty of manually hunting down the latest version bugs me a lot as well. Another thing I am pondering is that even if we don’t go down the compatible pinning route, maybe we should at least pin the user-provided version with >=. If the user doesn’t care about versioning, it is very likely they can work with the latest, and it might also be a good idea to tell the user if that doesn’t work with other dependencies. That would help the user to manually modify Pipfile to change >= from ~= as well. Would you care drafting up a PEEP about this? I believe it would be a lot less uncontroversial, but make the pinning process a lot easier.

[dlukes]

Would you care drafting up a PEEP about this?

If you meant me -- as I said above, if at all possible, I'd rather leave the details to someone who is better able to assess what's a good fit for Python culturally and for Pipenv technically :)

[awbacker]

I think this would be a great benefit, and am continually surprised its not in yet. I imagine every single pipenv user struggles with this.

What I do now is install with pipenv, then do pip list and see what it installed, then manually edit the pip file to pin == for that version. I am not in love with compatible version stuff, since I do not want any deps changing at all without my knowledge, but it makes sense as well.

Pinning to explicit versions would also solve all issues with calver. Perhaps a --pin-exact and --pip-compatible in that case, if you feel adventurous you can try the last one :)

[dlukes]

I stumbled across poetry today and based on a quick look, it seems like a very interesting alternative which might mitigate some of the issues listed here. It goes much further in the direction of e.g. Rust's/Cargo's Cargo.toml than Pipfile does (which to me personally sounds like an enticing proposition, but YMMV of course). So I just thought I'd mention it in case other people who end up reading this thread find it useful :)

[uranusjr]

@dlukes That was discussed a while before. I can’t find the thread, but the conclusion is that 1. Poetry assumes semver usage 2. Semver is not accepted practice in the Python community (while not only preferred but enforced by Cargo) 3. Semver-compatible pinning breaks in edge cases when the assumption is wrong. So while the benefits are obvious when the assumption holds, it is impossible for Pipenv to follow the same approach.

[dlukes]

@uranusjr Oh sorry for not being quite clear, I didn't mean that Pipenv should follow this approach, I understand (by now) it has a different philosophy :) I just meant that people who are using Pipenv but experience the issues described in this thread as stumbling blocks might be interested in trying out Poetry instead, because it might be a better fit.

[uranusjr]

I see, thanks for the clarification.