Pipenv: No documented way to unpin after converting requirements.txt

[tilgovi]

Describe you environment

1. MacOS 10.13
2. Python version: Python 3.6.3
3. Pipenv version: Pipenv 8.2.7

Expected result

Converting an existing project that uses requirements.txt, Pipenv creates a Pipfile with pinned versions and a Pipfile.lock with the versions from my requirements.txt. It then tells me I might want to unpin my Pipfile. I expected there to be a process for unpinning my Pipfile while maintaining my Pipfile.lock, but pipenv wants to recreate my Pipfile.lock whenever I change my Pipfile.

Actual result

I can't find any documented way to remove the version pins from Pipfile without the next invocation of pipenv install upgrading all my packages. The only way I have found is to let pipenv regenerate Pipfile.lock, then discard all the changes except the meta hash.

[erinxocon]

That is the expected behavior. Pinned versions will have a different Pipfile.lock than having all of the packages being unpinned. It tells you you may want to unpin them if you want pipenv to look for the latest versions of the packages, but generally a requirements.txt file has the specific version you had installed so we are just doing a direct migration of packages. If I'm misunderstanding something let me know! Closing for now.

[tilgovi]

A requirements.txt file does often have specific versions. My expectation is that a Pipfile would not, but that the Pipfile.lock would. By committing the Pipfile.lock to source control you get a reproducible environment.

Once I have an unpinned Pipfile and a pinned Pipfile.lock the important commands I would want are to add/remove packages and to upgrade packages (installing and locking a new version).

However, if the intended migration path to pipenv is to convert the requirements.txt to a Pipfile, with no documented way to then get an _unpinned_ Pipfile, it seems that something is lacking or my expectation is off somewhere.

[tilgovi]

with no documented way to then get an unpinned Pipfile

By this I mean no documented way to unpin your Pipfile without changing the locked versions in the Pipfile.lock.

[tilgovi]

It feels to me like it would make more sense if the conversion from requirements.txt generated a Pipfile without versions and a Pipfile.lock with the versions from the requirements.txt.

[erinxocon]

@tilgovi That doesn't really make sense. Your Pipfile.lock is supposed to represent the Pipfile so that it's reproducible. Why do you want them to be out of sync?

[tilgovi]

Maybe the intended use is that I set versions in my Pipfile and then Pipfile.lock is only there to lock my transitive dependencies. Perhaps pipenv should not tell me to unpin my Pipfile after I've converted from requirements.txt. When I read that, I expected to be able to follow those instructions without causing changes to my installed packages.

If I have "*" versions in my Pipfile and a fresh Pipfile.lock, then nothing is out of sync. If a dependency is later upgraded by its author, my Pipfile.lock should prevent accidental upgrade. I would not consider anything out of sync in this case. I would only consider my packages out of date.

If I recall correctly, in every other similar system I've used packages are never changed without explicit user request.

• bundler (Gemfile -> Gemfile.lock)
• npm (package.json -> package-lock.json)
• yarn (package.json -> yarn.lock)

It's true that often, with any of these systems, I might specify exact or approximate versions in the unlocked file. Nevertheless, I don't think any of these change their lockfile without an explicit request to add or upgrade a dependency.

I found it surprising that pipenv install changed my existing Pipfile.lock.

I see the --ignore-pipfile option. It works great for pipenv install, although I would expect it to be the default. What I don't see, though, is any workflow where I can pipenv install <package> without upgrading everything, unless I maintain version specifiers in my Pipfile. I was hoping that pipenv would alleviate much of the burden of manual pinning by locking my versions and giving me the tools to inspect and upgrade easily, selectively, and only ever explicitly.

[tilgovi]

Having just slept on this some more, I wanted to add that I hope it's helpful feedback and that I haven't offended. I'm genuinely trying, but struggling, to figure out if pipenv is the tool I want for myself and for coworkers and willing to contribute if I can understand what directions of improvement might be welcome.

[nateprewitt]

That's completely reasonable @tilgovi, and I don't believe you've offended anyone. We're happy to engage on these topics but unfortunately the time of our core maintainers is spread pretty thin currently. I haven't had time to do any deep dives but will try to make time to get back to you on this in the next couple days.

[tilgovi]

💯 no pressure to rush. I've been lurking for months and there's no need for immediate resolution. If there's a recommended way to get plugged in to helping out, please let me know. Happy to start trying to chip away at things to ease the load.

[techalchemy]

I'm not sure about the rest of the contributors or if they are on any real time discussion platforms like slack/irc. For now I am on IRC in the pipenv channel on freenode /cc @erinxocon @nateprewitt @vphilippon if there is some other platform you guys are on from time to time or would encourage discussion using

[erinxocon]

I'm in the IRC channel for it now

[tilgovi]

I'm still stuck trying to convert an existing project to pipenv.

First, I want to check my understanding. I have two assumptions:

1) Pipfile.lock maintains a hash of my Pipfile and it will not be regenerated unless I change my Pipfile
2) Pipfile can contain any version specifier and as long as what's in my Pipfile.lock satisfies that nothing short of a pipenv update should change my Pipfile.lock.

With these expectations in mind, I try to convert an existing project that has direct _and indirect_ dependencies already pinned in a requirements.txt file:

• pipenv install -r requirements.txt
• remove all the indirect dependencies from my Pipfile
• remove the version pins from my Pipfile

At this point, I see no way to get the Pipfile.lock hash updated without causing upgrades to occur. Even using pipenv lock --keep-outdated indirect dependencies get updated.

Please help me see what I am misunderstanding.

[tilgovi]

I've managed to do what I wanted by discarding changes other than the meta hash from Pipefile.lock as I incrementally mutate my Pipfile into where I'd like it to be long term. This process feels wrong.

[tilgovi]

The last comments of #966 about dependencies being unexpectedly upgraded sounds like what I need to follow. You can leave this closed. I'll head over there.

[benjaminjkraft]

Wanted to add one comment here: it seems to me like (once --keep-oudated is working per #966) it might be helpful to mention this strategy explicitly in documentation or in the "you may want to unpin your dependencies message". That is to say, if you want to migrate from requirements.txt to Pipfile, leaving your dependencies unpinned but without upgrading them today, you want to:

• pipenv install -r requirements.txt
• change all your specifiers to * in Pipfile
• pipenv lock --keep-outdated
  The last one is not at all obvious, and if you don't do it, you'll think you kept the pinned dependencies in Pipfile.lock, but actually as soon as you pipenv install they will get clobbered by the latest ones.

[techalchemy]

@benjaminjkraft we have been talking about tracking historical metadata for updated packages so you can see what has changed. Other than that if you need to keep specific versions you need to have those pinned in your pipfile or @ncoghlan will descend from the heavens and consume your soul with a security lecture about OWASP best practices. In the top 10 are using applications with known vulnerabilities and failure to patch / upgrade software. We trust you to tell us what you need to keep pinned. Pipenv will resolve the rest.

[benjaminjkraft]

The point isn't that we know we need to keep a specific version pinned, nor that we don't want to update sometime soon. We just don't want to be forced to update at the same time we switch to pipenv, because pipenv is supposed to be better at helping us to keep packages up to date while retaining the ability to manage the upgrades with care! In fact, even if literally the next commit I make is to upgrade all the packages, I'd like to do that upgrade separately, so I know, when inevitably something breaks, whether it was setting things up incorrectly when switching to pipenv or updating the packages. (And, depending how careful I'm being, I may want to do that separately for each package.)

Perhaps the better way to do that is to simply leave each the version pin in the Pipfile until we're ready to update that package? But then we lose the ability to use e.g. pipenv update --dry-run to see what we're going to update -- and it seems like this is not the best way for a Pipfile+Pipfile.lock to express what we want, and is different from what we would have if we had used pipenv from the start.

Perhaps this will all be clearer if I propose some alternate verbiage in context -- I'll make a pull request.

[tilgovi]

Thank you, @benjaminjkraft, that exactly captures my feelings, too.

[techalchemy]

Thanks for the PR! Concrete examples are easier to discuss than theoretical meanings. Will review shortly

[techalchemy]

P.s thank you for the explanation, it definitely clarifies the point for me.

[ncoghlan]

While I do get preachy about OWASP A9, --keep-outdated does have multiple legitimate use cases :)

(Plus there's the whole lock file "freshness" discussion in https://github.com/pypa/pipenv/issues/1886#issuecomment-381122438 )