Pipenv: How does pipenv know the virtualenv for current project ?

[sachinjain024]

I have just setup a python project using pipenv. One thing I want to know is where does pipenv store the mapping of project directories to virtualenvs.

To be clear, I am running the following command:

sacjain-macOS:coursera-classification sacjain$ pipenv --venv
/Users/sachinjain/.local/share/virtualenvs/coursera-classification-iTBt6WsT

I checked Pipfile and Pipfile.lock, I assumed this information should be present in Pipfile but it was not there.

Q2. How can we install a specific version of package using pipenv ?

[uranusjr]

The virtualenv’s name is the project’s root directory name, plus the hash of the full path to the project root. This guarantees the name to be unique and predictable as long as you don’t move the project. (AFAICT this name generation logic is an implementation detail that should not be relied on.)

To install a specific version of a package, use the same syntax as you would with pip and requirements.txt, e.g. pipenv install django==1.11.0.

[sachinjain024]

@uranusjr This means that if I rename the directory my project's virtualenv will be lost. That's bad. It can be a common use case to rename a directory and users may fall into this and wonder how their environment stopped working.

I would suggest to fix it and make an entry into Pipfile if that's possible. Or create another .pipenv file to store such metadata in each project directory.

What do you suggest ?

Thanks for answering both the questions!

[nateprewitt]

Hey @sachinjain024, we've had a few discussions on the local metadata file but the maintainer consensus has been that pipenv won't be supporting that at this time.

The directory path identifying a project was something that was implemented early on because projects with the same name, or multiple copies of the same project, were causing accidental overwriting of environment state.

In practice, we've found far fewer people have had issues with the location being part of the environment than silently having their work overwritten. Pipenv is a deployment management tool after all, so moving the directory should be a one command fix.

[sachinjain024]

Thanks @nateprewitt. It makes sense to quickly jump to easy implementation. Suppose I rename a project at later stage then it means my environment does not exist.

So what should I do in this situation ?

1. Resetup the environment because I have Pipfile already which has the packages information so it will be easy to resetup the env.
2. Somehow rename the virtualenv directory too
3. ???

What do you suggest in this case ? Just curious in case I get in this situation in future.

[uranusjr]

You can use pew cp to easily copy the existing virtualenv, but 1. would be the more “canonical” way to do it I think.

[nateprewitt]

@sachinjain024, whenever you need to move your project directory, you should only need to run pipenv install to get back to a working state. This will create an environment for you with the new hash and reinstall everything from the lockfile identically to the previous installation it was created with.

If you find yourself doing this often, you may want to use pew rm old_project-a3de90 to cleanup.

[nateprewitt]

I wouldn't recommend using pew cp unless you're modifying the virtual environment outside of pipenv. In that case, I think we'd have to consider that a "voided warranty" scenario, because there are all sorts of changes we can't account for reliably.

[sachinjain024]

Thanks @nateprewitt @uranusjr. Closing this ticket.

[wvxvw]

This is one of the worst ideas that came out of pipenv so far. Here's why:

I work in a company, where Python is used for test automation of a lot of internal projects. One tester usually works on a dozen of those. By convention, all test code for a project lives in directory called tests. This means, that each tester now has a bunch of virtual environments all called something like ~/.local/share/virtualenvs/tests-hKjFddnZ - awesome! It is a human nature to forget to switch virtual environment, so, every few days I'm called to a different workstation because the the person at that workstation thinks that they installed the package they need, they ran pipenv install, but the imports in the code don't work. Not only that, the orphan virtual environments accumulate over time, but since all of them have nondescript names, there's no way to tell if I'm deleting environment that's not in use, or the one that's actually been used. This means that in CI, we create dozens, if not hundreds of virtual environments daily. There's no way anyone can keep track of environments that have essentially random names, and there's just too many of them, so we have to delete all of them at least once a day. We pay a huge toll in waiting times only because of this super smart decision made by pipenv and someone in our company, who decided to use it...

[jtratner]

Set export PIPENV_VENV_IN_PROJECT=1 in your bashrc / shell configuration.
(Or change your shell scripts to create a venv at $PROJECT/.venv and then
pipenv will has that)
On Mon, Mar 19, 2018 at 6:28 AM wvxvw [e-mail] wrote:

This is one of the worst ideas that came out of pipenv so far. Here's why:

I work in a company, where Python is used for test automation of a lot of
internal projects. One tester usually works on a dozen of those. By
convention, all test code for a project lives in directory called tests.
This means, that each tester now has a bunch of virtual environments all
called something like ~/.local/share/virtualenvs/tests-hKjFddnZ -
awesome! It is a human nature to forget to switch virtual environment, so,
every few days I'm called to a different workstation because the the person
at that workstation thinks that they installed the package they need, they
ran pipenv install, but the imports in the code don't work. Not only
that, the orphan virtual environments accumulate over time, but since all
of them have nondescript names, there's no way to tell if I'm deleting
environment that's not in use, or the one that's actually been used. This
means that in CI, we create dozens, if not hundreds of virtual environments
daily. There's no way anyone can keep track of environments that have
essentially random names, and there's just too many of them, so we have to
delete all of them at least once a day. We pay a huge toll in waiting times
only because of this super smart decision made by pipenv and someone in
our company, who decided to use it...

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pypa/pipenv/issues/796#issuecomment-374211264, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABhjqyABdDHCB8WMm-hJwdkNptVlh8fuks5tf7KBgaJpZM4Pp3kf
.

[wvxvw]

@jtratner Then why do I need pipenv if I'm going to create virtual environment by hand?

Virtual environment is a piece of work on its own. It's lame everywhere and is broken on Windows (it assumes that on MS Windows, of all places, the default home directory is called "~"). I was hoping that if someone was to redo PIP and virtual environment, they would, you know, fix the obvious errors of their predecessors... instead, there's this mess multiplied by a layer of incompatibility.

[uranusjr]

You’re making wrong assumptions. Pipenv does not redo either virtualenv or pip. It builds on them.

[wvxvw]

@uranusjr that's a reading comprehension problem. I know very well, what pipenv does, since I spend much more time debugging its code than I'd like to. To redo virtual environment and PIP is the project's stated goal. That, instead of fixing them it decided to use them "as is" is another huge mistake on the part of pipenv, followed by using click library, and the list goes on.

[uranusjr]

To redo virtual environment and PIP is the project's stated goal.

Where did anyone state this?

[Flimm]

I also found this behaviour to be weird, I was expecting pipenv to pick a standard name for a virtualenv directory and put the directory in the current directory, simalar to the way npm uses node_modules.

[uranusjr]

@Flimm export PIPENV_VENV_IN_PROJECT=1. Read the documentation.

[Flimm]

@uranusjr I read all of https://docs.pipenv.org and https://docs.pipenv.org/basics/ , and unless I missed it, it never mentions where the virtualenv directories are installed, nor does it warn anybody that renaming or moving the project directory will cause the virtualenv directory to have to be recreated.

PIPENV_VENV_IN_PROJECT is confusingly named, since pipenv does not use venv shipped in the standard library, it uses virtualenv, which is a different project.

I'm just trying to provide feedback on the things I tripped over as I'm familiarising myself with this project.

[techalchemy]

@Flimm the point is to abstract this information. If you know how other tools that manage virtualenvs work you should already know that moving folder of virtualenvs makes them un-findable. If you aren’t familiar, you probably won’t go digging for this in the first place.

As for what backend library we use to generate virtualenvs and the naming of environment variables the former isn’t that important and the latter is shorthand that is widely accepted. While details are important, this one doesn’t feel like it is causing anyone any issues, and your primary objection seems based on you being unsure which backend we use to put a python executable in a folder. If you encounter bugs please report them. We really aren’t going to change environment variables _just because_.

The environment variable is readily available in the docs: http://pipenv.readthedocs.io/en/latest/advanced/#configuration-with-environment-variables so you probably didn’t read all of them if you didn’t see it

[Flimm]

@techalchemy By other tools I guess you mean something like virtualenvwrapper. I went for years without using virtualenvwrapper. I'm guessing that the number of Python developers who are familiar with the node_modules pattern of Node/NPM is greater than the number of Python developers who are familiar with the virtualenvwrapper or pipenv pattern of hiding the environment directory. But it's a guess. For those not familiar with any of these tools, I think it would be confusing not to know where the installed files are ending up. You point out that this is only explained in the advanced section of the docs under the environment variables subsection, I think it should be explained in the docs much earlier than that. Maybe I'll submit a pull request.

I've commented on the confusing reference to venv instead of virtualenv in the naming of PIPENV_VENV_IN_PROJECT in another issue #1919. Suffice to say, I don't quite hold the position I think you think I hold.

In any case, the question in the original issue is answered. I don't want to prolong this discussion, feel free to have the last word if you want it.

[wvxvw]

I understand this is the wrong audience, but since this discussion is going on... well, I don't think that node_modules were an inspiration for pipenv, and not for virtualenv either. My guess may be historically incorrect, but having to work with both rbenv and virtualenv, I can tell you that virtualenv feels like either a bad copy w/o understanding, or maybe an early prototype, which was improved on by rbenv. In either case, virtualenv is a laughing stock compared to rbenv, and here's why:

It copies packages for each environment. This is stupid, slow, and confusing. It's just a bad design decision, but more likely just a lack of decision: it just happened like that because someone wrote some code to put these packages into arbitrary directory. On top of that, it doesn't work on Windows really. To the point that it was never really tested on Windows... One ridiculous thing about it is that it tests for OS, and once it discovers that it runs on Windows, it sets home directory to ~, this default may later be overridden by some environment variables, but if you are actually trying to run your automation in a supervised and clean environment... virtualenv will simply never install the packages in the same directory.

So... pipenv alleges to be a tool for human beings, kind of like requests... I guess? So, it ought to improve the immature efforts made by its predecessors... well, it seems like that would be the way to go, if you want to make a program that does the same thing as those before you, but better, right? And yet, instead of throwing away the garbage of virtualenv, it uses it as is. The wrapper is only different from the wrapped program in that it doesn't allow / makes it more difficult to use some of the functions from the wrapped program...

In my life, I've seen a lot of attempts to do automation, which sucked, but you guys are shooting for the stars, you put all those undeserved appraisals on you site, even before you put any useful information. But you don't live up to the promise. In reality, you simply manipulate public opinion, without doing any real work. And now programmers who have to do automation need to face another evil, just because someone wanted to get more "likes" on GitHub...

[uranusjr]

@wvxvw Indeed you are incorrect. virtualenv predates rbenv by something like five (four, see below) years, and therefore can never be inspired by it. Also they are fundamentally different things, only achieving a similar goal in the end. The thing you are looking for (runtime management tool inspired by rbenv) is pyenv.

Also regarding virtualenv being not on par with other tools—do you know it has been around for ten (10) years? Software developement requirements change a lot, and assumptions it made have been gradually going out of date, and nobody has cared to step up to replace it. Do you care enough to step up?

I understand everyone deserves a chance to speak, but to join a discussion without minimal understanding to the topic, and start pointing fingers right away, this is a quite rude to contributors of Pipenv, virtualenv, and Python packaging systems as a whole. Please do not do this.

---

Edit: I did some digging. virtualenv’s initial release, 0.8, was made on 2007, which rbenv’s (0.1) was 2011, so they are four years apart, not five.

[techalchemy]

@wvxvw I’ve only seen you offer insults. This kind of attitude isn’t really welcome here. Please see our code of conduct before you continue posting.

[wakenmeng]

I got the same problem after changing the project path and lose the original virtualenv mapping, then I read this thread. First, I appreciate the work of the pipenv team. Just got some opinions i'd like to share about this discussion:

1. The document indeed should point out the mapping mechanism between project path and env, at least warn users that changing project path would cause unmapping the original env.

2. Will it be better if you can manually set the path to the env in the Pipfile? I mean people may have some special requirements to manually use the same env.

Just my opinions to share with you guys.

[ashnur]

It is completely unclear how to use pipenv. Should I have many virtual environment for one project? Should I share the virtual environments between projects? How do I install a different python version for a specific project with pipenv, if the python version is only needed for that project? Everyone is so full of themselves in this thread, assuming a bunch of things about other people, never trying to understand where others are coming from. When I read the praises of pipenv on the homepage, I believed that it will help me. Instead, I wasted 5 days of wrestling with it, and now I think I go back to pyenv because that was at least somewhat deterministic.

[gsemet]

if you want to use multiple environment for one project, use tox. Use pipenv for your main development environment, and tox for testing on multiple python version.

[techalchemy]

@ashnur you clearly need something to do. Instead of spreading negativity among an open source project run by volunteers, why don’t you try contributing something useful?

[devxpy]

@uranusjr
this is where the hashing happens?

I am working on a project and need to calculate the hash of a path.

[uranusjr]

@devxpy Yup, exactly.

[djangorobert]

I think you guys should put a basic tutorial on how to set up a virtualenv with pipenv something very basic because the easier it is on people the more people will use it but if it causes confusion then more people may not use it and may seek other languages or methods to build there projects

[AndrewPrifer]

@uranusjr what is the reason that Pipenv doesn't create the virtualenv by default in the project directory? The way I see it, it would solve the issue with orphaned environments when the project is renamed/moved/deleted. Besides, it would be less confusing to people who (somewhat rightfully) expect it to work like npm.

Is there any benefit to preferring this approach besides maybe tradition?

[gsemet]

Simply create the .venv directory before the pipenv install command. A —venv or —dot-venv option to pipenv install would be welcome, actually :)

[uranusjr]

Starting with 2018.10.9 there is another way to do this. You can add a .venv file containing the path to your virtual environment. (Sneaky new feature!)

@andrewpeterprifer Because we used to do this, and needed to change it because some people rejected very strongly. We need to choose one approach or another, and I had to admit that it is a better default to not put virtual environments inside the project directory.

p.s. Would you (or anyone) be interested in writing a faq entry about this? It would likely take a few paragraphs, but I would be more than happy to explain if someone promises to take time to polish the text and submit a PR.

[AndrewPrifer]

@uranusjr I’m super curious why it is a better default. I’m a relative noob to python and now I feel like I’m missing something obvious about best practices. :) Thanks!

PS.: I could write a FAQ entry if you explain why things are the way they are. ;)

[gioxc88]

Starting with 2018.10.9 there is another way to do this. You can add a .venv _file_ containing the path to your virtual environment. (Sneaky new feature!)

@andrewpeterprifer Because we used to do this, and needed to change it because some people rejected very strongly. We need to choose one approach or another, and I had to admit that it is a better default to not put virtual environments inside the project directory.

p.s. Would you (or anyone) be interested in writing a faq entry about this? It would likely take a few paragraphs, but I would be more than happy to explain if someone promises to take time to polish the text and submit a PR.

I am sorry to bump in ... I have this very same need to move projects folders that have a virtual environment created with pipenv.

Currently I do the following and have no problem at all:

• I move the project folder wherever I want
• in C:\Users\user\.virtualenvs I delete the folder of the venv associated to the project I just moved
• I navigate to the new project folder location via cmd and run pipenv install (or pipenv shell and then pipenv sync)

Everything works ok. Is this a bad practice?

Regarding @uranusjr comment if I understand correctly with the new feature I should add the .venv file (containing the path to the desired virtual environment) into the project folder right? And this would avoid me of having to do all the steps I previously metioned? If so it's great!!
If that is the case then wouldn't it be better if such a file were automatically created in the project folder when the virtual enviroment is initially created?

PS: I am also willing to write a FAQ

[wakenmeng]

Actually, I think @gioxc88 's idea is good, a newly created virtualenv can automatically generate .env file under project root, and contains env configs like PATH. It will make the environment more transparent to developers and a more convenient way to reconfigure.

[uranusjr]

I’ll try to answer your questions together. The approach isn’t bad (IMO; I use the same setup myself as well). It is, however, easier for an unaware user to trip on.

One way to think of it would be to consider putting a project in version control (say Git). If the environment is created in project root, it would naturally be in the repository root. People like you and I, who are used to this setup, obviously know we should add a rule in .gitignore (or the global ignore config) to prevent the .venv directory from being checked in, but an unsuspecting user would not, and could very easily checked in the environment by accident. This is not only bad for project management, but (more importantly) provides an vector to potential attacks by exposing local information. Therefore, it is a general rule for project management tools to avoid putting generated files inside project directory. It could be okay (still not optimal IMO) to do it if you’re designing an ecosystem from scratch (e.g. NPM’s node_modules), but definitely not a good idea for tools built for an ecosystem with established common practices, as the case of Pipenv.

If an environment is generated outside of project root by default, there is one less possible thing to worry about when you publish your project (e.g. push it to GitHub). It makes our (who prefer in-project environments) lives a bit more difficult, but from a risk perspective, the worst that can happen is to accidentally move the project and break the environment, or forgot to remove an environment when a project is removed. Either is much, much more easily recoverable than accidentally pushing your potentially sensitive environment information to GitHub.

The .venv file feature is still quite new (released only days ago), and we’re still exploring ways to best utilise it. It still has the same problem of a .venv directory, but hopefully not as bad. I really like the feature myself, and definitely hope we can use it to improve the user experience, but there are still a lot to consider here.

[gsemet]

Hi. For me, the .venv directory (old feature) and .venv file (new feature) are ok, but I would be very grateful to have an option in the pipenv install command.

Something like:

pipenv install

Would install in ~/.local/

pipenv install --venv

Would install in $PWD/.venv directory

pipenv install --venv-dir /my/custom-path

Would install in /my/custom-path.

[techalchemy]

If you want a new feature, please propose it properly by making an enhancement proposal to ~/.peeps. Please don't make enhancement proposals scattered randomly around various issues, it's not trackable.

[AndrewPrifer]

@uranusjr thank you for the detailed answer! Just to satisfy my curiosity, what kind of local (potentially sensitive) information could be exposed by a checked in python venv? I agree that it is annoying if node_modules gets checked in, but normally that wouldn’t contain any local information.

[uranusjr]

Python virtual environments contain several shell scripts (e.g. activate). A lot of people modify them to add environment variables to specify database passwords, path to another config file, etc. Modifying scripts in virtual environments is against best practice, but people do this nonetheless (and get defensive when you tell them to stop doing it—personal experience).

[uranusjr]

Also, you’re right, node_modules probably does not contain sensitive information. That is another benefit when you build an ecosystem from scratch. Python virtual environments was unfortunately invented way before this is a common problem (back then you’re already a guru if you use VCS at all).

[floatingpurr]

I arrived here while I was trying to understand how pipenv knows the right virtualenv : )

Thank you guys for that discussion. Probably, it could be a good idea specifying in the main page (e.g., here) that renaming the project path breaks the default mechanism of virtualenv binding.

[uranusjr]

All documentation pages are open for contribution. Don’t ask for things, do it :)

[floatingpurr]

I will!

[yashrastogi13]

@uranusjr This means that if I rename the directory my project's virtualenv will be lost. That's bad. It can be a common use case to rename a directory and users may fall into this and wonder how their environment stopped working.

I would suggest to fix it and make an entry into Pipfile if that's possible. Or create another .pipenv file to store such metadata in each project directory.

What do you suggest ?

Thanks for answering both the questions!

thanks for asking both the question. i also encounter the same problem ,

[rajaravivarma-r]

If you initialized pipenv in a wrong directory and have to utilize the virtualenv directory from the correct directory, you can get the virtualenv path by doing pipenv --venv and move the PIpfile and Pipfile.lock to the correct directory.

echo ${PATH_OBTAINED_FROM_PREVIOUS_COMMAND} > .venv in the correct directory and it should work properly.

[BeebBenjamin]

I noticed today that if there is a pipfile in the parent directory, pipenv ignores the environmental variable export PIPENV_VENV_IN_PROJECT=1 and installs a venv in parent directory instead. This is on release 2018.11.26. If you remove the pipfile from the parent directory, pipenv works as documented.