Pipenv: Predictable virtualenv location

[quentinsf]

Pipenv is looking very good for personal development, but it's not very easy to use it for deployments to our servers because of the unpredictable virtualenv locations:

• I don't want to put .venv in the source directory, because I believe that the directories on the server should be a clone of the git repo with as little corruption as possible. (It's nice to rsync from a clean clone, for example)

• I don't like the automatically-generated venv names because to have any control over them at all you need the anachronistically-named WORKON_HOME, they're hard to predict in advance (you can't even get pipenv to report the path until it's created the venv), and it's hard to know how to tidy them up afterwards.

At present, with pip, we know that the virtualenv for an app is in /virtualenvs/{app} on our servers and everything is very easy. We can configure uwsgi, for example, knowing where it should find the python environment.

I think the only official way to get this nice behaviour with pipenv is to create the environment independently using virtualenv or similar, activate it, and then run pipenv from within it - is that right?

I can live with that, but it seems a pity that pipenv will never be able to _replace_ our use of virtualenv until or unless it gets the ability to specify where the virtualenv should be.

If that really goes too much against the philosophy of the project, could the --venv option be modified to show where the virtualenv would be, even before it's created?

[uranusjr]

First of all, yes, the idiomatic way to do this currently is to create/activate the environment manually, and use Pipenv inside it. It automatically picks up the surrounding environment.

I think I agree with your reasoning. It would be very nice if it is possible to know where the environment would be before hand, without being stuck with ./.venv.[#] The problem here, however, is that this is already established, and already working. To offset that, you’d need to provide both a very convincing argument (you’re getting there, but I don’t think this is enough to convince all), and a working alternative. The second part is extremely important. It is not enough to say what you want; you need to actually try to provide a way to get there. This is how (non-corporate-backed) OSS works, generally.

[quentinsf]

Thanks @uranusjr ,

Yes, I'd be happy to do the work and provide a PR if I thought it might be accepted, but we'd need to get reasonable agreement before that made sense.

I _do_ understand and have some respect for the idea that the virtualenv should be named based on the lockfile that created it, but I can see problems there too in some situations. So I do think an option to override the location is definitely needed, if others can be convinced.

In the meantime, I may just have a more complex stack:

• Ubuntu gives me an environment in which I can run Python
• Python gives me an environment in which I can run pip
• pip gives me an environment in which I can run virtualenv
• virtualenv gives me an environment in which I can run pipenv

:-)

[techalchemy]

Your setup at work is like mine — where you put things on a server and deploy them. Pipenvs approach is built for the docker/kubernetes universe where everything is throwaway besides the instructions for rebuilding it. Deterministic hashing is designed to abstract away virtualenv locations and avoid naming collisions. It’s not configurable because knobs tend to make people think they _should_ configure things.

At work, we just set WORKON_HOME and use the hashed names. In the next release of pipenv it will be possible to figure out which venv is associated with a pipenv environment, and at some point we will make an interface for that. Until then pipenv pipes has been doing tons of work on this.

[tbsf]

I'm switching over from virtualenv. I set WORKON_HOME to our old virtualenv location to be consistent. The hashes are an issue since they are random. It would nice to be able to disable them since we don't need them.

Currently I'm using:

$(cd /pipenvs/<pipenv name>; pipenv --venv)

in place of the predictable location.

Any better solutions?

EDIT: It would also be nice to have a --name argument instead of using a folder location. Instead of cd-ing into a folder and creating an environment, I could create one with pipenv --name <pipenv_name> install Pipfile.<pipenv_name> and have it create a virtualenv in the typical location. If I want to use that folder location I can use $(pipenv --name <pipenv_name> --venv) without navigating to the folder where the pipenv was created.

[techalchemy]

Thanks for the feedback @tbsf. The hash is not random; if it were, we would never be able to find your virtualenv again. The hash is a segment of a sha256 of the absolute path to the project on your filesystem.

If you want to get the path to the virtualenv without changing directories, simply set the environment variable PIPENV_PIPFILE to the path to the relevant pipfile and then get the output of pipenv --venv.

To your other point— pipenv is designed to manage _specific projects and their environments_, which means a directory and its virtualenv. Pipenv is not meant to simply create and activate virtualenvs for you. If this is what you want to do, there are much better tools that already exist (such as virtualenvwrapper, pew, etc). The sha256sums are not really optional at this time as they prevent collisions, but there is a possibility that we will add flexibility around this in the future.

If you want a tool to manage your pipenv projects, check out https://github.com/gtalarico/pipenv-pipes by @gtalarico which is quite nice.

[gsemet]

Let keep thing simple. These “I don’t like this or this” are subjective.
Python is opinionated.
Setup.py is.
Pip is.
For example i don’t like the same manner setup.py clearly prefer the main lib folder to have the savent name has the library, instead of a “src” folder, but I live with it.
If you want to test your app on several environment, use tox.
Pipenv allows to get the same lib sources in différents folder without conflict, the job is done for the majority.
A .venv folder in the current dir is a pretty classic and acceptable solution. Just add it in your .gitignore.

The other solution would be to have a lot of environment variable to change the default behavior.

[quentinsf]

Thanks, Dan - that's useful.

I think I'm right In saying that ‘pipenv —venv’ will only tell you the location once the virtualenv exists? Would it be ok to allow it to report the path just based on the filename? I could work around most of my issues if I could predict the path in advance.

Then I could do

VENV_DIR=$(WORKON_HOME=foo/foo/foo PIPENV_FILE=bar/bar/bar pipenv —venv)

Or I could copy the hash function and right my own utility, but it would be sad if that ever got out of sync with pipenv's way of doing it.

Mmm. Perhaps you're right: it's easier just to use other tools like virtualenv for now :-)

[uranusjr]

Note: this is actually possible to do now with Python because I added a Project.get_location_for_virtualenv() method, but we need a place to put it in the CLI.

[consideRatio]

So I wanted to cache the virtual env, but with GitLab CI that required my ability to specify a location explicitly, and it was okay if it would reside in .venv which pipenv would pick up on if it was found.

So, I did the following

pip install pipenv
pipenv install --dev
mv $(pipenv --venv) .venv

[gsemet]

Simply create .venv prior to using pipenv:

mkdir .venv
pipenv install --dev