Poetry: Manually specifying the venv path

[viniciusd]

Is it possible to manually (via some config) specify which venv poetry is going to use?

Due to some requirements, I would like to have two projects sharing the same virtual environment.

Best case scenario would be setting the path via a config or environment variable

[thernstig]

@viniciusd Is this what you are looking for: https://poetry.eustace.io/docs/configuration/#settingsvirtualenvspath-string ?

[viniciusd]

Not exactly. This one defines where each project's virtual environment will be set.
For example, if I have settings.virtualenvs.path = /usr, and install two projects, A, and B, the first while will be located in /usr/A while the latter should be in /usr/B.
I want them to share the same virtual environment instead.

[ulgens]

pyenv solves this by .venv file. You can put your env path in it and it automatically uses it.

[finswimmer]

Hey @viniciusd ,

could you describe in which scenario two projects needs to share the same virtuell environment?

Thanks.

fin swimmer

[ulgens]

@finswimmer Other usecases are setting the name for venv or using an already existing venv for new project.

[marco-silva0000]

can this not be solved with https://poetry.eustace.io/docs/configuration/#settingsvirtualenvspath-string

[ulgens]

can this not be solved with https://poetry.eustace.io/docs/configuration/#settingsvirtualenvspath-string

Nope. It just gives a base dir for all venvs.

[finswimmer]

Is it possible to manually (via some config) specify which venv poetry is going to use?

To answer this first: AFAIK there is no way to do this (at the moment).

Due to some requirements, I would like to have two projects sharing the same virtual environment.

@finswimmer Other usecases are setting the name for venv or using an already existing venv for new project.

Mhh, I wouldn't do this. It looks to me like a bad practices. New projects should start with a fresh virtual environment to ensure only dependencies needed are installed. Different projects should have different environments to avoid any conflicts when (de)installing/updating/downgrading a package for one projects.

One use case for specifying the path to the venv I can imagine, is when you run out of space and one need to put the venv files to a different location. A workaround (I haven't tested) might be as follow:

When settings.virtualenvs.in-project (virtualenvs.in-project in poetry v1.0.0) is set to True, poetry expect the venv files in the .venv folder inside the project. One could create a symlink called .venv that point to any other location where the real files are located.

fin swimmer

[ulgens]

To be honest, i don't think this is not about a being good/bad practice. Personally, i don't want to see (foldername-8charhash-pyversion) for every venv i activated, in console prompt. Giving a user the chance to change this is about "giving the chance" :) . I still think pipenv's way to do it with .venv file is just okay.

[viniciusd]

In addition to what @ulgens mentioned, a use case would be having a shared project with shared modules and dependencies in order to guarantee the different services/projects are using the same versions of the shared dependencies.

[marco-silva0000]

on deployment jobs is also useful/common to setup the environment in stages and with more flexibility, currently I have to export to requirements.txt to then use the correct env with other tools.
It's not only about being good/bad practice, sometimes you want the minimum entropy change and gradually implement changes to the building or deploying, and having the choice is always good.

[finswimmer]

The more I think about this feature request and the more you told me about the use cases, the less I'm convinced that poetry should support is.

poetry is about managing python projects and not environments. poetry is also about best practices, as it always seeking for solutions that are already standardized or are respected as those. Also it limits its configuration to be able to have one way to do it and not many. This makes the projects highly compatible to another and on different platforms.

Being able to define the path to the virtual environment and so makes it possible that multiple projects uses it, introduces another problem: Who is responsible that the poetry.lock is always up-to-date in all projects? Who is responsible that the dependencies defined in pyproject.toml in each project are always valid.

The clean way would be, that you define a new environment on your system, where in the end all projects should play together, that have these projects as dependencies.

[ulgens]

poetry is about managing python projects and not environments

Poetry can create virtual environments, activate and deactivate them, use existing ones, but not allows to define a venv path. I'm not sure this is a "best practice".

Who is responsible that the poetry.lock is always up-to-date in all projects?

I think this is out of scope. For given usecases, it's not really important and people already have other solutions.

I feel like you're missing the point. It's not about changing the default behaviour. I can set the tool to put venv in the folder and i can create a symlink from venv folder to real environment, but what if i don't want to do it for every project? Is it a bad thing if we have that possibility?

There is some strong words about why Poetry shouldn't implement it but it's already here, just half baked.

[schinckel]

My workflow that I think could take advantage of this is using tox to run tests.

Currently, when you use poetry install inside a tox environment, it uses the virtualenv that poetry "owns" for that python interpreter. This means that it's not an isolated environment: it's probably shared with the development environment.

Being able to use a specific environment (or even the "active" environment) would remove this problem, and make running tests using tox isolated.

You can't change the settings on poetry to temporarily change the venv location, because those settings are always user-global: and there's no guarantee you are the only poetry install running at a given time, which can result in conflicts if you change settings like venv location.

[schinckel]

FWIW: I have a workaround: you can use poetry export -f requirements.txt, and then pip install from that. It's not flawless (poetry will output invalid requirements.txt at times, omitting python version markers), but that's another issue.

[JoeJasinski]

I think this would be a useful feature to specify the exact virtualenv you want to use. For example, I'm using Poetry inside of a Docker container and I'd like to specify the exact directory where the virtualenv should be created. And I don't want to use the "in-project" setting because while I'm developing with docker or docker-compose, I often like to mount in my source code into the docker container (so I can make live updates).... but I don't want to mount in the virtualenv necessarily. I'd like to exactly specify where in the container I'd like to create the virtualenv so all of my configuration and scripts can reference that location.

When I set ENV POETRY_VIRTUALENVS_PATH=/site/env/ in my Dockerfile, Poetry creates a virtualenv under that directory with a random name. That makes it hard to determine where my virtualenv is inside of the Dockerfile or with other scripts. I understand that a feature of Poetry is the ability to use more than one virtualenv (which is great), but it would be nice to specify exactly which environment to use when I know I want to use it. For example, it would be nice if there were a setting called POETRY_VIRTUALENV_PATH (no S) to specify the exact location.

Maybe there's a way to do this already - I'm still getting used to the new 1.0 features.

[garthk]

If you need a predictable PYTHONPATH for your Dockerfile, strikes me the return value of EnvManager.generate_env_name is stable for any combination of name and cwd. As long as the team keep it stable, for any particular source path you'll be able to poetry env list and derive, from the output, something like:

ENV PYTHONPATH=/opt/example/example-7rCCnWI7-py3.7/lib/python3.7/site-packages
RUN poetry env use `which python`
RUN test "$(poetry env list | cut -f 1 -d ' ')" == "example-cHSZFygF-py3.7"

The test will crash your docker build if the path drifts.

[schinckel]

Inside a docker container you can also prevent poetry from creating virtual environments: this will be local to the docker container.

RUN pip install poetry
WORKDIR /app
COPY poetry.lock pyproject.toml /app/
RUN poetry config virtualenvs.create false
RUN poetry install --no-dev -vvv

[geckon]

I'd also like to see a possibility to specify path to virtualenv manually. It's useful in docker and possibly in other use cases too.

I know that I can create the virtual env manually, activate it and then run poetry in it but it seems like unnecessary hassle considering how poetry makes my life easier in other areas.

[kkawabat]

As a novice in package management, this is my usecase where specifying the venv path would be useful. If I can install A inside B's venv then I would not need to install B twice.

https://stackoverflow.com/questions/60287564/how-to-manage-editable-shared-libraries-with-poetry

[caleb15]

@ulgens I don't see any documentation on a .venv file in https://github.com/pyenv/pyenv

How do I use it?

I tried creating a .venv file with contents /home/caleb/.cache/pypoetry/virtualenvs/fifteen5-deploy-GcwqD37l-py3.6 (the output of poetry env info under Path) but the poetry venv was not automatically loaded.

I can't find any info on how to automatically activate the poetry venv using pyenv :(

Edit: I avoided the issue alltogether by just configuring poetry to not create venvs

[CrossR]

Specifying just a specific name (without the hash) would be good enough for me, and I can't seem to find a nice way of doing that.

My use case is having a venv setup for neovim that contains some packages that my neovim plugins need. I used to just set the python_host_prog to /path/to/my/venvs/nvim-plugins-py3.7/bin/python and it worked across machines (assuming I set poetry to always install to the same location). Now I'm having to have awkwardly change it on each machine, and remember not to commit that change to my config.

I have similar issues in VSCode with Python paths.

[jagretti]

I'd also like to see a possibility to specify path to virtualenv manually. It's useful in docker and possibly in other use cases too.

I know that I can create the virtual env manually, activate it and then run poetry in it but it seems like unnecessary hassle considering how poetry makes my life easier in other areas.

Hey @geckon , I'm trying to do this exact thing (create a virtual env manually and then use poetry inside of it), but poetry for some reason is not detecting that virtual env. An important thing, I'm trying to do it in a Dockerfile. I activate the virtual env, and then I set the VIRTUAL_ENV var with the venv directory, and poetry is still creating a virtualenv on their own. Do you know how can I do this?

[geckon]

@jagretti, you should be able to do something like this...

# Create virtual environment.
RUN python -m venv "/path/in/the/container/venv"

# Activate the venv.
ENV VIRTUAL_ENV "/path/in/the/container/venv"
ENV PATH "$VIRTUAL_ENV:${PATH}"

# Install poetry into the venv.
RUN pip install poetry

# Install dependencies into the venv.
COPY poetry.lock pyproject.toml "/path/in/the/container/"
RUN cd "/path/in/the/container" && poetry install --no-dev

[geckon]

Or if that doesn't work for some reason, you can try something like:

# Create virtual environment.
RUN python -m venv "/path/in/the/container/venv"

# Install poetry into the venv.
RUN . /path/in/the/container/venv/bin/activate && pip install poetry

# Install dependencies into the venv.
COPY poetry.lock pyproject.toml "/path/in/the/container/"
RUN cd "/path/in/the/container" && . /path/in/the/container/venv/bin/activate && poetry install --no-dev

[JulianFerry]

@geckon

@jagretti, you should be able to do something like this...
...

How would you do this using the recommended poetry installation curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry. py | python?

I would still like to be able to have my poetry installation separate from my virtualenv in order to avoid installing any of poetry's dependencies without having specified them in pyproject.toml. However I would still like to be able to point to a specific virtual environment with the same setup as @JoeJasinski .

---

Edit: Actually I think I figured it out. This should install poetry as a separate installation, and then cause poetry add and poetry install to install packages to /opt/venv

RUN curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python - -y
RUN python -m venv /opt/venv
ENV PATH="/root/.poetry/bin:/opt/venv/bin:$PATH"
RUN poetry config virtualenvs.create false

[shughes-uk]

For people still wondering about use cases. I have two python projects, one is the central project and the other is a plugin for the project. I'd like have each be installed and editable in the same environment as I am often making changes to both in tandem.

In the past I would simply do pip install -e project1 project2 but that does not work without a setup.py

[schinckel]

@JulianFerry be aware that poetry settings can not be applied to just the shell or location: they are user-global.

[JulianFerry]

@JulianFerry be aware that poetry settings can not be applied to just the shell or location: they are user-global.

Thanks. I am just using this for docker builds, which means this is not an issue. But as you say, perhaps there could be a better solution for development environments.

[abn]

@shughes-uk does using a setup.py shim do the trick for your use case? A similar topic was dicuseed at https://github.com/python-poetry/poetry/issues/761#issuecomment-643830479.

For brevity, this is what a shim could look like the following snippet taken from this post.

import setuptools

if __name__ == "__main__":
    setuptools.setup()

Additionally, there is also #2270 being proposed to better enable usecases like this, where either code is shared and/or managed together.

[shughes-uk]

A setup shim might be best for now

[AndrewTrapani]

For people still wondering about use cases. I have two python projects, one is the central project and the other is a plugin for the project. I'd like have each be installed and editable in the same environment as I am often making changes to both in tandem.

In the past I would simply do pip install -e project1 project2 but that does not work without a setup.py

This can work with a setup.py auto generated during poetry build source distribution.

Warning: Your experiences may vary and I only validated with PyCharm

Let's say you want to link Project B into Project A (top level project), assumed locations are ~/Projects/A and ~/Projects/B

Goals:

1. Write code in both Projects A and B, each in their own directory I've cloned from the source control
2. Be able to use source control in both, e.g. change branches will auto update
3. Point my IDE to the Project A's poetry virtual environment and it will auto find the code in both projects
4. Debug from poetry shell and IDE, and set break points in Project B's source code which will actually work

This builds project B's source distribution and copies the setup.py into your top level project directory,

cd ~/Projects/B
poetry install
poetry build --format=sdist
tar -xvzf dist/B-${B_VERSION}.tar.gz --strip-components 1 B-${B_VERSION}/setup.py

Create project A's poetry virtual environment.
This builds project A's source distribution and copies the setup.py into your top level project directory,

cd ~/Projects/A
poetry install
poetry build --format=sdist
tar -xvzf dist/A-${A_VERSION}.tar.gz --strip-components 1 A-${A_VERSION}/setup.py

Setup for development for both projects A and B using setup.py:

poetry shell
(inside poetry shell)
cd ~/Projects/B
python3 setup.py develop
cd ~/Projects/A
python3 setup.py develop
exit
(exited poetry shell)

NOTE: I don't know how to do this outside of the poetry shell and manually typing it. I'd have preferred a Makefile target to do this automatically.

Where is it actually located in the Project A venv? (assuming python 3.7)

cd ~/Projects/A
cat `poetry env info -p`/lib/python3.7/site-packages/B.egg-info/
/absolute/path/to/Projects/B/src
../

If you point your IDE to Project A's poetry virtual env, many IDEs will automatically find Project B's source.

With PyCharm I'm able to debug and edit Projects A and B from the same window.
If I change the branch on Project B, PyCharm will find the changes automatically.