poetry creates virtualenv with newer python rather than using existing env

[cjw296]

• [x] I am on the latest Poetry version.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• OS version and name: macOS

• Poetry version: 1.0.5
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/cjw296/c60c14f105b24f3ca625a11c87789fa6

Issue

I've seen this across linux and mac, steps roughly as follows:

• my project has python = "^3.7"
• I run poetry shell which creates a virtualenv, ansible-config--DcQW89x-py3.7/ in this case.
• I do lots of setup, installation, use the env, months pass...
• I install a new version of Python on the machine (3.8 in this case)
• The next time I run poetry shell, rather than using the existing env, it creates a new, empty one, ansible-config--DcQW89x-py3.8/.

This should not happen, the existing environment should be used.

[cjw296]

Workaround 1: change to python = "~3.7", but this is hamstringing the project, it can run fine on Python 3.8.

[cjw296]

Workaround 2: go back to using python3.7 `which poetry` shell, but that means I need to figure out which version of Python was in use for that project, which poetry shell should be doing.

[finswimmer]

Hello @cjw296,

you can fix the venv right after you've created it with poetry env use python.

fin swimmer

[cjw296]

@finswimmer - that sounds like a hacky workaround, and doesn't actually solve my problem.
My problem is that I already have an environment set up, and that often be an expensive process when a project has a lot of packages. poetry should be using that environment rather than creating a new one every time a new version of python appears on my machine.

[finswimmer]

Hello again,

poetry expects in first place that the venv uses the same python version as the to whatever the python binary links to. Beside the way I explained above, another solution is to set virtualenvs.in-project to true. This will create the venv within you project folder and poetry will recognize it at first place.

[cjw296]

I'm sorry, but it feels like you're misunderstanding the issue I'm reporting.
poetry shell is the documented way to start working on a project, unless I'm misunderstanding you?

I believe that should use any existing virtualenv created by a previous run of poetry shell, rather than creating a new one, and that is why I submitted this bug.

The only exception I can think of to that would be if pyproject.toml had changed such that the python in that virtualenv was no longer valid for the project.

[MattiooFR]

I actually have the same problem, but without changing my version of python. I don't know why but it seems randomly poetry will forget the path of the previously created venv and will create a new empty one, quite annoying...

[finswimmer]

Hello again,

the behavior of poetry here is intended to be able to use pyenv. So, whenever I set the local python version to something different, it is expected, that poetry finds or creates a virtualenv with the same python version. I don't think we will change this.

Saying this, I really believe virtualenvs.in-project = true is what you need @cjw296 .

@MattiooFR: This should only happens if:

• the minor version of your python changed
• the name of the project change
• the location of the project change
• the venv with created with poetry 0.12 and you upgraded to 1.0

"randomly" doesn't sound as you could easily reproduce it, right? This would help a lot for finding the reasons.

[cjw296]

@finswimmer - please can you explain how building a new venv is a predicate for using pyenv?

This appears to be an artifact of using the latest compatible python each time poetry shell is started, rather that seeing if any compatible virtualenv has already been created, and that's the bug I'd like to see fixed.

I don't want virtualenvs in my project folders if at all possible.

Would it be possible to have another Poetry developer review this bug report, @sdispater perhaps?

[sdispater]

@cjw296 This is the intended behavior and not a bug.

Basically, Poetry will always use the currently activated/available Python executable to determine if the virtual environment exists.

What @finswimmer by mentionning pyenv, is that if you switch the Python version used by pyenv, you will always get the appropriate virtual environment. This makes it easier to develop on multiple Python versions.

But for your specific case, if you want Poetry to stick with your Python 3.7 environment, you can use the env use command: poetry env use 3.7.

[cjw296]

@sdispater - this output doesn't appear to agree with your statement:

$ ls -1 /Users/chris/Library/Caches/pypoetry/virtualenvs | grep ansible
ansible-config--DcQW89x-py3.7
(master) Mac Pro:ansible-config chris:
$ python --version
Python 2.7.16
(master) Mac Pro:ansible-config chris:
$ python3 --version
Python 3.7.1
(master) Mac Pro:ansible-config chris:
$ poetry shell
The currently activated Python version 2.7.16 is not supported by the project (^3.7).
Trying to find and use a compatible version. 
Using python3 (3.7.1)
Spawning shell within /Users/chris/Library/Caches/pypoetry/virtualenvs/ansible-config--DcQW89x-py3.7
. /Users/chris/Library/Caches/pypoetry/virtualenvs/ansible-config--DcQW89x-py3.7/bin/activate
(master) Mac Pro:ansible-config chris:
$ . /Users/chris/Library/Caches/pypoetry/virtualenvs/ansible-config--DcQW89x-py3.7/bin/activate
(ansible-config--DcQW89x-py3.7)(master) Mac Pro:ansible-config chris:
$ exit
(master) Mac Pro:ansible-config chris:

Now, let's "install" Python 3.8 and try again:

$ PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"
(master) Mac Pro:ansible-config chris:
$ export PATH
(master) Mac Pro:ansible-config chris:
$ python --version
Python 2.7.16
(master) Mac Pro:ansible-config chris:
$ python3 --version
Python 3.8.2
(master) Mac Pro:ansible-config chris:
$ poetry shell
The currently activated Python version 2.7.16 is not supported by the project (^3.7).
Trying to find and use a compatible version. 
Using python3 (3.8.2)
Creating virtualenv ansible-config--DcQW89x-py3.8 in /Users/chris/Library/Caches/pypoetry/virtualenvs
Spawning shell within /Users/chris/Library/Caches/pypoetry/virtualenvs/ansible-config--DcQW89x-py3.8
. /Users/chris/Library/Caches/pypoetry/virtualenvs/ansible-config--DcQW89x-py3.8/bin/activate
(master) Mac Pro:ansible-config chris:
$ . /Users/chris/Library/Caches/pypoetry/virtualenvs/ansible-config--DcQW89x-py3.8/bin/activate
(ansible-config--DcQW89x-py3.8)(master) Mac Pro:ansible-config chris:

The bug, for me, is that "trying to find and use a compatible python" should really be "trying to find an existing virtualenv for this project and use it" then "no virtualenv env found, trying to find a compatible python and then create and use one".

I don't think the pyenv stuff is really relevant here, since my python is 2.7 thoughout, but I will note that it saddens me to see poetry doing so much to work with pyenv, which honestly, from a linux sysadmin perspective is a bad tool. It causes a lot of problems, because it tries to compile its own python binaries, which is a terrible thing to do in environments where curated, and often optimised, binaries for a range of python versions are otherwise provided.

[MattiooFR]

@finswimmer it was probably because of a change of path then, this particular time I thought that nothing had change but I'm not 100% sure. So far it is working very well :)
Thanks !

[cjw296]

@sdispater - would be great to have your thoughts on https://github.com/python-poetry/poetry/issues/2282#issuecomment-613371329

[fsouza]

I'm running into a similar issue as well, poetry is able to find a matching version, but still creates the virtualenv with the wrong version. Reproducer from scratch:

％ mkdir /tmp/sample-proj
％ cd /tmp/sample-proj
％ poetry init -n --python '~3.8'

This command will guide you through creating your pyproject.toml config.


You can specify a package in the following forms:
  - A single name (requests)
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

％ poetry install
The currently activated Python version 3.9.0 is not supported by the project (~3.8).
Trying to find and use a compatible version.
Using python3 (3.8.6)
Creating virtualenv sample-proj-WINP6DfI-py3.8 in /Users/fsouza/Library/Caches/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file
％ /Users/fsouza/Library/Caches/pypoetry/virtualenvs/sample-proj-WINP6DfI-py3.8/bin/python --version
Python 3.9.0

The interesting bits are:

• poetry correctly complains about my current version of Python being not supported
• poetry correctly finds a supported version
• poetry incorrectly creates the virtualenv with the wrong version

For now, as a workaround, I've been manually managing my virtualenvs, but I wonder if this is actually considered a bug and if so, if yall would be willing to take a PR fixing it. I'm happy to try and take a stab at it.

[fsouza]

@cjw296 can you check if #3283 fixes the behavior you're observing?

[fsouza]

After reading this again, I realize I have a different issue. Opened #3284.