Poetry: Create an implicit virtual environment based on pyproject.toml and following PEP 394

[maggyero]

Issue

_In the absence of an activated virtual environment_ (I insist), the poetry add, poetry build, poetry install, poetry lock, poetry remove, poetry run, poetry shell, poetry show and poetry update commands _implicitly_ create a Poetry _built-in_ virtual environment. However currently, the Python interpreter that they include in this virtual environment is the one called by the #!/usr/bin/env python shebang line of the ~/.poetry/bin/poetry script. The problem is that on MacOS and most Linux distributions, /usr/bin/env python resolves to the Python 2.7 interpreter.

So if for instance a user specified in his pyproject.toml file a dependency on the Python 3.7 interpreter:

[tool.poetry.dependencies]
python = "^3.7"

when he runs one of the poetry add, poetry build, poetry install, poetry lock, poetry remove, poetry run, poetry shell, poetry show and poetry update commands for the first time, a RuntimeError is raised:

[RuntimeError]
The current Python version (2.7.14) is not supported by the project (^3.7)
Please activate a compatible Python version.

This is a terrible user experience. Many users have been repeatedly complaining about this since Poetry's inception, so this issue cannot be ignored: https://github.com/sdispater/poetry/issues/77, https://github.com/sdispater/poetry/issues/305, https://github.com/sdispater/poetry/issues/522, https://github.com/sdispater/poetry/issues/655, https://github.com/sdispater/poetry/issues/721, https://github.com/sdispater/poetry/issues/952, https://github.com/sdispater/poetry/issues/988.

Solution

In https://github.com/sdispater/poetry/issues/988, @Suor made an excellent suggestion that could address this issue:

• add a command that allows the _explicit_ creation of Poetry built-in virtual environments with an option to specify the Python interpreter;
• use the Python interpreter specified in the tool.poetry.dependencies.python property of the pyproject.toml file for the _implicit_ creation of Poetry built-in virtual environments.

In PR https://github.com/sdispater/poetry/pull/731 that will be part of Poetry 1.0.0, @sdispater added a new poetry env command that implements the first point, which is great. Thank you for that @sdispater. But that only addresses half of the problem, the _explicit_ virtual environment creation problem. That does not address the _implicit_ virtual environment creation problem that still raises a RuntimeError.

If we also implemented the second point I think it would greatly improve user experience.

[Edit] Specification for the second point

We should just stick to PEP 394:

Distributors may choose to set the behavior of the python command as follows:

• python2,
• python3,
• not provide python command,
• allow python to be configurable by an end user or a system administrator.

This means that Poetry should only look up python and pythonX in the OS PATH (not pythonX.Y nor pythonX.Y.Z), that is to say Poetry should not be more specific than the _major_ Python version. And since we want to look up the _highest_ Python version for consistency with the dependency lookup mechanism, we should hard code in Poetry the _current highest_ Python major version released (3). With the relatively slow Python release cycle, there will be practically no maintenance: just a number to update every 10 years or so (Python 3 was released in 2008 and Python 4 will not be released before 2020, and possibly never).

So with the following pyproject.toml requirements, Poetry will use the following PATH lookup orders:

• python = ">=3.4.1": [python3, python];
• python = ">=2.6.3": [python3, python2, python];
• python = ">=2.6.3,<3.0.0": [python2, python].

When Python 4 is available, we will update the current highest Python major version released to 4 in Poetry, so the PATH lookup orders of Poetry will become:

• python = ">=3.4.1": [python4, python3, python];
• python = ">=2.6.3": [python4, python3, python2, python];
• python = ">=2.6.3,<3.0.0": [python2, python].

If no pythonX or python is found in the PATH, the following error message will be raised:

[RuntimeError]
No current Python version.
Please activate a compatible Python version explicitly with poetry env use <python> or another virtual environment manager.

If a pythonX or python is found in the PATH but its version (given by pythonX -V and python -V respectively) does not match the pyproject.toml requirements, the following error message will be raised:

[RuntimeError]
The current Python version (2.7.14) is not supported by the project (^3.7).
Please activate a compatible Python version explicitly with poetry env use <python> or another virtual environment manager.

(Example given with a pyproject.toml requirement of python = "^3.7" and an OS without python3 in the PATH and a python in the PATH pointing to Python 2.7.14.)

[sdispater]

That's something that I already mentioned before but, for instance, ^3.4 means >=3.4,<4.0 so which Python version should Poetry choose? The highest one available? The lowest? There is no clear answer and letting Poetry guess might not be the best approach here.

Now, maybe we could improve the error message by mentioning the env command.

[David-OConnor]

Agree on mentioning the env command.

Recommend hard-setting the Python version rather than making it a semvar requirement.

[maggyero]

@sdispater

That's something that I already mentioned before but, for instance, ^3.4 means >=3.4,<4.0 so which Python version should Poetry choose?

Contrary to any other dependencies, the _lowest_ version available I guess. Because otherwise I am not sure how Poetry would do the lookup. For instance with the >=3.4,<4.0 constraint, which file would Poetry look up in the OS PATH environment variable first? python3.9? What if there is a python3.10 or python3.11?

And even if Poetry looked up only by the major version number, it would have the same issue for constraints without upper limits. For instance with the >=3.7 constraint, where would Poetry start? python3? What if there was a python4 or python5? So unless there is a reliable way to get the highest version available, I think we have to choose the lowest version available.

So here are a few examples showing how I see the lookup order (of the lowest version available):

• >=3.4.1: [python3.4.1, python3.4, python3, python];
• >=3.4: [python3.4, python3, python];
• >=3: [python3, python];
• >=2.7.10: [python2.7.10, python2.7, python2, python];
• >=2.7: [python2.7, python2, python];
• >=2: [python2, python];
• >3.4.1: [python3.4.2, python3.4, python3, python];
• >3.4: [python3.5, python3, python];
• >3: [python4, python];
• >2.7.10: [python2.7.11, python2.7, python2, python];
• >2.7: [python2.8, python2, python];
• >2: [python3, python];
• <=3.4.1: [python];
• <=3.4: [python];
• <=3: [python];
• <=2.7.10: [python];
• <=2.7: [python];
• <=2: [python];
• <3.4.1: [python];
• <3.4: [python];
• <3: [python];
• <2.7.10: [python];
• <2.7: [python];
• <2: [python];
• >=3.4.1,<4.0.0: [python3.4.1, python3.4, python3, python];
• >=3.4,<4.0: [python3.4, python3, python];
• >=3,<4: [python3, python];
• >=2.7.10,<3.0.0: [python2.7.10, python2.7, python2, python];
• >=2.7,<3.0: [python2.7, python2, python];
• >=2,<3: [python2, python];
• >3.4.1,<4.0.0: [python3.4.2, python3.4, python3, python];
• >3.4,<4.0: [python3.5, python3, python];
• >3,<4: [];
• >2.7.10,<3.0.0: [python2.7.11, python2.7, python2, python];
• >2.7,<3.0: [python2.8, python2, python];
• >2,<3.0: [].

As you can see, prior to trying python like currently, Poetry will try more specific commands based on the version constraints given in the tool.poetry.dependencies.python property of the pyproject.toml file. What do you think?

Now, maybe we could improve the error message by mentioning the env command.

+1 Yes, definitely.

But before failing with this error message, I think that Poetry should try several commands instead of merely python, using the implicit lookup described above.

[Edit] Forget all of this and see my initial post for the proposed solution.

[David-OConnor]

Here's one approach to consider:

Attempt to run python -V, python3 -V, and pythonA.B -V, where A and B are each possible python version. Parse stdout and stderr - note that old versions of Python output this to stderr. Use this to compile a list of Python versions on the PATH, and their aliases. Match the compatible-with-pyproject.toml subset's length: 0 means abort with an error. 1 Means use that alias. Multiple means ask the user.

Note that this won't pick up Python installations not on the PATH.

[stinovlas]

That's something that I already mentioned before but, for instance, ^3.4 means >=3.4,<4.0 so which Python version should Poetry choose?

Contrary to any other dependencies, the _lowest_ version available I guess. Because otherwise I am not sure how Poetry would do the lookup. For instance with the >=3.4,<4.0 constraint, where would Poetry start? python3.9? What if there is a python3.10 or python3.11?

If you want to use the lowest version possible, does it mean that you would prefer 3.6.0 to 3.6.1 if both are installed?

[maggyero]

@stinovlas

If you want to use the lowest version possible, does it mean that you would prefer 3.6.0 to 3.6.1 if both are installed?

If you have the ^3.4 (>=3.4,<4.0) constraint in your pyproject.toml file, Poetry would use that lookup order: [python3.4, python3, python]. So the Python version that would be called depends on which Python version python3, and if it does not exist, python, point to on your machine (assuming python3.4 does not exist).

[stinovlas]

@maggyero

If you have the ^3.4 (>=3.4,<4.0) constraint in your pyproject.toml file, Poetry would use that lookup order: [python3.4, python3, python]. So the Python version that would be called depends on which Python version python3, and if it does not exist, python, point to on your machine (assuming python3.4 does not exist).

Oh, i see, so you are just trying different binaries in PATH, from the most specific to the least specific.

So, let's say that I have python >= 3.6 specified in my pyproject.toml. According to you, the resolution order would be [python3.6, python3, python]. Now, let's assume that I have python2.7, python3.5 and python3.7 installed on my system. Binary python3 is pointing to python3.5 and binary python is pointing to python2.7.

• The first option (python3.6) does not exist, so we skip it.
• The second option (python3) points to python3.5, so we should not create virtualenv with it (project requires python >= 3.6)
• The third option (python) points to python2.7, so we should not create virtualenv with it (same as above)

It seems that there is no option left. But there is! We also have python3.7 installed on our system, which satisfies the project requirement.

[David-OConnor]

What does the alias matter? It's the associated Python version that's significant. Search all possible aliases for associated versions, and pick the one with the newest compatible version.

[stinovlas]

What does the alias matter? It's the associated Python version that's significant. Search all possible aliases for associated versions, and pick the one with the newest compatible version.

I can agree with that. But if that's the case, then the argument against using the latest version possible is no longer valid.

[stinovlas]

@David-OConnor I like the resolution you suggested before (letting the user choose when there are multiple versions available), I'm just disputing the "use the minimal version" attitude :-).

[dmontagu]

Contrary to any other dependencies, the lowest version available I guess.

This seems very bad to me. I think the interpretation of ^3.4 should be the same for python as it is for any other dependency.

I don’t see why poetry shouldn’t treat python the same as any other dependency in this regard — if python 3.9, 3.10, etc are available on the path then use it, the same as you would if dependency X was pinned in such a way that v3.9 was the most recent compatible version available on PyPI.

If you want to impose a cap on the python version then just mark it as such.

I think it could make sense to be able to specify the desired version of python for a development environment as part of pyproject.toml, and to use it if possible (allowing and using compatible fallbacks), but I don’t think there should be an implicit python version selection process that differs from the standard dependency version selection process.

[maggyero]

@stinovlas

So, let's say that I have python >= 3.6 specified in my pyproject.toml. According to you, the resolution order would be [python3.6, python3, python]. Now, let's assume that I have python2.7, python3.5 and python3.7 installed on my system. Binary python3 is pointing to python3.5 and binary python is pointing to python2.7.

• The first option (python3.6) does not exist, so we skip it.
• The second option (python3) points to python3.5, so we should not create virtualenv with it (project requires python >= 3.6)
• The third option (python) points to python2.7, so we should not create virtualenv with it (same as above)

It seems that there is no option left. But there is! We also have python3.7 installed on our system, which satisfies the project requirement.

Yes of course, but how could Poetry know that python3.7 is installed? If you expect Poetry to increment the minor version number until it finds a Python interpreter (so [python3.6, python3.7, python3.8, …]), where should it stop if there is no Python 3 interpreter installed on the system? That is why I said that Poetry should only look for the version specified in the pyproject.toml file, from the most specific to the least specific. So for the python = ">=3.6" constraint, as you said, the lookup order would be: [python3.6, python3, python]. If these files do not exist in the PATH environment variable, Poetry should display the error message suggesting to specify the Python interpreter explicitly with the poetry env command.

[maggyero]

@dmontagu

Contrary to any other dependencies, the lowest version available I guess.

This seems very bad to me. I think the interpretation of ^3.4 should be the same for python as it is for any other dependency.

I don’t see why poetry shouldn’t treat python the same as any other dependency in this regard — if python 3.9, 3.10, etc are available on the path then use it, the same as you would if dependency X was pinned in such a way that v3.9 was the most recent compatible version available on PyPI.

I would also prefer Poetry to find the highest Python version available in the OS PATH environment variable, instead of the lowest for consistency with other dependencies. But how would you do that reliably? If the constraint is python = ">=3.6", do you start to look for a python3.9 file in the PATH? Or a python3.24 file? Or a python5.3? For library dependencies that is different, PyPI API can give you the last version, but here we don't use PyPI API, we use the OS PATH environment variable.

[David-OConnor]

@David-OConnor I like the resolution you suggested before (letting the user choose when there are multiple versions available), I'm just disputing the "use the minimal version" attitude :-).

I agree - lowest version is inconsistent with what a user expects from semvar, and how dependency versions are chosen.

[David-OConnor]

Yes of course, but how could Poetry know that python3.7 is installed? If you expect Poetry to increment the minor version number until it finds a Python interpreter (so [python3.6, python3.7, python3.8, …]), where should it stop if there is no Python 3 interpreter installed on the system?

3.12 or so, so we won't have to change it for a while. It's computationally quick to check these. Re 3.24 and 5.3: Could go up to 3.24, but no point. 5.3 isn't possible.

For library dependencies that is different, PyPI API can give you the last version, but here we don't use PyPI API, we use the OS PATH environment variable.

We know what versions of Python are avail, and expected to be released soon.

[dmontagu]

Yeah, my point wasn't that I think poetry needs to support finding exactly the right version in the PATH according to semver; it was just that if poetry is going to support this, I don't think it should be using different logic. (In particular, I would prefer there be no implicit python selection over choosing the minimum compatible.)

From a practical perspective, I personally would be fine with @David-OConnor 's suggestion of checking for a limited list of future versions (if it were sufficiently fast). I would also be fine with requiring some other form of explicit specification for the desired default environments (e.g., an extra key in the poetry config with value listing out the python executables to look for), or with some form of interactive selection (skippable in builds).

[bjoernpollex]

This has been discussed before, and other people have linked to this comment in other issues:

It has been discussed before and the decision remains: Poetry is not and never will be a Python management tool. This is something that tools like pyenv do really well and is not something a package/dependency manager should do.

I don't think this is going to happen, and in my personal experience, working with pyenv works very well for this, especially if you just add a .python-version file to your project.

Edit Hm, I've been behind the curve on 1.0.0 developments, apparently this is partially supported now, so ignore my comment!

[maggyero]

@David-OConnor

3.12 or so, so we won't have to change it for a while. It's computationally quick to check these. Re 3.24 and 5.3: Could go up to 3.24, but no point. 5.3 isn't possible.

5.3 is not _currently_ possible.

We know what versions of Python are avail, and expected to be released soon.

Yes, but that means we have to hard code this into Poetry and update it frequently (new Python _minor_ versions are released every year). It creates a maintenance burden. I would prefer a future-proof algorithm, that is why I suggested to look up the _lowest_ Python version instead of the _highest_ one. But this is not ideal.

@dmontagu

Yeah, my point wasn't that I think poetry needs to support finding exactly the right version in the PATH according to semver; it was just that if poetry is going to support this, I don't think it should be using different logic.

I agree.

Solution

The more I think about it, the more I think we should just stick to PEP 394:

Distributors may choose to set the behavior of the python command as follows:

• python2,
• python3,
• not provide python command,
• allow python to be configurable by an end user or a system administrator.

This means that Poetry should only look up python and pythonX in the OS PATH (not pythonX.Y nor pythonX.Y.Z), that is to say Poetry should not be more specific than the _major_ Python version. And since we want to look up the _highest_ Python version for consistency with the dependency lookup mechanism, we should hard code in Poetry the _current highest_ Python major version released (3). With the relatively slow Python release cycle, there will be practically no maintenance: just a number to update every 10 years or so (Python 3 was released in 2008 and Python 4 will not be released before 2020, and possibly never).

So with the following pyproject.toml requirements, Poetry will use the following PATH lookup orders:

• python = ">=3.4.1": [python3, python];
• python = ">=2.6.3": [python3, python2, python];
• python = ">=2.6.3,<3.0.0": [python2, python].

When Python 4 is available, we will update the current highest Python major version released to 4 in Poetry, so the PATH lookup orders of Poetry will become:

• python = ">=3.4.1": [python4, python3, python];
• python = ">=2.6.3": [python4, python3, python2, python];
• python = ">=2.6.3,<3.0.0": [python2, python].

If no pythonX or python is found in the PATH, the following error message will be raised:

[RuntimeError]
No current Python version.
Please activate a compatible Python version explicitly with poetry env use <python> or another virtual environment manager.

If a pythonX or python is found in the PATH but its version (given by pythonX -V and python -V respectively) does not match the pyproject.toml requirements, the following error message will be raised:

[RuntimeError]
The current Python version (2.7.14) is not supported by the project (^3.7).
Please activate a compatible Python version explicitly with poetry env use <python> or another virtual environment manager.

(Example given with a pyproject.toml requirement of python = "^3.7" and an OS without python3 in the PATH and a python in the PATH pointing to Python 2.7.14.)

What do you think @sdispater?

[David-OConnor]

@David-OConnor

3.12 or so, so we won't have to change it for a while. It's computationally quick to check these. Re 3.24 and 5.3: Could go up to 3.24, but no point. 5.3 isn't possible.

5.3 is not _currently_ possible.

We know what versions of Python are avail, and expected to be released soon.

Yes, but that means we have to hard code this into Poetry and update it frequently (new Python _minor_ versions are released every year). It creates a maintenance burden. I would prefer a future-proof algorithm, that is why I suggested to look up the _lowest_ Python version instead of the _highest_ one. But this is not ideal.

Let's be practical: It will likely be decades before we encounter a python 5.3, if ever, at which point the ecosystem will have changed beyond recognition. It will be years before we encounter a Python 3.11. When we hit the latter, it'll be a trivial update.

This means that Poetry should only look up python and pythonX in the OS PATH (not pythonX.Y nor pythonX.Y.Z), that is to say Poetry should not be more specific that the major Python version.

Popular linux distros like Ubuntu distinguish using python3.6, python3.7 etc. Ignoring this will not meet the intent, which is to distinguish minor versions.

[maggyero]

@David-OConnor

Popular linux distros like Ubuntu distinguish using python3.6, python3.7 etc. Ignoring this will not meet the intent, which is to distinguish minor versions.

I think it is not Poetry's responsibility to look up minor versions. Poetry should only look up python, python2 and python3, as required by PEP 394. If you want a specific Python version you should tell it explicitly with poetry env use <python>. In addition, minor versions change every year, and the community plans to accelerate the cycle, so hard coding the latest minor version is a maintenance burden that is better avoided in my opinion. While the major versions change every 10 years and probably never again, so there is no maintenance. PEP 394 should be our reference.

[David-OConnor]

I understood your original issue as wanting to specify the environment's Python version version using pyproject.toml.

The current Python version (2.7.14)

current Python version is a trap, when you can (and commonly do on Linux) have multiple versions of Python installed. I'm in favor of specifying a version or semvar requirement in pyproject.toml, using that version if available, and displaying an error if not.

[maggyero]

@David-OConnor

I understood your original issue as wanting to specify the environment's Python version version using pyproject.toml.

My concern was that Poetry looks up python in the PATH, but not python3 or python2, which is contrary to PEP 394 recommendations (that I realized afterwards).

current Python version is a trap, when you can (and commonly do on Linux) have multiple versions of Python installed. I'm in favor of specifying a version or semvar requirement in pyproject.toml, using that version if available, and displaying an error if not.

I am against this as Poetry provides an explicit way to do it: poetry env use <python>. That <python> argument is _OS specific_ (it can be /usr/bin/python3.7, python3.7, 3.7) so it should not go into the pyproject.toml. Environment parameters should be outside the pyproject.toml.

[David-OConnor]

That argument is OS specific (it can be /usr/bin/python3.7, python3.7, 3.7) so it should not go into the pyproject.toml. Environment parameters should be outside the pyproject.toml.

I think the python-version should be project specific - If a project contains something like dataclasses or f-strings, you can specify 3.7 or ^3.7 in pyproject.toml.

[matthijskooijman]

It seems this issue has been addressed in #1477, so maybe this issue can be closed?

[maggyero]

Hi @sdispater I have just tested your PR #1477 with the latest Poetry (1.0.0b5) on MacOS Catalina (10.15.1). (Sorry for not providing any feedback earlier.) Poetry now correctly creates virtual environments based on the requirements of tool.poetry.dependencies.python in pyproject.toml! However it does not follow the specifications in my initial post (https://github.com/sdispater/poetry/issues/1386#issue-495076159, section "[Edit] Specification for the second point").

Consequently, with the value ">=2.7" for tool.poetry.dependencies.python in pyproject.toml, I get two different Python versions installed, _according to the Python running the poetry script_ (note: on my system, python and python2 points to Python 2.7 and python3 on Python 3.7):

• Python 2.7 is installed if poetry is run with Python 2.7:

[maggyero@MacBookPro~/Desktop/foo]$ python ~/.poetry/bin/poetry install 
Creating virtualenv foo-1XhTtlMR-py2.7 in /Users/maggyero/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.


Package operations: 1 install, 0 updates, 0 removals

  - Installing pika (1.1.0)
  - Installing foo (0.1.0)

• Python 3.7 is installed if poetry is run with Python 3.7:

[maggyero@MacBookPro~/Desktop/foo]$ python3 ~/.poetry/bin/poetry install   
Creating virtualenv foo-1XhTtlMR-py3.7 in /Users/maggyero/Library/Caches/pypoetry/virtualenvs
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.


Package operations: 1 install, 0 updates, 0 removals

  - Installing pika (1.1.0)
  - Installing foo (0.1.0)

Both installed Python respect the ">=2.7" requirements but I would expect Poetry to always install the same Python: the highest version available (Python 3.7 in this case), _regardless of the Python running poetry_.

[sdispater]

@maggyero This is the expected behavior. Poetry will always prioritize the currently used Python version. The reason for that is to support the use case of managing Python environments with pyenv for instance, like I do.

[maggyero]

@sdispater That makes sense. I am closing this issue. Thanks a lot for you hard work!