Poetry: No method to control the python version used

[epage]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10
• Poetry version: 0.12.2

Issue

I found it confusing when my pyproject.toml file says python = "3" and yet a Python 2.7 venv was being created.

I then noticed

The created virtualenv will use the Python executable for which poetry has been installed.
on https://poetry.eustace.io/docs/basic-usage/

I assumed it was using the version of python I ran get-poetry with. I tried again with python3 and still no luck.

It seems its just hard coded in poetry.bat as python and for me to change, I need to change that file.

To me, the ideal thing would be to a way to get arbitrary versions of python, like tox, and allow it to be configured on a per-project basis (like as a dev-dependency).

Workround

Edit poetry.bat to change what python gets invoked.

[xmo-odoo]

To me, the ideal thing would be to a way to get arbitrary versions of python, like tox, and allow it to be configured on a per-project basis (like as a dev-dependency).

And specify the "current python" when running impacted commands e.g. run: tox is good for running automated tests, but if you want to run the project to debug an issue it's not really helpful as that's not its purpose.

Having a way to e.g. poetry run -p 3.5 <script> and have it set up or reuse a previously set up Python 3.5 virtualenv would be super useful. The default should be whatever is specified by the toml file I guess?

[epage]

There are ways to do experimentation in tox but they are a bit confusing and hacky if your problem is user reported and not a tox test failure. Yeah, it seems like it'd make sense to be able to control this on the command line.

Another perspective on this is Rust's toolchain file. Users can pass in a --target flag to specify their toolchain but a file on disk can change what the default toolchain is.

[sdispater]

@epage

The created virtualenv will use the Python executable for which poetry has been installed.
on https://poetry.eustace.io/docs/basic-usage/

This is no longer true (I forgot to update this part of the documentation).

As of version 0.12.0, the poetry script will use the currently activated Python version to create the virtualenv.

Basically, you want to use a tool like pyenv to switch between Python versions, or use a project-specific .python-version so that pyenv knows which Python version you want for your project.

Making Poetry a Python versions manager is not a direction I want to go in since there are tools that do it already.

[epage]

Nice! Looks like that solves the underlying requests which is fine than its different than the suggestion.

So it seems that this is now just down to documentation updates?

[epage]

Drat, doesn't look like pyenv supports Windows.

[sdispater]

I haven't used it but maybe https://docs.python.org/3/using/windows.html#python-launcher-for-windows can help here.

[mmerickel]

I'm experiencing issues here as well when trying to bootstrap a project that requires a different version of python than that which I used to install poetry.

1) poetry should be checking that the running python is compatible with the python= requirements in pyproject.toml and raising an exception if not. This should prevent you from accidentally doing a poetry install using python 2.7 when you have python = 3.6 in your pyproject.toml.

2) poetry really needs to support some sort of --python <PATH> on the commands otherwise poetry's virtualenv creation is semi-worthless afaict. The user interface around this is of course a bit wonky because you don't want the user to have to specify --python $HOME/.pyenv/versions/3.6.6/bin/python for every single command, so poetry needs some memory here. I think pipenv actually does a pretty good job at this by creating a virtualenv named after the hash of the path to the folder. Thus if you find a virtualenv at $cache/$hash_of_path then it should use that one with whatever python that one contains. This means it's remembered the first time you run poetry install --python $home/.pyenv/versions/3.6.6/bin/python. If a different --python is specified there's a open question what to do but probably deleting the old venv and creating a new one.

Another approach would be to let the user create the virtualenv themselves. I know this is supported via the in-project setting but I think the defaults are weird around it right now. Ideally this would be a global opt-out setting where poetry automatically detects the folder by default and uses it. Right now the setting means poetry will always use .venv if turned on instead of letting me select it on a per-project basis.

[sdispater]

@mmerickel

I'm experiencing issues here as well when trying to bootstrap a project that requires a different version of python than that which I used to install poetry.

That's why you should now use the recommended installer which will make Poetry pick up the currently activated Python version. That makes it particularly handy when coupled with pyenv.

poetry should be checking that the running python is compatible with the python= requirements in pyproject.toml and raising an exception if not. This should prevent you from accidentally doing a poetry install using python 2.7 when you have python = 3.6 in your pyproject.toml.

That's planned actually since that's something that has bitten users before.

poetry really needs to support some sort of --python on the commands otherwise poetry's virtualenv creation is semi-worthless afaict.

I have said it before but I have no plan to make Poetry a Python version manager. pyenv exists and does the job really well and if you use the recommended installer pyenv local (or pyenv shell for that matter) does the same thing as a potential --python option.

Right now the setting means poetry will always use .venv if turned on instead of letting me select it on a per-project basis.

Just so you know, I plan on adding support for per-project settings via the config command.

[jaswilli]

I assumed it was using the version of python I ran get-poetry with. I tried again with python3 and still no luck.

Yes, this change just bit me on my build servers (Ubuntu 18.04) where I have always installed poetry via curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python3 in order to use py3.

Are you saying that going forward that absolutely will not work and I will need to bootstrap pyenv on my build workers in order to use poetry?

[sdispater]

@jaswilli Not necessarily. If your default python points to Python 3, Poetry will pick it up and there is nothing you have to do. However, if your default Python does not point to Python 3 you have to switch to Python 3 for Poetry to use it.

[mmerickel]

I have said it before but I have no plan to make Poetry a Python version manager. pyenv exists and does the job really well and if you use the recommended installer pyenv local (or pyenv shell for that matter) does the same thing as a potential --python option.

@sdispater I 100% agree about avoiding integrating any sort of pyenv support into poetry. The --python flag is an escape hatch to avoid doing that, by allowing the user to specify which python to use without making any assumptions.

That's why you should now use the recommended installer which will make Poetry pick up the currently activated Python version. That makes it particularly handy when coupled with pyenv.

Poetry is shipped on PyPI and as such supports being installed into a virtualenv. It's an official installation method. Decoupling poetry from the version of python it's running on should really be supported.

I don't have a habit of switching the default python in my shell and I consider it a bad practice. My "python" command in my shell always points at the system python. This is why things like poetry run are so great is that it doesn't change that... it just temporarily activates a virtualenv for the duration of the command. I'm hoping that poetry can do a better job supporting that for the poetry install command as well, as it can activate the virtualenv defined and run pip inside of it to install the packages. The actual work that poetry needs to do inside the user's virtualenv is fairly minimal and poetry is already doing it in a subprocess by shelling out to pip, the rest of the logic can be done on whatever python that poetry is running on.

[rnwolf]

These steps work for me.
I am using:

• Ubuntu 18.04,
• with pyenv
  
  
  
  • and pyenv-virtualenv https://github.com/pyenv/pyenv-virtualenv. This is configured to automatically activate/deactivate python on entry/exit to directory.
  
  
• poetry installed via pipsi (This is using python 3.6.6)

I want an environment with python 3.7.0

# List all available python versions
pyenv install -l
# Install release
pyenv install 3.7.0
# Create working directory
mkdir myapp-project
cd myapp-project
pyenv virtualenvs
pyenv local 3.7.0

# Now we create a virtual env based on the version of python specified above.
# The following will also install all of the default packages as specified in ~/.pyenv/default-packages
# These default packages may be used to install useful commands inside of the project, such as black for example.
pyenv virtualenv myapp
pyenv local myapp
poetry new --src myapp
cd myapp
echo 'Python in virtual environment should be: '$HOME'/.pyenv/versions/myapp/bin'
sed -i 's/^python\ =\ \".*/python\ =\ \"3.7.0\"/' pyproject.toml
poetry install

As I type this out it occurs to me that I could most probably just add poetry to ~/.pyenv/default-packages that would mean I don't need to install via pipsi.

[butla]

I'm just evaluating poetry today and for my normal usecases being able to easily work with a couple of Python versions without shuffling some global state around is absolutely crucial.

I often worked with people who aren't experts on Python and its wacky (even though it performs very well, in my opinion) way of isolating dependencies and runtime versions with virtualenvs. So I had to explain that, and debug and set it up on people's machines, and on CI servers. But it was doable. Pyenv would complicate that, I'm affraid.

On Ubuntu machines we'd just install whatever Pythons we need from deadsnakes, on Macs that'd be something from Homebrew. With Docker images we just pick the version of python or python-alpine we need, but the command in them is often python3 not python. Then a project would have a tox.ini with a specific version of Python (only one in most cases). Anyone could just pull the project, run tox, activate the virtualenv and work, so that was very nice.

And we could maintain some legacy projects in 2.7, stable stuff in 3.6, and evaluating 3.7.

@sdispater If what I'm talking about is outside of the scope of this project, then maybe we can create some disclaimer in the docs saying what workflows this project isn't for? Cause for me "Python dependency management and packaging made easy" is a bit of a false advertising right now :)

But I'll be sad if I can't use poetry in the future, cause it seems way well thought through and performant than Pipenv, which I use right now in a few projects, while disliking its confusing and non-standard CLI.

[sdispater]

@butla

Cause for me "Python dependency management and packaging made easy" is a bit of a false advertising right now :)

How so? The tagline does not mention Python versions management so I don't see how it is false advertising.

then maybe we can create some disclaimer in the docs saying what workflows this project isn't for?

It's explicitly stated in the documentation how Poetry should be used: https://poetry.eustace.io/docs/#installation or https://poetry.eustace.io/docs/basic-usage/#poetry-and-virtualenvs

Then a project would have a tox.ini with a specific version of Python (only one in most cases). Anyone could just pull the project, run tox, activate the virtualenv and work, so that was very nice.

I don't see how Poetry prevents you from doing that. You can use tox with Poetry see an example here: https://github.com/sdispater/tomlkit/blob/master/tox.ini

Now, I will say it again: Poetry is not and will never be a Python versions manager. It may be that Pipenv does it and people are now used to it but this is not the role of a package/dependency manager. And this choice has precedence since that's what a lot of package managers do in other languages: composer for PHP, bundler for Ruby or npm for Javascript.

[butla]

@sdispater I think you misunderstood me and I misunderstood the relation of poetry to virtualenvs. I thought that it manages virtualenvs (and I find them to be crucial for Python dependency management), quite like Pipenv does by creating its own, but now I see that it doesn't have to do that. The usecase you've linked to with tox should work perfectly for me, so probably I'll switch from Pipenv back to tox, but start using poetry in place of pip-tools.

Sorry about the confusion, but I was confused as well :)

[sdispater]

@butla Poetry does create virtualenvs to install the packages needed for the project in order to work isolated. The difference with PIpenv is that Poetry will pick up the currently activated Python version to create or use the virtualenvs.

[butla]

@sdispater Well, yeah, but that it'll only create a virtualenv for the Python versions it's installed with, or the one set by pyenv, which is weird for me. Switching my user python with pyenv all the time seems like not as nice of a user experience. But I've noticed a lot of Mac people using pyenv, so that might be an OS culture difference? I don't know. For me and a lot of other devs I know it's natural to have multiple versions system-wide as python, python3, python3.4, python3.7 etc. But virtualenvs don't need to know about that.

Anyway, since were on managing virtualenvs - with virtualenv or virtualenvwrapper I can specify the python version I want as a parameter. Here I don't see that option. If I do poetry init (poetry new doesn't ask me questions, even though it has a --no-interaction option) and select the python version I want (different from the one poetry was installed with), then do poetry add (which seems like a reasonable workflow) I'll be greated with an error:

[RuntimeError]
The current Python version (2.7.15) is not supported by the project (3.6)
Please activate a compatible Python version.

And I know that this might be totally coherent with the way this project is supposed to be used, but it's just surprising for me, and probably other people with habits similar to mine.

[mmerickel]

This workflow basically means I need to install poetry into every virtualenv I use and create the virtualenvs myself and activate the virtualenvs myself. This is objectively worse than pipenv which creates the virtualenv for me using the python interpreter that I tell it to use. Note again, this has nothing to do with pyenv or any other python version management being integrated into poetry. I just want the ability to tell poetry which python to use explicitly when it creates the virtualenv instead of being coupled to the version of python running poetry. Right now I have two options:

1) Set the in-venvs setting to true globally. Then create a .venv manually using the python I want, via /path/to/python -m venv .venv, then use poetry and have it use the .venv I created.

2) Create a virtualenv where I want (out of source preferably in the cache folder), then activate it, install poetry into it, then use that poetry.

Proposed workflow:

1) poetry install --python /path/to/python and poetry creates the virtualenv in the caches folder and on subsequent runs of poetry run or poetry install it uses that virtualenv automatically.

[butla]

@mmerickel You have no reason to install poetry into every virtualenv. The global installation will detect and work with whatever virtualenv you have activated. So it's

1. Create a virtualenv where I want (out of source preferably in the cache folder), then activate it, install poetry into it, then use that poetry.

but without installing poetry into the virtualenv, just using the global one. So it's not as dramatic as you describe it.

[dalito]

@epage On windows you can change poetry.bat to execute the launcher (replace "python" by "py") and use an environment variable to specify which of the installed Python versions should be used:

> set PY_PYTHON=3.7
> poetry install

...will create a Python 3.7 virtual env.

It would be nice if the launcher could be configured via a py.ini file in the project directory. This is not possible at the moment but it would not be difficult to write a Python 2/3 compatible script to look for such a py.ini-file. In poetry.bat this py.ini-finder should then be run as first step with whichever python version is executed before the launcher is used to run poetry with the python version specified in py.ini.

[mikelane]

I have multiple versions of python installed and activated on my machine (a mac using pyenv). This breaks poetry:

$ pyenv versions
  system
* 2.7.15 (set by /Users/mike/.pyenv/version)
* 3.7.1 (set by /Users/mike/.pyenv/version)
$ python -V
Python 2.7.15
$ python3 -V
Python 3.7.1
$ cat pyproject.toml
[tool.poetry]
name = "stuff"
version = "0.1.0"
description = "stuff"
authors = ["Mike Lane <[email protected]>"]
license = "MIT"

[tool.poetry.dependencies]
python = "^3.6"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
$ poetry add django

[RuntimeError]
The current Python version (2.7.15) is not supported by the project (^3.6)
Please activate a compatible Python version.

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

Would be nice if I could at least specify which python to use for poetry somehow.

[garyo]

If you run poetry in a virtualenv with your desired python, it should work. (Not saying that's the way it ought to be, but it should work anyway.)

[matthewfeickert]

For those who don't want to go an edit the shebang in $HOME/.poetry/bin/poetry this function worked well when used before poetry commands as part of a installation script to setup and install an environment with Poetry on machines that had python being Python 2 and python3

function ensure_python3 {
    # Check if `python` evalutes to Python 2 or Python 3
    if [[ "$($(which python) -c 'import sys; print(sys.version_info[:][0])')" -lt 3 ]]; then
        # Check if there is a venv in the working directory with a Python 3 environment installed
        if [[ ! -z "$(find . -wholename "*/bin/python3")" ]]; then
            local python3_venv_bin_dir="$(dirname "$(find . -wholename "*/bin/python3")")"
            source "${python3_venv_bin_dir}/activate"
        else
            printf "\nPlease activate a Python 3 virtual environment and then rerun\n\nbash scripts/install_env.sh\n\n"
            exit 1
        fi
    fi
    return 0
}

This isn't helpful for people who want to have their day to day activities be uninterrupted, but you're probably already in virtual environments most of the time anyway(?) so that hopefully isn't too big of a deal. If you don't want to use pyenv then you can also just use this small wrapper function in your shell to quickly get Python 3 virtual environments up.

[nealmcb]

Like many others, I have found the the current situation with Python versions confusing, and hope to clarify or pin some things down here.

Many operating systems (e.g. Ubuntu) support multiple Python versions simply by having official versions with different names, e.g. python for Python 2 and python3 for Python 3. Recent versions of Ubuntu make Python 2 optional, but still call it python.

These systems have multiple Python installations. So using a phrase in the Poetry documentation like use the current Python installation is ambiguous. It would at the very least be helpful to clarify that the new behavior of version 0.12.x is to always search $PATH for the first thing called python.

There can even be system tools (or project scripts) that assume that python is Python 2. Forcing Poetry users to change the "OS API" at that level can be disruptive and dangerous. I tend to avoid running tools like pyenv which alter these assumptions unless I really need them.

For these sorts of reasons, most other Python tools provide methods to control which version of Python they are used with. It doesn't seem complicated, and they don't portray that as becoming "Python version managers". They just allow command-line arguments or configuration settings to specify the name of the executable to use for Python.

So like many others, I hope to see that supported in Poetry. And it sounds like previous versions did essentially have an easy way to get a Poetry installation that that worked by default with python3 - is that right?

In the meantime, it would be helpful to build on the post by @matthewfeickert to explore approaches that result in the smallest impact to users while providing the flexibility we seek. These seem like possible options, and I'd like to know which might work reliably:

• Change the shebang in $HOME/.poetry/bin/poetry to #!/usr/bin/env python3, assuming we want to just permanently transition to python3.
• Establish a shell alias for poetry (or poetry3) which runs $HOME/.poetry/bin/poetry via e.g. python3.
• Temporarily change the meaning of python just for the duration of the script, as @matthewfeickert's script does. I guess that might be necessary if Poetry also uses python internally to do other things.

[matthewfeickert]

• Establish a shell alias for poetry (or poetry3) which runs $HOME/.poetry/bin/poetry via e.g. python3.

This seems like the most user friendly option, but requires the most work from whoever is going to write the PR. Also, as 2020 and the end of Python 2.7 support looms it might be worth considering if making the shell alias poetry3 might add confusion/future API breaking changes when Poetry eventually switches to being a Python 3 centric project (i.e., when the $HOME/.poetry/bin/poetry shebang is changed to #!/usr/bin/env python3 in master).

I realize that's not a very substantive comment, but just my thoughts.

[ulope]

To jump in on this, @sdispater you're very often using the phrase current python the problem with that is, that (AFAIK) outside of pyenv that just isn't a thing.
For most other methods of installing Python (e.g. pythonz, homebrew, system package manager, compiling from source, etc.) there exists no 'current' Python. Just a bunch of Python executables with different names somewhere in the $PATH.

For example on my machine (where I use pythonz) I have:

• python2, python2.7
• python3, python3.7
• python3.6
• python3.5

/edit:
To expand on this. This works pretty seamless with virtualenvwrapper:

mkvirtualenv -p $(which python3.7) somevenv

From that point forwards everything in that venv / project (the two concepts are kind of linked in my use) will be on Py3.7 without having to modify 'global' state.

[dimaqq]

That's why you should now use the recommended installer which will make Poetry pick up the currently activated Python version. That makes it particularly handy when coupled with pyenv.

From the user perspective, this is a big problem. It is for me too.

Typically, poetry is installed outside of the project virtualenv. This allows for poetry init as well as poetry publish as well as perhaps poetry run tox, etc. This makes poetry a system-wide tool. Thus, for an average poetry user (i.e. non-poetry-dev, rather some-lib-dev or tester or integrator), there should (IMO) be only one poetry installed system-wide.

At the same time, different projects require different Python versions.

I feel there's a conceptual clash here — poetry init doesn't strictly care what python is used to interpret poetry, it may and perhaps should be the system Python. Meanwhile poetry run entrypoint very much must ensure that user code is interpreted by correct Python version.

Other tools a) are happy to use one of the system Python binaries (-m venv; virtualen; pipenv), read the version from the project spec and find python in $PATH (tox, pipenv), as well as allow to specify highly-specific Python build (-m venv, pipenv --python=, pyenv virtualenv).

In the ideal world, I'd use python3 (whatever's latest) for both poetry and user code.

In the practical wold, some projects are stuck on 3.6; some unlucky dev may run across a project written for python 2.x and it will be their job to update the project to work on 3.x, which is often an incremental process, during which the developer might even alternate between 2.x and 3.x branches in the repository.

[treadup]

This is a bit of a hack but if you are on Ubuntu, are not using pyenv and want projects created with poetry to use python3 then you can use the following command when installing poetry.

curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py  | \
sed 's_/usr/bin/env python_/usr/bin/env python3_g' | python3

I'm not a pyenv expert but pyenv does create a shim for python3 so you should still be able to use pyenv to select python version.

Don't use the above command if you are planning on using python2.

[ulope]

FYI: As a stopgap I'm for now using this shell function (ZSH specific, sorry) to wrap poetry and this basically solves 95% of the problem for me.

[jimrthy]

FWIW, I don't like pyenv at all. It doesn't seem to provide enough benefit to justify how much it interferes with my workflow. (This may be because I have a long history of coping with lots of different python versions that I had to compile myself and then manage in lots of different virtualenvs because pyenv hadn't been written yet.)

I'm obviously in the group with lots of different python versions available on my PATH. And I'm using virtualenvwrapper.

Be that as it may, I've had very good luck using python3.x -m venv ~/.virtualenvs/foo to generate the venv. Then I call workon foo to use it. I can run my [globally installed] poetry in that environment as I see fit.

To me that seems less burdensome than pyenv (maybe I just need to give it more of a chance).

[HolgerPeters]

I have said it before but I have no plan to make Poetry a Python version manager. pyenv exists and does the job really well and if you use the recommended installer pyenv local (or pyenv shell for that matter) does the same thing as a potential --python option.

pyenv is not a solution for

• people who work under windows
• people who don't have a compiler installed on their machines (some can't because of IT settings)
• people who want to use a very specific python build (Let's say a python built with jmalloc).

A --python flag should IMHO be supported.

[CptSpaceToaster]

Looks like we'll get what we need when 1.0 releases: https://github.com/sdispater/poetry/pull/731 has already been merged in and provides poetry env use /full/path/to/python

[sdispater]

Like said in the comment above, there is now a env group of command that make it possible to better manage Python versions used by a project and Poetry will now try to find compatible Python version to use (see #731 and #1477).

You need to use the latest beta release to be able to use this.