poetry appears to be trying to install with the wrong version of pip

[nottrobin]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Docker image: ubuntu:xenial (as root)
• Poetry version: 0.12.10
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/nottrobin/e104b9ec140cae4f403f381ed71b3307

Issue

In my system, I have both python and pip, and python3 and pip3 commands. (Installed with apt install python-pip python3-pip).

I installed poetry with pip3 install poetry, and you can see that it's using python3 as its binary:

$ head -n 1 `which poetry`
#!/usr/bin/python3

I've also configured it not to create virtualenvs, and I'm running this as root inside a ubuntu:xenial docker container.

So now my poetry install errors, because it claims not to have python3.5:

# poetry install -vvv
Skipping virtualenv creation, as specified in config file.
Installing dependencies from lock file


Package operations: 1 install, 0 updates, 0 removals, 1 skipped

  - Skipping pytz (2018.7) Already installed
  - Installing django (2.1.4)

[EnvCommandError]
Command ['/usr/bin/pip', 'install', '--no-deps', 'django==2.1.4'] errored with the following output:                                           
Collecting django==2.1.4                                                                                                                       
  Using cached https://files.pythonhosted.org/packages/83/f7/4939b60c4127d5f49ccb570e34f4c59ecc222949220234a88e4f363f1456/Django-2.1.4.tar.gz  
    Complete output from command python setup.py egg_info:                                                                                     

    ==========================                                                                                                                 
    Unsupported Python version                                                                                                                 
    ==========================                                                                                                                 

    This version of Django requires Python 3.5, but you're trying to                                                                           
    install it on Python 2.7.                                                                                                                  

    This may be because you are using a version of pip that doesn't                                                                            
    understand the python_requires classifier. Make sure you                                                                                   
    have pip >= 9.0 and setuptools >= 24.2, then try again:                                                                                    

        $ python -m pip install --upgrade pip setuptools                                                                                       
        $ python -m pip install django                                                                                                         

    This will install the latest version of Django which works on your                                                                         
    version of Python. If you can't upgrade your pip (or Python), request                                                                      
    an older version of Django:                                                                                                                

        $ python -m pip install "django<2"                                                                                                     

    ----------------------------------------                                                                                                   
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-dlrX8a/django/                                                   
You are using pip version 8.1.1, however version 18.1 is available.                                                                            
You should consider upgrading via the 'pip install --upgrade pip' command.                                                                     

Exception trace:
 /usr/local/lib/python3.5/dist-packages/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /usr/local/lib/python3.5/dist-packages/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /usr/local/lib/python3.5/dist-packages/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /usr/local/lib/python3.5/dist-packages/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /usr/local/lib/python3.5/dist-packages/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /usr/local/lib/python3.5/dist-packages/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /usr/local/lib/python3.5/dist-packages/poetry/console/commands/install.py in handle() at line 57
   return_code = installer.run()
 /usr/local/lib/python3.5/dist-packages/poetry/installation/installer.py in run() at line 76
   self._do_install(local_repo)
 /usr/local/lib/python3.5/dist-packages/poetry/installation/installer.py in _do_install() at line 287
   self._execute(op)
 /usr/local/lib/python3.5/dist-packages/poetry/installation/installer.py in _execute() at line 295
   getattr(self, "_execute_{}".format(method))(operation)
 /usr/local/lib/python3.5/dist-packages/poetry/installation/installer.py in _execute_install() at line 320
   self._installer.install(operation.package)
 /usr/local/lib/python3.5/dist-packages/poetry/installation/pip_installer.py in install() at line 91
   self.run(*args)
 /usr/local/lib/python3.5/dist-packages/poetry/installation/pip_installer.py in run() at line 112
   return self._env.run("pip", *args, **kwargs)
 /usr/local/lib/python3.5/dist-packages/poetry/utils/env.py in run() at line 354
   raise EnvCommandError(e)

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

But:

$ python3 --version
Python 3.5.2
$ pip3 --version
pip 8.1.1 from /usr/lib/python3/dist-packages (python 3.5)

[nottrobin]

I realise this is the 4th issue I've filed today. Sorry for the noise, I'm trying to be helpful rather than difficult.

I love the attitude of poetry (especially as compared to the attitude of Pipenv) and I want it to succeed, I'm just trying to see if I can make it work with our workflow.

I'll try to delve into this issue more if I get a chance, and maybe even submit a PR or two.

[nottrobin]

Ah, so after a little more investigation, it's clear Poetry always tries to use pip rather than pip3.

E.g. if I create a Dockerfile, to make my steps here clear:

FROM ubuntu:xenial

RUN apt-get update && apt-get install -y python3-pip python3-venv
RUN pip3 install poetry
RUN poetry config settings.virtualenvs.create false

WORKDIR /srv
RUN poetry init --no-interaction --dependency django
RUN poetry install

Then if I run it:

$ docker build .
Sending build context to Docker daemon  5.632kB
Step 1/7 : FROM ubuntu:xenial
 ---> e13f3d529b1a
Step 2/7 : RUN apt-get update && apt-get install -y python3-pip python3-venv
 ---> Using cache
 ---> a0b412396326
Step 3/7 : RUN pip3 install poetry
 ---> Using cache
 ---> 12402aa3434a
Step 4/7 : RUN poetry config settings.virtualenvs.create false
 ---> Running in b7803af228b6
Removing intermediate container b7803af228b6
 ---> e240daca2351
Step 5/7 : WORKDIR /srv
 ---> Running in 10740da04d08
Removing intermediate container 10740da04d08
 ---> 90186dd0b6b0
Step 6/7 : RUN poetry init --no-interaction --dependency django
 ---> Running in bed6d4262adc

This command will guide you through creating your pyproject.toml config.


Using version ^2.1 for django
Removing intermediate container bed6d4262adc
 ---> 2d10b043a3b3
Step 7/7 : RUN poetry install
 ---> Running in 76283c0673db
Skipping virtualenv creation, as specified in config file.

[FileNotFoundError]            
[Errno 2] No such file or directory: 'pip'  

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

The command '/bin/sh -c poetry install' returned a non-zero code: 2

[uSpike]

It seems to me that a simple solution is to run [env python] -m pip instead of pip directly.

[sdispater]

Yes, at the moment Poetry just calls pip. This is an issue that has arisen before so we should definitely tackle this by implemented the solution proposed by @uSpike.

[nottrobin]

even subprocess.call(['env', 'python', '-m', 'pip', 'install']) is potentially problematic because the version of python that happens to be running Poetry is not necessarily env python. It might be env python3, or it might simply be something else altogether. E.g. I might have run ~/.virtualenvs/something/bin/python -m poetry install.

I think the most reliable method would be to use the pip module directly within Python, e.g.:

try:
    from pip import main as pipmain
except ImportError:
    from pip._internal import main as pipmain

pipmain("install", "mypackage")

I'll try to look into this in the next couple of days if no-one else has already.

[uSpike]

@nottrobin wouldn't that method conflict with https://github.com/sdispater/poetry/issues/621 ?

If I'm understanding correctly, the intent is to be able to use a different version of python for running poetry vs the virtual environment that is created for your project.

[nottrobin]

@uSpike ah I see what you mean. So there are 2 scenarios here that need to both be covered:

1. Poetry is managing the virtual environment for the project, and so poetry install should clearly use the version of Python declared inside that virtual environment to install dependencies.
2. Poetry has been asked not to manage the virtual environment for the project (settings.virtualenvs.create == false).

In scenario 2, I think the least surprising thing would be for the same version of Python used to run poetry itself to be used for everything. But of course this is explicitly not the case in scenario 1.

So I think it might make sense to use the method I describe for when settings.virtualenvs.create == false, and the method you describe otherwise. Does that make sense?

[uSpike]

@nottrobin I think your analysis makes sense. Would it be equivalent to:

import sys
if (settings.virtualenvs.create == false):
    cmd = [sys.executable, "-m", "pip"]
else:
    cmd = ["python", "-m", "pip]

where sys.executable is the python on which poetry is running?

[nottrobin]

@uSpike yes exactly, and I suppose the same should apply to anything running outside the virtual environment - for example, I haven't looked into how poetry creates the virtual environment, but if it uses venv, then that should also be [sys.executable, "-m", "venv"].

[uSpike]

Poetry seems to use the venv module and falls back to virtualenv - https://github.com/sdispater/poetry/blob/c99ae5f96c6227491d36477657a1c82157567c7c/poetry/utils/env.py#L275

[nottrobin]

Oh that's fine then.

Of course, it's perfectly posssible that settings.virtualenvs.create == false is too superficial and there's a much better way inside Poetry to test for whether it's running everything inside a virtual env right now.

[nottrobin]

@sdispater, @uSpike thanks :heart:. When should this make it into a release?

[t2wu]

I think I'm running into similar if not the same problem. How do I install this version of Poetry to test fix (#831)?