pip 10 is unable to install from non-editable VCS URLs

[RazerM]

• Pip version: 10.0.0
• Python version: 3.5.4
• Operating system: Linux

Description:

I'm trying to install a package from Git.

What I've run:

$ pip3 install git+ssh://[email protected]/mypackage.git#egg=mypackage

Building wheels for collected packages: mypackage
Exception:
Traceback (most recent call last):
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/commands/install.py", line 305, in run
    session=session, autobuilding=True
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 773, in build
    python_tag=python_tag,
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 633, in _build_one
    python_tag=python_tag)
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 637, in _build_one_inside_env
    if self.__build_one(req, temp_dir.path, python_tag=python_tag):
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 663, in __build_one
    base_args = self._base_setup_args(req)
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 659, in _base_setup_args
    SETUPTOOLS_SHIM % req.setup_py
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 416, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for mypackage from git+ssh://[email protected]/mypackage.git#egg=mypackage in /home/p/local/lib/python3.5/site-packages

If I uninstall the package and then install, it works. But it continues to fail when trying install again.

[bruceduhamel]

I saw this same error on Python 2 and found that using pip 9 worked as expected. I wonder if this is related to the deprecation and removal of the egg switch? Removing that also seemed to solve the issue.

[mmulich]

I'm experiencing this issue on Travis-CI with Python 3.5.2 and pip==10.0.0. I do not experience it with Python 3.6 (local) or 2.7 (both local and on travis).

Traceback (most recent call last):
...
  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 415, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for cnx-query-grammar from git+https://github.com/Connexions/cnx-query-grammar.git#egg=cnx-query-grammar in /home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages (from -r requirements/./test.txt (line 9))

[makmanalp]

Exact same issue, on Ubuntu 14.04.2 LTS, pip 10.0.0 installed with get-pip.py on python 3.4.0 specifically on my git installs.

[makmanalp]

Also using the #egg= parameter. I found this thread: https://github.com/pypa/pip/issues/1749

[makmanalp]

Oddly removing the whole virtualenv and reinstalling (including the get-pip bit) worked.

[makmanalp]

... but it failed again on the second deploy on top of an existing virtualenv (which should have been a no-op since all the packages are installed already but I guess it redownloads the git packages each time).

[mmulich]

Initial installation works, but attempts to reinstall are the problem space. So far I'm seeing a difference in the InstallRequirement's build_env value, where the initial install produced a NoOpBuildEnvironment object and a reinstall has a BuildEnvironment.

[JordanSlaman]

Also experiencing this issue with pip 10.0.0
This breaks our ops/build workflows. 😢

[mmulich]

In case others are also debugging this. I've narrowed down the two paths, first install and reinstall, to the difference in the results coming out of the resolver's _get_abstract_dist_for. Prior to that call both paths have <InstallRequirement>.source_dir == None. After this call, only the first install path has a source_dir set to something other than None. I'm guessing this is because it needs to go off and fetch the source during the first-install, but not on the reinstall.

... I could be off in the weeds though. It is my first time looking at this code. 😕

[makmanalp]

My workaround: pip install pip==9.0.3

[mmulich]

O... so the requirements file I'm working on isn't using -e prefixes for vcs installs. Adding -e to the beginning of the line does make this work. And in @RazerM 's case: pip3 install -e git+ssh://[email protected]/mypackage.git#egg=mypackage

So part of this issue is user error??? The other part of it is that we should get back something to the effect of Requirement already satisfied: ... and instead we get a Traceback.

[makmanalp]

If it helps, mine also doesn't use -e, though I thought VCS package URLs needn't be editable.

[RazerM]

Correct, they needn't be:

VCS projects can be installed in editable mode (using the --editable option) or not.

• For editable installs, the clone location by default is "\/src/SomeProject" in virtual environments, and "\/src/SomeProject" for global installs. The --src option can be used to modify this location.
  
  
  
  • For non-editable installs, the project is built locally in a temp dir and then normally. Note that if a satisfactory version of the package is already installed, the VCS source will not overwrite it without an --upgrade flag. VCS requirements pin the package version (specified in the setup.py file) of the target commit, not necessarily the commit itself.
  
  
  • The pip freeze subcommand will record the VCS requirement specifier (referencing a specific commit) if and only if the install is done using the editable option.
  
  

[techalchemy]

FYI we are running into this in pipenv (our testing infrastructure caught this), here's what I can say on the topic:

• This only impacts non-editable VCS installs (i.e. not passing -e
• We explicitly pass a PIP_SRC_DIR during testing and it seems to make no difference
• All of our tests run in isolated environments

As far as the distinction between editable and non-editable dependencies, from a resolution standpoint we can resolve dependency graphs for editable dependencies but can't for non-editable dependencies. Still-- we test to ensure that they can be installed, which now fails.

As a temporary fix we are moving our pip install calls to python -m <vendored pip9> install for compatibility.

[pradyunsg]

I am unable to reproduce this. Could someone provide steps to reproduce? If it's with in a docker container, that'd be awesome.

[eely22]

Just run this twice with pip 10 in a clean venv, it will fail on the second try:

pip install git+https://github.com/requests/requests.git#egg=requests

[di]

I can't reproduce:

/tmp $ python -m venv env
/tmp $ source env/bin/activate
(env) /tmp $ pip install -U pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 854kB/s
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-10.0.0
(env) /tmp $ pip install git+https://github.com/requests/requests.git#egg=requests
Collecting requests from git+https://github.com/requests/requests.git#egg=requests
  Cloning https://github.com/requests/requests.git to /private/var/folders/px/dlfyqxyx3mx0t_yx3dvbb9nc0000gn/T/pip-install-32abmlma/requests
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 6.9MB/s
Collecting idna<2.7,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 9.3MB/s
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 10.9MB/s
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 10.3MB/s
Installing collected packages: chardet, idna, urllib3, certifi, requests
  Running setup.py install for requests ... done
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
(env) /tmp $ pip install git+https://github.com/requests/requests.git#egg=requests
Requirement already satisfied: requests from git+https://github.com/requests/requests.git#egg=requests in ./env/lib/python3.6/site-packages (2.18.4)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./env/lib/python3.6/site-packages (from requests) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in ./env/lib/python3.6/site-packages (from requests) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in ./env/lib/python3.6/site-packages (from requests) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.6/site-packages (from requests) (2018.4.16)

[techalchemy]

@di @pradyunsg first spotted: https://travis-ci.org/pypa/pipenv/jobs/367322542#L715

log:

 ◰³ tempenv-3321664538cc  ~/g/pypa-pipenv   tmp $…  pip install --no-cache-dir pip
Requirement already satisfied: pip in /home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages (10.0.0)
 ◰³ tempenv-3321664538cc  ~/g/pypa-pipenv   tmp $…  pip install git+https://github.com/benjaminp/six.git#egg=six
Collecting six from git+https://github.com/benjaminp/six.git#egg=six
  Cloning https://github.com/benjaminp/six.git to /tmp/pip-install-2ff500pr/six
Building wheels for collected packages: six
  Running setup.py bdist_wheel for six ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-ebgi9ij6/wheels/a9/36/0a/8173089e25b3c1df51c0bb21d21ec8dc5d7ee9d0ec9b6b51e1
Successfully built six
Installing collected packages: six
Successfully installed six-1.11.0
 ◰³ tempenv-3321664538cc  ~/g/pypa-pipenv   tmp $…  pip install git+https://github.com/benjaminp/six.git#egg=six
Requirement already satisfied: six from git+https://github.com/benjaminp/six.git#egg=six in /home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages (1.11.0)
Building wheels for collected packages: six
Exception:
Traceback (most recent call last):
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 305, in run
    session=session, autobuilding=True
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 773, in build
    python_tag=python_tag,
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 633, in _build_one
    python_tag=python_tag)
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 637, in _build_one_inside_env
    if self.__build_one(req, temp_dir.path, python_tag=python_tag):
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 663, in __build_one
    base_args = self._base_setup_args(req)
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 659, in _base_setup_args
    SETUPTOOLS_SHIM % req.setup_py
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 415, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for six from git+https://github.com/benjaminp/six.git#egg=six in /home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages

[techalchemy]

And a Dockerfile which reproduces:

FROM python:3.6-alpine3.7

ENV PYTHONUNBUFFERED=1
ENV LC_ALL=en_US.UTF-8

RUN apk update \
    && apk add git

WORKDIR /app/


RUN set -ex \
    && pip install --upgrade pip \
    && pip install git+https://github.com/benjaminp/six.git#egg=six \
    && pip install git+https://github.com/benjaminp/six.git#egg=six

CMD ["/bin/bash"]

[eely22]

We can reproduce on our build machines, try this:

docker run -it circleci/python:2.7.13
cd ~
virtualenv env
. env/bin/activate
pip install -U pip
pip install git+https://github.com/requests/requests.git#egg=requests
pip install git+https://github.com/requests/requests.git#egg=requests

Here's my output:

ericely$ docker run -it circleci/python:2.7.13
$ cd ~
$ virtualenv env
New python executable in /home/circleci/env/bin/python
Installing setuptools, pip, wheel...
done.
$ $ . env/bin/activate
(env) $ pip install -U pip
Requirement already up-to-date: pip in ./env/lib/python2.7/site-packages (10.0.0)
(env) $ pip list
Package    Version
---------- -------
pip        10.0.0 
setuptools 39.0.1 
wheel      0.31.0 
(env) $ pip install git+https://github.com/requests/requests.git#egg=requests
Collecting requests from git+https://github.com/requests/requests.git#egg=requests
  Cloning https://github.com/requests/requests.git to /tmp/pip-install-I8n5LX/requests
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 3.9MB/s 
Collecting idna<2.7,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 7.9MB/s 
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 8.0MB/s 
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 4.3MB/s 
Building wheels for collected packages: requests
  Running setup.py bdist_wheel for requests ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-F7Eu9c/wheels/8d/84/fa/1ded919cc5b46b899830a8f8d18817f59abde627aa07362b23
Successfully built requests
Installing collected packages: chardet, idna, urllib3, certifi, requests
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
(env) $ 
(env) $ pip install git+https://github.com/requests/requests.git#egg=requests
Requirement already satisfied: requests from git+https://github.com/requests/requests.git#egg=requests in ./env/lib/python2.7/site-packages (2.18.4)
Requirement already satisfied: idna<2.7,>=2.5 in ./env/lib/python2.7/site-packages (from requests) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in ./env/lib/python2.7/site-packages (from requests) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python2.7/site-packages (from requests) (2018.4.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./env/lib/python2.7/site-packages (from requests) (3.0.4)
Building wheels for collected packages: requests
Exception:
Traceback (most recent call last):
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 305, in run
    session=session, autobuilding=True
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 773, in build
    python_tag=python_tag,
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 633, in _build_one
    python_tag=python_tag)
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 637, in _build_one_inside_env
    if self.__build_one(req, temp_dir.path, python_tag=python_tag):
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 663, in __build_one
    base_args = self._base_setup_args(req)
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 659, in _base_setup_args
    SETUPTOOLS_SHIM % req.setup_py
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 415, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for requests from git+https://github.com/requests/requests.git#egg=requests in ./env/lib/python2.7/site-packages
(env) $ 

[pradyunsg]

@pfmoore This is an important core use-case which is broken. I'm looking into a fix right now.

[bbakersmith]

Also experiencing this issue

[pfmoore]

@pradyunsg Cool, keep me posted - if there's anything I can help with let me know.

[pradyunsg]

If someone could take #5280 for a spin and check if that fixes this issue, that would be awesome! :)

[bbakersmith]

@pradyunsg I didn't do extensive testing, but #5280 resolves the issue for our case.

[techalchemy]

Thanks for the quick turnaround @pradyunsg — perhaps someone else can whip up a test case

Haven’t tested yet but can in the morning. What is the planned release timeline for pip 10.0.1? If we can avoid hacking around this in our point release I’d prefer to stick to our normal subprocess invocation of pip install but we have some bugfixes in the pipeline so I’m trying to assess whether to hold ours back for this— guidance appreciated!

/cc @pfmoore

[pradyunsg]

What is the planned release timeline for pip 10.0.1?

It'll happen this week, in all likelihood. :)

[pfmoore]

Correct - basically once all the issues/PRs marked as release blockers are resolved, I'll be doing a final pass of the issues and then releasing 10.0.1.

[RazerM]

5280 works for me too

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.