Pip-tools: pip-sync re-install VCS package everytime

[staring-frog]

Hi,

if e.g. requirements.txt has
-e hg+https://bitbucket.org/hardbyte/python-can@8f3908c6293c#egg=python-can
'pip-sync requirements.txt' fails to install it. But pip installs alright.

$ ./env/bin/pip-sync requirements.txt
You are using pip version 7.1.0, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting python-can
Could not find a version that satisfies the requirement python-can (from versions: )
No matching distribution found for python-can
$ ./env/bin/pip freeze
You are using pip version 7.1.0, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
click==5.1
first==2.0.1
pip-tools==1.1.4
six==1.9.0
wheel==0.24.0

[spookylukey]

I'm seeing this too. If it can find it on PyPI, then it will install from there instead of from the VCS repo, and then exit successfully. This is a serious bug, because you think everything is installed as specified, but in fact a different version gets installed.

[benzkji]

same here. If I cannot use some of my open source packages under development, I must go back to pip install -r. In fact, how can it fail? is it not doing exactly the same as pip install -r when installing a single package?

[benzkji]

so for now, I'm going with pip-compile, and then pip install -r.

[benje]

also came across this issue, thanks for the tip @benzkji

[AlexRiina]

There may be other cases where this doesn't work, but the following passed the tests and worked for me with an editable dependency:

--- a/piptools/sync.py
+++ b/piptools/sync.py
@@ -108,7 +108,7 @@ def diff(compiled_requirements, installed_dists):

     for key, requirement in requirements_lut.items():
         if key not in satisfied:
-            to_install.add(str(requirement.req))
+            to_install.add(str(requirement.link or requirement.req))

[nvie]

I've added support for this in 9501b18e37ccbce1cb69aa2cf3cd89953c3a2ece. Thanks, @AlexRiina

[spookylukey]

This is not really fixed.

If you have requirements from a VCS URL, and do pip-sync, then it will uninstall and reinstall them every time you do pip-sync. e.g. I see something like:

Uninstalling ometria-0.1.5:
  Successfully uninstalled ometria-0.1.5
Collecting ometria from git+https://github.com/spookylukey/ometria.git@a636fa529ac98d7420494f200b299df728135451#egg=ometria
  Cloning https://github.com/spookylukey/ometria.git (to a636fa529ac98d7420494f200b299df728135451) to /tmp/pip-build-QRVTtO/ometria
  Could not find a tag or branch 'a636fa529ac98d7420494f200b299df728135451', assuming commit.

This means a lot of unnecessary network traffic.

[benzkji]

@spookylukey I see the same behaviour when doing a pip install -r requirements.txt, so this might be an issue with pip itself?

[spookylukey]

@benzkji I don't see that behaviour with most recent pip (7.1.2). I see instead this:

Obtaining ometria from git+https://github.com/spookylukey/ometria.git@a636fa529ac98d7420494f200b299df728135451#egg=ometria (from -r requirements.txt (line 46))
  Updating /home/luke/.virtualenvs/wolfandbadger/src/ometria clone (to a636fa529ac98d7420494f200b299df728135451)
  Could not find a tag or branch 'a636fa529ac98d7420494f200b299df728135451', assuming commit.

In other words, pip just updates the repo and runs the setup.py again, while pip-tools uninstalls completely and then clones the repo etc.

[benzkji]

@spookylukey true. should have had a closer look.

[nirvana-msu]

I have exactly the same problem. It seems impossible to specify a VCS package pointing to a particular tag/commit without it being reinstalled each time by pip-sync.

I traced the problem down to the way it compares installed packages vs requirements. In the case of installed packages, the key is basically installed_dists = pip.get_installed_distributions(skip=[]); for dist in installed_dists: key = key_from_req(dist), which just returns a package name. Whereas for the requirements, the key is {r.link or key_from_req(r.req): r for r in compiled_requirements}, i.e. instead of using key_from_req it uses r.link which is the full VCS URL. Obviously full VCS URL would never match a package name either way you try to specify it.

Is there any workaround for this?

[Herst]

@nirvana-msu Following your explanation it should never work if I understand it correctly but in my case I have four VCS entries in my requirements file and it works for three of them.

[spookylukey]

This appears to be fixed for me, perhaps due to a setuptools/pip change or something? At any rate, after the first pip-sync run, subsequent pip-sync runs to completion successfully without any internet connection.

[vphilippon]

@spookylukey Could you reproduce and link you pip/setuptools versions for reference please? that would be great. Thanks!

[spookylukey]

@vphilippon

It looks like I'm using:

setuptools 39.0.1
pip 9.0.3
pip-tools 1.11.0

I have git+git and git+https requirements lines, and these work fine. I've found that Mercurial repos (hg+https scheme) do not have the same behaviour - they crash if I run pip-sync without internet connection.

[atugushev]

I believe it was fixed in pip-tools==1.10.0. I've checked on the latest pip-tools==4.1.0 and it works fine:

$ cat requirements.txt
-e git+https://github.com/spookylukey/ometria.git@a636fa529ac98d7420494f200b299df728135451#egg=ometria

$ pip-sync
Obtaining ometria from git+https://github.com/spookylukey/ometria.git@a636fa529ac98d7420494f200b299df728135451#egg=ometria (from -r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmp8_xh1dwh (line 1))
  Cloning https://github.com/spookylukey/ometria.git (to revision a636fa529ac98d7420494f200b299df728135451) to ./.venv/src/ometria
  Running command git clone -q https://github.com/spookylukey/ometria.git /private/tmp/tmp2/.venv/src/ometria
  Running command git checkout -q a636fa529ac98d7420494f200b299df728135451
Installing collected packages: ometria
  Running setup.py develop for ometria
Successfully installed ometria

$ pip-sync
Uninstalling ometria-0.1.5:
  Successfully uninstalled ometria-0.1.5
Obtaining ometria from git+https://github.com/spookylukey/ometria.git@a636fa529ac98d7420494f200b299df728135451#egg=ometria (from -r /var/folders/4x/rz_w89dx1z3bydfd6qpdjn_h0000gn/T/tmpih51s327 (line 1))
  Skipping because already up-to-date.
Installing collected packages: ometria
  Running setup.py develop for ometria
Successfully installed ometria

[atugushev]

I'll close this based on the above, but please let us know if it doesn't resolve your issue. Thanks!

[staring-frog]

Thanks!