Pip-tools: Freaks out when compiling with a -e include that uses setuptools_scm

[danni]

Add a -e git module to requirements.in that uses setuptools_scm to set it's version number. Then try a pip-compile and it blows up with a bad exit code.

[nvie]

Not sure how to tackle this. If you have a suggestion, please submit a PR.

[blueyed]

The problem is that pip exports the source into a separate directory without the .git dir:

requirements.in:

-e git+https://github.com/pytest-dev/pytest-django@master#egg=pytest_django

Output from pip-compile with printing the cmd in pip using the following patch:

diff --git i/pip/vcs/__init__.py w/pip/vcs/__init__.py
index e0da09d..45876e1 100644
--- i/pip/vcs/__init__.py
+++ w/pip/vcs/__init__.py
@@ -315,6 +315,7 @@ def run_command(self, cmd, show_stdout=True, cwd=None,
         command name, and checks that the VCS is available
         """
         cmd = [self.name] + cmd
+        print(cmd)
         try:
             return call_subprocess(cmd, show_stdout, cwd,
                                    on_returncode, command_level,

The call from pip-tools to pip is dependencies = reqset._prepare_file(self.finder, ireq).

['git', 'clone', '-q', 'https://github.com/pytest-dev/pytest-django', '/tmp/pip-cxmdcbog-export']
pip-clone: https://github.com/pytest-dev/pytest-django
 - Add classifier for Python 3.5 in setup.py (#366) (3 weeks ago, 175de01, user Hahler)
['git', 'show-ref']
['git', 'rev-parse', 'HEAD']
['git', 'checkout-index', '-a', '-f', '--prefix', '/tmp/tmprsvzkgtzsource/pytest-django/']
Traceback (most recent call last):
  File "…/pyenv/project/bin/pip-compile", line 9, in <module>
    load_entry_point('pip-tools', 'console_scripts', 'pip-compile')()
  File "…/pyenv/project/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "…/pyenv/project/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "…/pyenv/project/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "…/pyenv/project/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "…/Vcs/pip-tools/piptools/scripts/compile.py", line 163, in cli
    results = resolver.resolve()
  File "…/Vcs/pip-tools/piptools/resolver.py", line 103, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "…/Vcs/pip-tools/piptools/resolver.py", line 185, in _resolve_one_round
    for best_match in best_matches
  File "…/Vcs/pip-tools/piptools/resolver.py", line 186, in <genexpr>
    for dep in self._iter_dependencies(best_match))
  File "…/Vcs/pip-tools/piptools/resolver.py", line 243, in _iter_dependencies
    for dependency in self.repository.get_dependencies(ireq):
  File "…/Vcs/pip-tools/piptools/repositories/pypi.py", line 132, in get_dependencies
    dependencies = reqset._prepare_file(self.finder, ireq)
  File "…/Vcs/pip/pip/req/req_set.py", line 491, in _prepare_file
    abstract_dist.prep_for_dist()
  File "…/Vcs/pip/pip/req/req_set.py", line 127, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "…/Vcs/pip/pip/req/req_install.py", line 428, in run_egg_info
    command_desc='python setup.py egg_info')
  File "…/Vcs/pip/pip/utils/__init__.py", line 718, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/tmprsvzkgtzsource/pytest-django/

[dfee]

I have an idea – what if we don't use git checkout-index? The problem is that checkout-index effectively gets rid of the scm data, which setuptools_scm needs.

Why are we doing the git checkout-index step?

[dfee]

Perhaps this is a bug in pip itself.

Just to aid in debugging here's a rich stack trace of how we get to the problem (starting at the last piptools call):
5 - piptools.repositories.pypi:PyPIRepository.get_dependencies -> reqset._prepare_file
4 - pip.req.req_set:RequirementSet._prepare_file -> req_to_install.update_editable
3 - pip.req.req_install:InstallRequirement.update_editable -> vcs_backend.export
2 - pip.vcs.git:Git.export -> self.run_command
1 - pip.vcs:VersionControl.run_command -> call_subprocess
0 - pip.utils:call_subprocess -> subprocess.Popen

If we go ahead and stop the execution before the last frame gets called and copy the .git directory into the tmp folder, execution completes normally. Of course, without the SCM folder, setuptools_scm fails.

and here's the full traceback (for me):

  /private/tmp/abra/env/bin/pip-compile(11)<module>()
      9 if __name__ == '__main__':
     10     sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
---> 11     sys.exit(cli())

  /private/tmp/abra/env/lib/python3.6/site-packages/click/core.py(722)__call__()
    721         """Alias for :meth:`main`."""
--> 722         return self.main(*args, **kwargs)
    723

  /private/tmp/abra/env/lib/python3.6/site-packages/click/core.py(697)main()
    696                 with self.make_context(prog_name, args, **extra) as ctx:
--> 697                     rv = self.invoke(ctx)
    698                     if not standalone_mode:

  /private/tmp/abra/env/lib/python3.6/site-packages/click/core.py(895)invoke()
    894         if self.callback is not None:
--> 895             return ctx.invoke(self.callback, **ctx.params)
    896

  /private/tmp/abra/env/lib/python3.6/site-packages/click/core.py(535)invoke()
    534             with ctx:
--> 535                 return callback(*args, **kwargs)
    536

  /private/tmp/abra/env/lib/python3.6/site-packages/piptools/scripts/compile.py(184)cli()
    183                             clear_caches=rebuild, allow_unsafe=allow_unsafe)
--> 184         results = resolver.resolve(max_rounds=max_rounds)
    185         if generate_hashes:

  /private/tmp/abra/env/lib/python3.6/site-packages/piptools/resolver.py(108)resolve()
    107             import ipdb; ipdb.set_trace()
--> 108             has_changed, best_matches = self._resolve_one_round()
    109             log.debug('-' * 60)

  /private/tmp/abra/env/lib/python3.6/site-packages/piptools/resolver.py(196)_resolve_one_round()
    195         for best_match in best_matches:
--> 196             for dep in self._iter_dependencies(best_match):
    197                 import ipdb; ipdb.set_trace()

  /private/tmp/abra/env/lib/python3.6/site-packages/piptools/resolver.py(265)_iter_dependencies()
    264         if ireq.editable:
--> 265             for dependency in self.repository.get_dependencies(ireq):
    266                 yield dependency

  /private/tmp/abra/env/lib/python3.6/site-packages/piptools/repositories/local.py(62)get_dependencies()
     61     def get_dependencies(self, ireq):
---> 62         return self.repository.get_dependencies(ireq)
     63

  /private/tmp/abra/env/lib/python3.6/site-packages/piptools/repositories/pypi.py(146)get_dependencies()
    145             import ipdb; ipdb.set_trace()
--> 146             self._dependencies_cache[ireq] = reqset._prepare_file(self.finder, ireq)
    147         return set(self._dependencies_cache[ireq])

  /private/tmp/abra/env/lib/python3.6/site-packages/pip/req/req_set.py(517)_prepare_file()
    516                 req_to_install.ensure_has_source_dir(self.src_dir)
--> 517                 req_to_install.update_editable(not self.is_download)
    518                 abstract_dist = make_abstract_dist(req_to_install)

  /private/tmp/abra/env/lib/python3.6/site-packages/pip/req/req_install.py(589)update_editable()
    588             else:
--> 589                 vcs_backend.export(self.source_dir)
    590         else:

  /private/tmp/abra/env/lib/python3.6/site-packages/pip/vcs/git.py(75)export()
     74                 ['checkout-index', '-a', '-f', '--prefix', location],
---> 75                 show_stdout=False, cwd=temp_dir)
     76         finally:

  /private/tmp/abra/env/lib/python3.6/site-packages/pip/vcs/__init__.py(325)run_command()
    324                                    command_desc, extra_environ,
--> 325                                    spinner)
    326         except OSError as e:

> /private/tmp/abra/env/lib/python3.6/site-packages/pip/utils/__init__.py(666)call_subprocess()
    664         try:
--> 665         proc = subprocess.Popen(
    667             cmd, stderr=subprocess.STDOUT, stdin=None, stdout=stdout,

[blueyed]

@dfee
Have you tried https://github.com/jazzband/pip-tools/pull/385?
Needs to be rebased, but might be easy.

[dfee]

Thanks @blueyed. I saw the tag pr-wanted and figured no one had proposed any changes. This could've saved myself a few hours!

[blueyed]

@dfee
Sorry about that.
By now the project moved to jazzband so I could fix the label at least.
As for the PR, you might want to revisit/update it maybe?
(I've put too much time/effort into pip-tools already, which has mostly not been accepted or ignored when it wasn't a jazzband project, so I came passive in that regard)

[dfee]

@blueyed it seems that PR is scheduled for the next patch release, no?

[dfee]

Ugh. It looks like https://github.com/jazzband/pip-tools/milestones/1.9.1 was deleted. Any way we can get this in https://github.com/jazzband/pip-tools/milestone/3 or https://github.com/jazzband/pip-tools/milestone/2?

[blueyed]

@dfee
The milestone seems to be 1.10.1 now: https://github.com/jazzband/pip-tools/milestone/2