Pip: Detect GIT http URL

[techtonik]

• Pip version: 9.0.1
• Python version: 2.7
• Operating system: Windows

Description:

Git http URLs can be detected with an additional GET request by just appending /info/refs?service=git-receive-pack to the URL:

> http https://github.com/nodebox/nodebox-opengl.git/info/refs?service=git-receive-pack
HTTP/1.1 200 OK
Cache-Control: no-cache, max-age=0, must-revalidate
Content-Type: application/x-git-upload-pack-advertisement
Expires: Fri, 01 Jan 1980 00:00:00 GMT
Pragma: no-cache
Server: GitHub Babel 2.0
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Frame-Options: DENY
X-GitHub-Request-Id: 0E25:6858:D796B8:160CDD3:595F47E1

This is known as smart protocol.

What I've run:

Currently pip fails to process such URLS:

> pip install -vvv https://github.com/nodebox/nodebox-opengl
...
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\python27\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "c:\python27\lib\site-packages\pip\wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "c:\python27\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "c:\python27\lib\site-packages\pip\req\req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "c:\python27\lib\site-packages\pip\download.py", line 821, in unpack_url
    hashes=hashes
  File "c:\python27\lib\site-packages\pip\download.py", line 663, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "c:\python27\lib\site-packages\pip\utils\__init__.py", line 620, in unpack_file
    'Cannot determine archive format of %s' % location
InstallationError: Cannot determine archive format of c:\users\user\appdata\local\temp\pip-c7ik5u-build

[techtonik]

pip 9.0.1 is unable to test server for supported backend - it detects vcs by comparing URL against supported vcs schemes (is_vcs_url). Every pip.vcs.VersionControl object exposes schemes list, which explicitly lists all prefixes for the given URL:

    schemes = (
        'git', 'git+http', 'git+https', 'git+ssh', 'git+git', 'git+file',
    )

To add autodetection:

• [ ] add InstallationError handler in download.py:unpack_url
  
  
  
  • [ ] replace InstallationError with InvalidFormat exception
  
  
• [ ] add detect_vcs function to download.py
• [ ] add detect() False stub method to vcs.VersionControl
  
  
  
  • [ ] document in vcs static and dynamic detection
  
  
• [ ] add detect() method to vcs.Git
• [ ] run vcs.unpack(location)

[pradyunsg]

@techtonik I'm not very familiar with pip's VCS interactions and likes. I'll label this as needs discussion for now, because I am assuming this is something that someone else would want to comment upon... :)

[pfmoore]

I'm not sure there's a problem here to be solved? You can simply use a git+http protocol, and I don't know of anyone who's reported that this is a significant problem for them. @techtonik do you have any evidence that this is a significant enough problem to justify the cost of implementing auto-detection?

[pradyunsg]

@techtonik Just curious - do you mean that pip install https://github.com/nodebox/nodebox-opengl should work when just adding 4 extra characters already works - pip install git+https://github.com/nodebox/nodebox-opengl?

[techtonik]

@pfmoore this is a simple DX/UX use case. Just ask anyone around who use pip occasionally how they would like to install packages from GitHub with pip. Target people who are using some other package manager primarily, like npm or whatever.

@pradyunsg these 4 characters work only for people who spend enough time reading pip related documentation and was using it often enough to memorize this pip-specific feature. My audience are people who need to install Python packages, because their tools are written in Python and who just need to try newer version. Not people who write code in Python daily and know about pip.

[pfmoore]

OK. I'm not convinced this is warranted. Most users don't use URLs they install from an index via pip install package. So those using URLs can be assumed to be somewhat experienced, and having to add "git+" isn't hard to explain. But if you want to create a PR and see how it fares with other pip devs, that's perfectly OK.

[pradyunsg]

if you want to create a PR and see how it fares with other pip devs, that's perfectly OK.

+1

[pradyunsg]

I've labelled this issue as an "deferred till PR".

This label is essentially for indicating that further discussion related to this issue should be deferred until someone comes around to make a PR. This does not mean that the said PR would be accepted - it has not been determined whether this is a useful change to pip and that decision has been deferred until the PR is made.

[xavfernandez]

Cf #539 & #1738 for previous discussion...

[dstufft]

It appears this issue has been opened three times by @techtonik and it's been closed twice, and all three times there have been different pip committers expressing that they think this is a bad idea.

We can't implement this sort of probing for every type of VCS there is or else things are going to become a lot slower, nevermind the fact that it is super hard to tell the difference between a transient failure during a probe and a real error that such a thing doesn't exist.

I'm going to close this again, please don't open another issue for the same thing.

[techtonik]

@dstufft it is not about every type of VCS - it is about the most popular VCS.