Pip: Cannot pip install a URL that redirects to a valid wheel

[ezyang]

Environment

• pip version: 19.1.1
• Python version: 3.7
• OS: Linux

Description
I attempted to pip install a URL which 302 redirects to a valid whl.

curl -v http://ezyang.com/nightly.php
*   Trying 2401:db00:20ff:ff70:face:b00c:0:1e10...
* TCP_NODELAY set
* Connected to fwdproxy (2401:db00:20ff:ff70:face:b00c:0:1e10) port 8080 (#0)
> GET http://ezyang.com/nightly.php HTTP/1.1
> Host: ezyang.com
> User-Agent: curl/7.59.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 302 Moved Temporarily
< Date: Wed, 31 Jul 2019 21:25:09 GMT
< Server: Apache
< X-Powered-By: PHP/5.5.26
< Location: https://download.pytorch.org/whl/nightly/torchaudio_nightly-0.4.0.dev20190731-cp37-cp37m-linux_x86_64.whl
< Content-Type: text/html
< Via: HTTP/1.1 18.4.86.46:80 (fwdproxy2/350 69.171.251.28)
< X-Connected-To: 18.4.86.46
< X-FB-IP-Type: allowed
< Connection: keep-alive
< Content-Length: 0
< 
* Connection #0 to host fwdproxy left intact

pip installing the redirected URL gets to dependency solving, as desired:

$ pip install https://download.pytorch.org/whl/nightly/torchaudio_nightly-0.4.0.dev20190731-cp37-cp37m-linux_x86_64.whl
Collecting torchaudio-nightly==0.4.0.dev20190731 from https://download.pytorch.org/whl/nightly/torchaudio_nightly-0.4.0.dev20190731-cp37-cp37m-linux_x86_64.whl
  Using cached https://download.pytorch.org/whl/nightly/torchaudio_nightly-0.4.0.dev20190731-cp37-cp37m-linux_x86_64.whl
Collecting torch-nightly==1.2.0.dev20190731 (from torchaudio-nightly==0.4.0.dev20190731)
  ERROR: Could not find a version that satisfies the requirement torch-nightly==1.2.0.dev20190731 (from torchaudio-nightly==0.4.0.dev20190731) (from versions: none)
ERROR: No matching distribution found for torch-nightly==1.2.0.dev20190731 (from torchaudio-nightly==0.4.0.dev20190731)

However, pip installing the redirecting URL fails earlier in a mysterious way.

$ pip install http://ezyang.com/nightly.php
Collecting http://ezyang.com/nightly.php
  Using cached http://ezyang.com/nightly.php
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/ezyang/miniconda3/envs/scratch1/lib/python3.7/tokenize.py", line 447, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-d0odg3he/setup.py'
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-d0odg3he/

Expected behavior
I expect the second command to work.

[uranusjr]

pip relies on the URL to determine whether a package is wheel or not. Since the URL you passed does not end with .whl, it treats it as a source package. The redirect does not help since the check happens before the package is downloaded.

It is definitely technically possible to change this behaviour, but it would be quite involved, and I am not confident anyone would be willing to take on the task (it contributes to why I gave up on #6329 eventually, fwiw). At this point I’d choose to be practical and treat this as a feature™ (maybe document it explicitly) instead.

[ezyang]

If I make the URL end in whl, is that sufficient? Or does the rest of the filename have to be correct too.

[pradyunsg]

As things stand, I think the filename needs to be correct.

[ezyang]

OK, sounds like documenting this is in order.

[pradyunsg]

Since you clearly hit this issue, knowing how you looked for the relevant information can help inform if, where and how we document. 🙂

[ezyang]

TBH, I'm not really sure where in the pip manual I would have expected this information to live. A better error message instead of FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-d0odg3he/setup.py' would have been much better.

[uranusjr]

I wouldn’t be surprised if the implementation allows you to only provide the .whl suffix, but you probably shouldn’t rely on it in that case either. The code really is written in a way to expect the user to provide the correct file name in the URL.

[ei8fdb]

TBH, I'm not really sure where in the pip manual I would have expected this information to live. A better error message instead of FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-req-build-d0odg3he/setup.py' would have been much better.

@ezyang can I ask since you now know why this failed, if you could rewrite the error message - what information would have been helpful to you in deciphering what went wrong?

I'm working on pip usability and part of this is looking at error messages.

[ezyang]

In this case, an error saying, "Hey, the URL you passed me doesn't look like a valid whl URL" would have been good.

[chrahunt]

What gets passed may be a wheel or sdist, so we might need to be a little more generic like "the file downloaded from isn't a valid package (not a wheel or valid sdist)".

[deveshks]

Hi @chrahunt , @uranusjr , @pradyunsg

I traced through the code and found that we are parsing the arguments passed to pip install in pip._internal.req.constructors. parse_req_from_line . Is there something we can check for here to catch such a case of the argument not being valid?

Or do we want to get till pip._internal.operations.build.metadata_legacy and check if the source directory contains setup.py or not. Also is there a way in between where we can perform such a check?

[deveshks]

Pinging @chrahunt , @uranusjr and @pradyunsg to get their thoughts on how to tackle this :)