Pip: Minor: Undefined behavior leads to confusing error with --index-url arg.

[SalomonSmeke]

Environment

• pip version: 19.2.3
• Python version: 3.8
• OS: Arch Linux

I do not believe this is limited to arch, but it was discovered as such here: https://github.com/SalomonSmeke/rere/issues/5

_note: as pointed out by @uranusjr below this is partially fixed in 19.3_

Description

A very minor problem, borderline not a bug but simply less than ideal behavior. When using a --index-url with pip install, passing something other than a url causes a confusing error.

This can manifest like in the issue above:
pip install --index-url --user https://test.pypi.org/simple/ reretrieve
when trying to:
pip install --user --index-url https://test.pypi.org/simple/ reretrieve

Expected behavior

An error about how the index url is invalid, not found, or otherwise messed up.

How to Reproduce

1. pip install --index-url --user https://test.pypi.org/simple/ reretrieve

Output

$ pip install --index-url --user https://test.pypi.org/simple/ reretrieve
Looking in indexes: --user
Collecting https://test.pypi.org/simple/
  Using cached https://test.pypi.org/simple/
  ERROR: Cannot unpack file /tmp/pip-unpack-w9t0tixn/simple (downloaded from /tmp/pip-req-build-s5ub6ixa, content-type: text/html; charset=UTF-8); cannot detect archive format
ERROR: Cannot determine archive format of /tmp/pip-req-build-s5ub6ixa.

_note: as pointed out by @uranusjr below in 19.3 the output will look a little different._

Edit

I just wanted to clarify that this happens with any bad text input/invalid url, not just --user and such.

$ pip install --index-url foo  https://test.pypi.org/simple/ reretrieve
Looking in indexes: foo
Collecting https://test.pypi.org/simple/
  Using cached https://test.pypi.org/simple/
  ERROR: Cannot unpack file /tmp/pip-unpack-160ive4r/simple (downloaded from /tmp/pip-req-build-8dw7kt5w, content-type: text/html; charset=UTF-8); cannot detect archive format
ERROR: Cannot determine archive format of /tmp/pip-req-build-8dw7kt5w

I dont blame you if you decide to wontfix this one, but wanted to make sure it was an intentional decision. thank you in advance for taking a look!

[uranusjr]

pip 19.3 has a warning like this:

WARNING: Url 'foo/pip/' is ignored. It is either a non-existing path or lacks a specific scheme.

that would probably help debug the problem. I kind of feel it can do a better job though; maybe detect the malformed index URL before it gets concatenated to the package name.

The cannot determine archive format error is much more complicated, unfortunately. pip allows specifying a package by URL, e.g.

pip install https://github.com/pypa/pip/archive/master.zip

and pip has no way to know whether you actually want to install https://test.pypi.org/simple/ as a package. So that part should probably be wontfix.

[SalomonSmeke]

@uranusjr of course the next version from mine got the warning! That would have definitely saved me some time in debugging, but I agree that ideally the message is clearer.

cannot determine archive format is unimportant here. if something like:

$ pip install --index-url --user https://test.pypi.org/simple/ reretrieve
ERROR: index-url --user is invalid.
Collecting https://test.pypi.org/simple/
  Using cached https://test.pypi.org/simple/
  ERROR: Cannot unpack file /tmp/pip-unpack-w9t0tixn/simple (downloaded from /tmp/pip-req-build-s5ub6ixa, content-type: text/html; charset=UTF-8); cannot detect archive format
ERROR: Cannot determine archive format of /tmp/pip-req-build-s5ub6ixa.

happens, the true source of the problem would be immediately clear to me, or at worst just a lot less difficult to track down.

In fact, I would argue cannot determine archive format is accurate and should remain! Assuming pip would not rather just exit after ERROR: index-url --user is invalid..

[chrahunt]

I'm +1 on mentioning --index-url in the warning message, assuming it doesn't make a mess in the code.

People can have --index-url and --extra-index-url in requirements.txt which may point to directories that don't always exist, so I would be wary of turning this into a hard failure.

[deveshks]

Hi @chrahunt , @uranusjr

Can I work on this PR? Also I was thinking about what would be the best place to check for such an issue.

I was thinking that we can insert an error in the below function, when we are checking the index urls. Or maybe there is a better place to put this error perhaps?

https://github.com/pypa/pip/blob/6086f71cde81454fda588c20868a241561809836/src/pip/_internal/models/search_scope.py#L77-L84

[uranusjr]

I’m not entirely sure, to be honest. I think part of the reason this is marked as awaiting PR is because it’s difficult to gauge the situation without diving into the implementation, and once you do that you’re in the best position to produce a PR. So I’d say go ahead writing some code first and we can see what we can work out.

[deveshks]

Thanks @uranusjr , I have created a PR considering one possible approach. Please take a look :)