Pip: Installing from git repo in local directory fails if specified in requirements.txt and path contains spaces

[lululukas]

Environment

• pip version: 19.3.1
• Python version: 3.7.4
• OS: Windows 10

Description
I am able to install a package from a git repo in a local directory as such:

pip install git+file:///"k/my/path with/lots of/spaces/repo/.git/"

However when I put git+file:///"k/my/path with/lots of/spaces/repo/.git/" into my requirements.txt and then try to install with pip install -r requirements.txt it fails with ValueError: No closing quotation.

Expected behavior
Expected the package to be installed from local directory just as it is when I run the command directly.

How to Reproduce

1. Specify local git repo at a path containing spaces as a dependency in requirements.txt
2. Then run pip install -r requirements.txt
3. An error occurs.

Output

pip install -r .\requirements.txt
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\username\AppData\Local\Continuum\anaconda3\lib\site-packages\pip\_internal\cli\base_command.py", line 178, in main
    status = self.run(options, args)
  File "C:\Users\username\AppData\Local\Continuum\anaconda3\lib\site-packages\pip\_internal\commands\install.py", line 326, in run
    self.name, wheel_cache
  File "C:\Users\username\AppData\Local\Continuum\anaconda3\lib\site-packages\pip\_internal\cli\base_command.py", line 288, in populate_requirement_set
    use_pep517=options.use_pep517):
  File "C:\Users\username\AppData\Local\Continuum\anaconda3\lib\site-packages\pip\_internal\req\req_file.py", line 112, in parse_requirements
    for req in req_iter:
  File "C:\Users\username\AppData\Local\Continuum\anaconda3\lib\site-packages\pip\_internal\req\req_file.py", line 173, in process_line
    shlex.split(options_str), defaults)  # type: ignore
  File "C:\Users\username\AppData\Local\Continuum\anaconda3\lib\shlex.py", line 305, in split
    return list(lex)
  File "C:\Users\username\AppData\Local\Continuum\anaconda3\lib\shlex.py", line 295, in __next__
    token = self.get_token()
  File "C:\Users\username\AppData\Local\Continuum\anaconda3\lib\shlex.py", line 105, in get_token
    raw = self.read_token()
  File "C:\Users\username\AppData\Local\Continuum\anaconda3\lib\shlex.py", line 187, in read_token
    raise ValueError("No closing quotation")
ValueError: No closing quotation

requirements.txt:

git+file:///"k/my/path with/lots of/spaces/repo/.git/"
pandas
requests
retrying
xlsxwriter

See also related SO question.

[chrahunt]

The handling of quotes when pip is executed directly is governed by your shell. In Bash, for example, non-nested pairs of quotes are stripped and their contents concatenated with what comes before/after to create each argument.

Requirements files, on the other hand, are read as-is by optparse and no special quote handling or stripping takes place.

That said, in this case when trying to provide a URL, the non-valid URL characters (like space) must be percent-encoded (space is %20). See this script for an example that successfully installs a git repo that had spaces in the path:

repro.sh

#!/bin/sh
cd "$(mktemp -d)"

dir="$PWD/a b c/d e f"
mkdir -p "$dir"
cd "$dir"
cat <<EOF > setup.py
from setuptools import setup

setup(name='example')
EOF

git init
git add .
git commit -m init
cd -


python -V
python -m venv venv
venv/bin/python -m pip install --upgrade pip

cat <<EOF | tee requirements.txt
git+file://$PWD/a%20b%20c/d%20e%20f
EOF

python -m pip install -r requirements.txt

Output

Initialized empty Git repository in /tmp/user/1000/tmp.kITsbsZCPO/a b c/d e f/.git/
[master (root-commit) d347157] init
 1 file changed, 3 insertions(+)
 create mode 100644 setup.py
/tmp/user/1000/tmp.kITsbsZCPO
Python 3.7.2
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 18.1
    Uninstalling pip-18.1:
      Successfully uninstalled pip-18.1
Successfully installed pip-19.3.1
git+file:///tmp/user/1000/tmp.kITsbsZCPO/a%20b%20c/d%20e%20f
Collecting git+file:///tmp/user/1000/tmp.kITsbsZCPO/a%20b%20c/d%20e%20f (from -r requirements.txt (line 1))
  Cloning file:///tmp/user/1000/tmp.kITsbsZCPO/a b c/d e f to /tmp/user/1000/pip-req-build-ts2s099_
Requirement already satisfied (use --upgrade to upgrade): example==0.0.0 from git+file:///tmp/user/1000/tmp.kITsbsZCPO/a%20b%20c/d%20e%20f in /home/chris/.pyenv/versions/3.7.2/lib/python3.7/site-packages (from -r requirements.txt (line 1))

[lululukas]

This worked thank you so much!!

Do you want to answer the SO question? Otherwise I'm going to quote your explanation as the answer, if you don't mind.

I'm going to close this issue, out of curiosity, is this a Windows only issue? Because according to user 00 on SO, it works for him without %20 on Mac.

[chrahunt]

I'm glad it helped. :)

Please go ahead!

I would be a little surprised if the behavior is platform-specific. The code here should be what splits out the URL and I don't see anything that would change across platforms. In any case, percent-encoding should work everywhere out of the box so it's definitely the way to go even for people who may get it working some other way on other platforms.