Pipx: `pipx install` doesn't seem to work with `--python python3.4`

[daytonb]

Describe the bug

The pipx documentation states that the --python flag can be used to specify Python versions 3.3 and up for running the code installed in a given pipx virtual environment. However there seems to be some sort of json decoder issue with this. My first glance at the output makes it look like it's an error with Python type annotation.

How to reproduce

The environment I encountered this on is a CentOS 7 Linux computer. CentOS 7 repo's provided Python 3.4 and I have manually installed Python 3.5, 3.6, and 3.7.

$ python3.7 -m pip install --user pipx
$ python3.7 -m pipx run  --python python3.4 cowsay 'Hello, world!'
  _____________
< Hello, world! >
  =============
                  \
                   \
                     ^__^
                     (oo)\_______
                     (__)\       )\/\
                         ||----w |
                         ||     ||


$  python3.7 -m pipx install --python python3.4 cowsay
  File "<string>", line 89
    app_paths_of_dependencies: Dict[str, List[str]] = {}
                             ^
SyntaxError: invalid syntax

⢿ installing package 'cowsay'Traceback (most recent call last):
  File "/home/bkboulton/.pyenv/versions/py37/bin/pipx", line 10, in <module>
    sys.exit(cli())
  File "/home/bkboulton/.pyenv/versions/py37/lib/python3.7/site-packages/pipx/main.py", line 547, in c
    exit(run_pipx_command(parsed_pipx_args))
  File "/home/bkboulton/.pyenv/versions/py37/lib/python3.7/site-packages/pipx/main.py", line 166, in r
    include_dependencies=args.include_deps,
  File "/home/bkboulton/.pyenv/versions/py37/lib/python3.7/site-packages/pipx/commands.py", line 318,
    if venv.get_venv_metadata_for_package(package).package_version is None:
  File "/home/bkboulton/.pyenv/versions/py37/lib/python3.7/site-packages/pipx/Venv.py", line 161, in g
    self.python_path, VENV_METADATA_INSPECTOR, package, str(self.bin_path)
  File "/home/bkboulton/.pyenv/versions/3.7.4/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/bkboulton/.pyenv/versions/3.7.4/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/bkboulton/.pyenv/versions/3.7.4/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I got the same error specifying --python python3.5.

It sort of looks like maybe we're trying to evaluate some Python 3.7 code with the Python (3.4 or 3.5) interpreter that we put into the virtual environment.

Expected behavior

If I instead just use the default Python interpreter by not specifying the --python flag at all or if I specify Python 3.6 or 3.7, it works as expected:

$ python3.7 -m pipx install --python python3.6 cowsay
  installed package cowsay 2.0.3, Python 3.6.8
  These apps are now globally available
    - cowsay
done! ✨ 🌟 ✨

[itsayellow]

This sounds like venv_metadata_inspector.py, because it's running with the venv python, needs to be compatible with any venv python. Thus it might either have to abandon type annotations or abandon support for lower python versions.

[itsayellow]

Or I suppose it could just use type hints in comments instead of the python 3.5+ type hints.

[cs01]

Or I suppose it could just use type hints in comments instead of the python 3.5+ type hints.

Ah yeah I think you're right. Type annotations can be moved into comments and mypy will still check them.

The unit tests should also be updated to test this.

[itsayellow]

I've been experimenting with fixing this, and have gotten the type annotations fixed, but now I'm trying to figure out what to do with pathlib.Path(), because for example there is no pathlib in python 3.3. The paths need to be cast to strings, and I'm figuring out how exactly to do this and still keep mypy happy.

[itsayellow]

work in progress: https://github.com/itsayellow/pipx/tree/venv_mypy_comments

[itsayellow]

So it appears that mypy won't work, even using type comments, without the typing library.

It seems that we either need to add the typing library to every venv (creating possible conflicts with the app) or abandon mypy type-checking for venv_metadata_inspector.py.
https://mypy.readthedocs.io/en/latest/python2.html

To run your program, you must have the typing module in your Python 2 module search path. Use pip install typing to install the module. This also works for Python 3 versions prior to 3.5 that don’t include typing in the standard library.

[itsayellow]

My opinion is to drop typing for venv_metadata_inspector.py for maximum compatibility. What do you think, @cs01 ?

[cs01]

Thanks for doing that investigating.

I usually lean toward optimizing developer experience and correctness of code. If I don't like working on the code or I have a hard time getting things right, it makes me not enjoy working on a project and harder for others to contribute. That is one of the reasons I made pipx 3.6+ -- f-strings and types.

I would probably lean toward making the types inline comments and only supporting 3.5+. I would probably not sacrifice confidence in code (removing types) to support what I assume are corner cases of installing < 3.5. Is there a good reason people are using < Python 3.5 (or even 3.6) to install packages with pipx?

[itsayellow]

I agree in general with your development strategy, but was thinking that venv_metadata_inspector.py might be a special case because it's not running on "pipx's python" but the venv python. And it's just one file.

On the broader point, I can't imagine any reason for me personally to use python < 3.5, but I was trying to be charitable to the general user public, like the OP of this issue, @daytonb, who has a system that's giving him python 3.4.

Perhaps @daytonb can chime in

[daytonb]

I am actually ok with dropping support for Python 3.4 especially since 3.8 has just been released. I’d be satisfied with a commit to the docs/help output that updates the claim from working with 3.3+ to 3.5+.

I wasn’t actually testing cowsay when I hit this. I had set up a Python package index server for a local network and was testing that the packages I had loaded on it worked with all the versions on Python found on that network. This is mostly made up of Centos 7 computers which provide Python 3.4 as the default python3. That’s why I was trying to use pipx all the way back to Python 3.4. Users on the network can still install the packages with pip and use virtualenvs. They just can’t use pipx with 3.4.

[daytonb]

By they way, I really appreciate that you looked into this issue and discussed ways to address it. Pipx is really helpful.

[itsayellow]

Thanks @daytonb for the reply!

Yep I'm cool with having us be a python 3.5+ solution in that case. Let me check, it could be that the only change is, as @daytonb says, a documentation/requirements change.

[cs01]

I am actually ok with dropping support for Python 3.4 especially since 3.8 has just been released. I’d be satisfied with a commit to the docs/help output that updates the claim from working with 3.3+ to 3.5+.

Sounds good.

Python 3.5 will still fail on that file because there an inline type definition.

# This is how you declare the type of a variable type in Python 3.6
age: int = 1

# In Python 3.5 and earlier you can use a type comment instead
# (equivalent to the previous definition)
age = 1  # type: int

https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html#variables