Pipx: Windows support?

[pfmoore]

It's not clear from the docs if pipx supports Windows, but given that it relies on symlinks, I assume not? (Symlinks typically require extra privileges on Windows, that the average user doesn't have). Are there plans to support Windows?

[cs01]

I'd like it to support Windows. The code already checks if it's running under windows and adds the ".exe" suffix in some places, but I haven't tested it. Based on on your comment it almost certainly does not work :). It shouldn't be very hard to modify the code to copy instead of symlink for the Windows platform.

I can modify the code to do this, but would have to rely on others testing it. If someone would like to modify the code and test it, that would be fine too. Are you interested in helping with this?

Is there anything else that needs to be done? Checks for some kind of .bat file extensions maybe?

[cs01]

btw thank you for the work you've done on pipsi

[pfmoore]

In theory, I'd be interested in seeing this happen, but to be honest, I'm currently using (and poking round in the internals of) about four different tools for bundling Python apps, as well as writing one of my own, off and on, so I think I'd be stretched too thin to commit to helping here right now!

But I will add this project to the list of ones I'm playing round with, and if I do come up with anything that might be helpful, I'll submit a PR or issue. Your approach looks nice.

Glad the changes I contributed to pipsi were of use to you :-)

[cs01]

I just added support for Windows but am not able to test it.

If you are a Windows user and have tried to use pipx, please post here and let me know if it worked or not.

[sahensley]

I should be able to test this evening on a Windows 10 VM if no one else has a chance to test before then.

[pfmoore]

I did a quick test and got an error about WindowsPath is not iterable I wasn't able to confirm it in a "normal" environment, as I don't want to install this in my system environment, and I'm not clear if it's expected to work when run from a virtualenv (I don't want to waste people's time reporting bugs that are due to my weird environment). I'll try to make some time to set up a VM to test in. (Hmm, installing Windows in virtualbox, meh).

Which does actually beg the question - what is the intended installation method for someone who doesn't want to install applications into their system Python installation?

[sahensley]

Editing yet again to not clutter thread and remove old information -

There are a couple parts to the issue:

• get-pipx.py needs to be updated in a similar fashion to main.py to account for windows venvs Scripts folder.
• subprocess.run in main.py is receiving a WindowsPath for self.python_path which is not iterable

[cs01]

Thanks @sahensley!

You're right, I forgot to update get-pipx.py! Darn it. I'll update it and let you know when I'm done so you can test again.

subprocess.run in main.py is receiving a WindowsPath for self.python_path which is not iterable

How annoying. Why would WindowsPath be different than Path?

Anyway, I guess what I should do is something like

cmd = [str(c) for c in cmd]

prior to running any commands.

[cs01]

@pfmoore

as I don't want to install this in my system environment, and I'm not clear if it's expected to work when run from a virtualenv

It works fine from a virtualenv if you do pip install -e .

what is the intended installation method for someone who doesn't want to install applications into their system Python installation?

Are you referring to installations into PIPX_HOME and PIPX_BIN_DIR? In that case the method would be to set the environment variables. My personal workflow is to just install/uninstall into their standard locations, but setting the env vars makes sense if you don't want to impact things you've already installed.

[pfmoore]

Are you referring to installations into PIPX_HOME and PIPX_BIN_DIR?

Ignore me. I was misunderstanding how get-pipx.py worked (I'd had trpuble with pipsi's bootstrapping method, and instinctively avoided get-pipx.py without thinking). Sorry for the noise.

How annoying. Why would WindowsPath be different than Path?

It's not. But subprocess.run on Windows has to convert argv into a command line (string), and that's where the error occurs. The details are here

[sahensley]

It looks like the WindowsPath not being iterable is a feature bug as well. Related issues on Python issue tracker 31961 and 34384. From the discussions, it will likely land in 3.8 then be backported.

The next "fun" thing I could see coming up with the string conversion is Windows paths triggering unicode escaping.

[pfmoore]

The next "fun" thing I could see coming up with the string conversion

Why do you say that? The referenced discussion is a simple error by someone not correctly quoting a string containing backslashes. There's plenty of fun you can have with encodings on Windows (because naive assumptions about what encoding to use don't always apply, you really do need to know what you're doing) but this doesn't seem like it's one of them :-)

[cs01]

@sahensley @pfmoore I updated master with a fix for get-pipx and when running as a subprocess. Let me know how it goes.

in get-pipx.py

• account for Scripts folder in get-pipx.py
• copy instead of symlink

in main.py and get-pipx.py

• convert Sequence[Path] to Sequence[str] before passing to subprocess.run

[cs01]

btw the best way to test get-pipx.py is to enter a virtualenv then run python get-pipx.py --src .

[sahensley]

Here are the results from running the newest get-pipx.py installer with Python 3.6.7.

(pipxvenv) C:\tmp\pipx>python get-pipx.py --src . --verbose
Installing pipx
virtualenv location is C:\Users\username\.local\pipx\venvs\pipx
running C:\tmp\pipxvenv\Scripts\python.exe -m venv C:\Users\username\.local\pipx\venvs\pipx
running C:\Users\username\.local\pipx\venvs\pipx\Scripts\pip.exe install --upgrade pip
ERROR: To modify pip, please run the following command:
c:\users\username\.local\pipx\venvs\pipx\scripts\python.exe -m pip install --upgrade pip
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "get-pipx.py", line 262, in <module>
    main()
  File "get-pipx.py", line 242, in main
    args.verbose,
  File "get-pipx.py", line 156, in install
    venv.create_venv()
  File "get-pipx.py", line 69, in create_venv
    self.upgrade_package("pip")
  File "get-pipx.py", line 81, in upgrade_package
    self._run_pip(["install", "--upgrade", package])
  File "get-pipx.py", line 88, in _run_pip
    _run(cmd)
  File "get-pipx.py", line 54, in _run
    raise PipxError(f"{cmd_str!r} failed")
__main__.PipxError: 'C:\\Users\\username\\.local\\pipx\\venvs\\pipx\\Scripts\\pip.exe install --upgrade pip' failed

When this happens, it creates a _C:\Users\username\.local\bin_ directory and a virtualenv at _C:\Users\username\.localpipx\venvspipx_. If the installer script is run again, it will exit with the message _You already have pipx installed. Type pipx to run._ until the .local directories are removed.

[pfmoore]

You should run pip as python -m pip, certainly in the case of upgrading pip itself, as the upgrade can't replace pip.exe if it's in use (which it is, by the OS, if you're running pip install -U pip).

The pip error message says this (although it's a bit buried under the rest of the output):

running C:\Users\username\.local\pipx\venvs\pipx\Scripts\pip.exe install --upgrade pip
ERROR: To modify pip, please run the following command:
c:\users\username\.local\pipx\venvs\pipx\scripts\python.exe -m pip install --upgrade pip

[cs01]

I updated to use python.exe -m pip instead of pip.exe.

Thanks for all the help with this, I think we're getting close.

[sahensley]

We are certainly close! It looks like the binary check and some WindowsPath issues are still creeping up.

Here's the issues I've noticed:

• Pipx installs via get-pipx.py but the binary check fails (file extension?)
  
  
  
  • Message: Expected to find C:\Users\username\.local\pipx\venvs\pipx\Scripts\pipx
  
  
• Running (via full path) pipx install <package> creates a venv, installs packages then fails after the "Successfully installed" line.

get-pipx.py output

(pipxvenv) C:\tmp\pipx>python get-pipx.py --src . --verbose
Installing pipx
virtualenv location is C:\Users\username\.local\pipx\venvs\pipx
running C:\tmp\pipxvenv\Scripts\python.exe -m venv C:\Users\username\.local\pipx\venvs\pipx
running C:\Users\username\.local\pipx\venvs\pipx\Scripts\python.exe -m pip install --upgrade pip
<pip upgraded>
running C:\Users\username\.local\pipx\venvs\pipx\Scripts\python.exe -m pip install .
Processing c:\tmp\pipx
<packages installed>
downloaded new binaries: C:\Users\username\.local\pipx\venvs\pipx\Scripts\pipx-script.py, C:\Users\username\.local\pipx\venvs\pipx\Scripts\pipx.exe, C:\Users\username\.local\pipx\venvs\pipx\Scripts\chardetect.exe
Expected to find C:\Users\username\.local\pipx\venvs\pipx\Scripts\pipx

pipx install black output

(pipxvenv) C:\tmp\pipx>C:\Users\username\.local\pipx\venvs\pipx\Scripts\pipx.exe install black --verbose
pipx (run_pipx_command:505): virtualenv location is C:\Users\username\.local\pipx\venvs\black
pipx (_run:198): running C:\Users\username\.local\pipx\venvs\pipx\Scripts\python.exe -m venv C:\Users\username\.local\pipx\venvs\black
pipx (_run:198): running C:\Users\username\.local\pipx\venvs\black\Scripts\python.exe -m pip install --upgrade pip
<pip installed>
pipx (_run:198): running C:\Users\username\.local\pipx\venvs\black\Scripts\python.exe -m pip install black
<packages downloaded and installed>
Successfully installed appdirs-1.4.3 attrs-18.2.0 black-18.9b0 click-7.0 toml-0.10.0
Traceback (most recent call last):
  File "C:\Users\username\.local\pipx\venvs\pipx\Scripts\pipx-script.py", line 11, in <module>
    load_entry_point('pipx==0.0.0.14', 'console_scripts', 'pipx')()
  File "C:\Users\username\.local\pipx\venvs\pipx\lib\site-packages\pipx\main.py", line 728, in cli
    run_pipx_command(args)
  File "C:\Users\username\.local\pipx\venvs\pipx\lib\site-packages\pipx\main.py", line 511, in run_pipx_command
    install(venv_dir, package, package_or_url, local_bin_dir, args.python, verbose)
  File "C:\Users\username\.local\pipx\venvs\pipx\lib\site-packages\pipx\main.py", line 413, in install
    if venv.get_package_version(package) is None:
  File "C:\Users\username\.local\pipx\venvs\pipx\lib\site-packages\pipx\main.py", line 116, in get_package_version
    stderr=subprocess.DEVNULL,
  File "C:\Users\username\AppData\Local\Programs\Python\Python36\Lib\subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\username\AppData\Local\Programs\Python\Python36\Lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "C:\Users\username\AppData\Local\Programs\Python\Python36\Lib\subprocess.py", line 971, in _execute_child
    args = list2cmdline(args)
  File "C:\Users\username\AppData\Local\Programs\Python\Python36\Lib\subprocess.py", line 461, in list2cmdline
    needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'WindowsPath' is not iterable

[cs01]

Once again thank you. Your summaries help me in making the fixes faster, much appreciated. Just pushed an update to master. lmk how it goes.

[sahensley]

Glad to help! Windows support is quite the undertaking (especially since you are writing it without a machine to test on) and I think it could really be the "killer feature" of the app.

get-pipx.py

The install is so close now. The get-pipx.py script now completes without any errors. The emoji appear as boxes in both cmd and PowerShell since neither shell appears to support utf-8 by default.

The PATH modification notes in the post-install output will need to be updated for Windows. I'd be glad to update the docs once we wrangle this beast.

_C:\Users\username\.local\bin_ has a pipx file without an extension which cannot be executed. Renaming .local\bin\pipx file to pipx.exe allows pipx.exe to be executed but then throws the error: _Cannot open C:\Users\username\.local\bin\pipx-script.py_.

Running pipx.exe via full path

The path strings will need "doubled backslashes" (i.e. C:\\Users\\bob) for escaping on subcommands such as list or install.

pipx list --verbose

C:\Users\username\.local\pipx\venvs\pipx\Scripts>pipx.exe list --verbose
venvs are in C:\Users\username\.local\pipx\venvs
symlinks to binaries are in C:\Users\username\.local\bin
  File "<string>", line 9
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
   package pipx, 0.0.0.14
pipx (_list_installed_package:313):     python: C:\Users\username\.local\pipx\venvs\pipx\Scripts\python.exe

[pfmoore]

The emoji appear as boxes in both cmd and PowerShell since neither shell appears to support utf-8 by default.

The boxes are because you're using a font that doesn't include the emoji. The console (and Python 3.6+) supports UTF-8 by default, the shell you're using isn't relevant. But beware, if you pipe output to a file or another process, the encoding of sys.stdout changes (to the user's defined 8-bit codepage).So you should test that get-pipx.py >get-pipx.log doesn't fail with an encoding error. If it does (which I'd expect), you likely need to avoid printing emoji (you should never print characters that the output encoding doesn't support anyway, this is really just a particular case of this). The issue can probably be demonstrated on Unix, simply by setting the encoding (LANG environment variable, IIRC) to something that isn't UTF-8.

See https://github.com/linkedin/shiv/issues/82 for something similar.

[cs01]

Without interactively debugging that last one myself on a windows computer I don’t know if I’ll be able to fix it. I will have to get ahold of one.

I agree that windows support is important. I see it as the last big piece that needs to fall into place before I start trying to get it to be seen as the “right way” to install global binaries, at least for my projects.

[cs01]

I tried to set up a Docker image and sort of got it working but then ran into some path issues, so I gave up and got ahold of a Window's laptop. I got it working there and updated master to v0.9.

LMK how it works on your machines. I would not be surprised if there are a few more surprises left 😆but I'm hopeful I got it working for the majority of cases.

[sahensley]

Awesome!!

I did all my testing with the black and httpie packages. I didn't have a chance to test reinstall-all yet (only one Python executable configured currently) but I'll make sure to test it something in the coming week.

Testing results:

• binary runs work (minor note, ephemeral is misspelled as ephemral in the main help output)
  
  
  
  • binary with spec works, i.e. pipx --spec httpie http https://duckduckgo.com
  
  
• install then running black works
  
  
  
  • install with --spec works
  
  
• upgrade works
• update-all works
• uninstall deletes the expected venv but leaves behind binaries in %userprofile%\.local\bin
• uninstall-all deletes package venvs (leaving binaries in bin) but fails to remove the pipx venv
• reinstall-all did not test
• list works

Here's the uninstall-all error:

PermissionError: [WinError 5] Access is denied: 'C:\\Users\\username\\.local\\pipx\\venvs\\pipx\\Scripts\\python.exe'

Keep up the awesome work! 🎉

[cs01]

Cool thanks.

• I worked around the permission error so the exception isn't raised, but the uninstall is still a little unclean. I think it has to do with the lack of symlinks on Windows. I tried with pipsi and it runs into the PermissionError and exits with a bare exception, so this is at least better than that.
• updated the code to remove the binaries
• fixed spelling error

Fixes are in v0.9.2 and published to pipx-app on PyPI.

[cs01]

This issue is getting really large with all the comments. Since windows works in 0.9.2 I am going to close it so we can work new any new bugs in dedicated issues. Thank you both for helping me out here!