Pipx: Some executables not linked during pipx install

[elgertam]

pipx install is not finding all executables for certain packages.

For example, here is what happened when I tried to install jupyter using pipx:

$ pipx install jupyter --include-deps
  installed package jupyter 1.0.0, Python 3.7.3
  These binaries are now globally available
    - iptest
    - iptest3
    - ipython
    - ipython3
    - jupyter-bundlerextension
    - jupyter-console
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-nbconvert
    - jupyter-nbextension
    - jupyter-notebook
    - jupyter-qtconsole
    - jupyter-run
    - jupyter-serverextension
    - jupyter-trust
done! ✨ 🌟 ✨
$ 
$ 
$ jupyter
-bash: jupyter: command not found
$ which jupyter
$ 
$ cd ~/.local/bin
$ ls *jupyter*
jupyter-bundlerextension jupyter-nbconvert        jupyter-run
jupyter-console          jupyter-nbextension      jupyter-serverextension
jupyter-kernel           jupyter-notebook         jupyter-trust
jupyter-kernelspec       jupyter-qtconsole
$ # Looking in ~/.local/bin, none found.
$ 
$ cd ~/.local/pipx/venvs/jupyter/bin/
$ ls *jupyter*
jupyter                  jupyter-migrate          jupyter-run
jupyter-bundlerextension jupyter-nbconvert        jupyter-serverextension
jupyter-console          jupyter-nbextension      jupyter-troubleshoot
jupyter-kernel           jupyter-notebook         jupyter-trust
jupyter-kernelspec       jupyter-qtconsole
$ # Looking in ~/.local/pipx/venvs/jupyter/bin. Expected no `jupyter` executable;
$ # found `jupyter` executable, as well as `jupyter-migrate` and
$ # `jupyter-troubleshoot`, none of which were not found in ~/.local/bin.

According to the readme's "How it Works" section, pipx install scans the new virtualenv's bin/ directory, then links found executables into ~/.local/bin.

Further, output included in a comment from a previous issue indicates that pipx properly linked jupyter, jupyter-migrate and jupyter-troubleshoot into ~/.local/bin. Is this a regression?

I am running macOS 10.14.4 and Python 3.7.3 installed with pyenv. pipx is likewise using Python 3.7.3.

[cs01]

Yup looks like this is a regression. Thanks for reporting.
With pipx 0.13.1.0:

  These binaries are now globally available
    - iptest
    - iptest3
    - ipython
    - ipython3
    - jupyter-bundlerextension
    - jupyter-console
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-nbconvert
    - jupyter-nbextension
    - jupyter-notebook
    - jupyter-qtconsole
    - jupyter-run
    - jupyter-serverextension
    - jupyter-trust
done! ✨ 🌟 ✨

With pipx 0.12.3.0:

    - easy_install
    - easy_install-3.5
    - easy_install-3.6
    - iptest
    - iptest3
    - ipython
    - ipython3
    - jsonschema
    - jupyter
    - jupyter-bundlerextension
    - jupyter-console
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-migrate
    - jupyter-nbconvert
    - jupyter-nbextension
    - jupyter-notebook
    - jupyter-qtconsole
    - jupyter-run
    - jupyter-serverextension
    - jupyter-troubleshoot
    - jupyter-trust
    - pygmentize
done! ✨ 🌟 ✨

[cs01]

Hmm I tried to bisect commits to find the one that caused the regression and I cannot reproduce it anymore. Even the latest release works for me. Maybe we one of the jupyter releases was updated which fixed this.

Since jupyter is a metapackage, pinning its version wouldn't help because it just installs the latest version of all its dependencies.

>> pipx --version
0.13.1.0
>> pipx install jupyter --include-deps
  installed package jupyter 1.0.0, Python 3.6.7
  These binaries are now globally available
    - easy_install
    - easy_install-3.6
    - iptest
    - iptest3
    - ipython
    - ipython3
    - jsonschema
    - jupyter
    - jupyter-bundlerextension
    - jupyter-console
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-migrate
    - jupyter-nbconvert
    - jupyter-nbextension
    - jupyter-notebook
    - jupyter-qtconsole
    - jupyter-run
    - jupyter-serverextension
    - jupyter-troubleshoot
    - jupyter-trust
    - pygmentize
done! ✨ 🌟 ✨
>> jupyter --version
4.4.0
>> jupyter-console --version
6.0.0
>> jupyter-notebook --version
5.7.8

[elgertam]

@cs01 I wondered about that myself. The difference in behavior between 0.12.3.0 and 0.13.1.0 seemed fairly convincing, but the behavior now is entirely what I expect. In fact, when I just ran it now, pipx ignored easy_install, pygmentize and some others, which is much better behavior to me.

$ pipx install --include-deps jupyter
  installed package jupyter 1.0.0, Python 3.7.3
  These binaries are now globally available
    - iptest
    - iptest3
    - ipython
    - ipython3
    - jupyter
    - jupyter-bundlerextension
    - jupyter-console
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-migrate
    - jupyter-nbconvert
    - jupyter-nbextension
    - jupyter-notebook
    - jupyter-qtconsole
    - jupyter-run
    - jupyter-serverextension
    - jupyter-troubleshoot
    - jupyter-trust
done! ✨ 🌟 ✨

Thank you for looking into this. I'll close the issue.

[aboutaaron]

pipx 0.13.1.1 installed the jupyter binary but it failed to symlink it to ~/.local/bin. I had to do that manually but everything worked after that.

[pcr910303]

@elgertam Hmm, it looks like in pipx 0.14.0.0 this behavior has regressed, I have installed package jupyter with --include-deps and I get this:

$ pipx install jupyter --include-deps
  installed package jupyter 1.0.0, Python 3.7.4
  These apps are now globally available
    - iptest
    - iptest3
    - ipython
    - ipython3
    - jupyter-bundlerextension
    - jupyter-console
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-nbconvert
    - jupyter-nbextension
    - jupyter-notebook
    - jupyter-qtconsole
    - jupyter-run
    - jupyter-serverextension
    - jupyter-trust
done! ✨ 🌟 ✨

$ pipx --version
0.14.0.0

[sahensley]

I'm reopening this issue to consolidate some of the other issues created around the jupyter package.

[pcr910303]

The latest release 0.15.1.1 still shows this behavior (at least in my machine), are there any pointers for me to fix things up?
It's a bit major pain for me, so I'm willing to try to fix this if I have some pointers.
Thanks in advance. (And sorry for just not reading the whole code, :-()

$ pipx install jupyter --include-deps
  installed package jupyter 1.0.0, Python 3.7.6
  These apps are now globally available
    - iptest
    - iptest3
    - ipython
    - ipython3
    - jupyter-bundlerextension
    - jupyter-console
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-nbconvert
    - jupyter-nbextension
    - jupyter-notebook
    - jupyter-qtconsole
    - jupyter-run
    - jupyter-serverextension
    - jupyter-trust
done! ✨ 🌟 ✨

$ pipx --version
0.15.1.1

[itsayellow]

I started trying to drill down into the dependencies, installing with --include-deps each time to try and find a time when the jupyter app was actually installed.

jupyter is NOT installed with pipx install jupyter-console

carvel:~ pipx install jupyter-console --include-deps
  installed package jupyter-console 6.0.0, Python 3.7.6
  These apps are now globally available
    - iptest
    - iptest3
    - ipython
    - ipython3
    - jupyter-console
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-run
    - pygmentize
done! ✨ 🌟 ✨

jupyter IS installed with pipx install jupyter-client, the dep of jupyter-console.

carvel:~ pipx install jupyter_client --include-deps 
  installed package jupyter_client 5.3.4, Python 3.7.6
  These apps are now globally available
    - jupyter
    - jupyter-kernel
    - jupyter-kernelspec
    - jupyter-migrate
    - jupyter-run
    - jupyter-troubleshoot
done! ✨ 🌟 ✨

venv_metadata_inspector.py finds jupyter in the loop: for name in pkg_resources.get_entry_map(dist) in get_apps()

jupyter-core actually seems to be the python package that installs jupyter. It is a dep of jupyter_client

[itsayellow]

I think the problem is in venv_metadata_inspector _dfs_package_apps:

Specifically if this function finds apps for a particular package, it will NOT search that package's dependency tree for apps. (Which is NOT correct.)

In this code:

    for d in dependencies:
        app_names = get_apps(d, bin_path)
        if app_names:
            apps = [str(Path(bin_path) / app) for app in app_names]
            app_paths_of_dependencies[d] = apps
        # recursively search for more
        if d not in app_paths_of_dependencies:
            # only search if this package isn't already listed to avoid
            # infinite recursion
            app_paths_of_dependencies = _dfs_package_apps(
                bin_path, d, app_paths_of_dependencies
            )

If the following statement is removed:

        if d not in app_paths_of_dependencies:

Then everything seems to work fine. jupyter is installed without that statement.

[itsayellow]

I have a functional branch that fixes this and still avoids infinite recursion. I'll make sure it's ok and submit tomorrow.

[pcr910303]

@itsayellow Ah☺️ thanks for inspecting this instead!

_{Sent with GitHawk}

[cs01]

Huge thanks to @itsayellow for drilling into this bug and fixing it. This one was pretty tricky.

[Ricyteach]

Thank you, @itsayellow!