Setuptools: Please make it possible to use '#!/usr/bin/env ...' for wrapper scripts

[ghost]

Originally reported by: mgorny (Bitbucket: mgorny, GitHub: mgorny)

---

I have recently hit a number of issues that Python scripts generated by setuptools could not have been run because the shebang length exceeded the kernel limit (due to virtualenv being used in long path). Sadly, I see no solution to the problem except for using shebangs like:

#!/usr/bin/env pythonX.Y

and relying on the PATH lookup to find the correct executable without hitting any hard path limits.

Please make it possible to write #!/usr/bin/env shebangs in scripts generated via entry_points.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/494

[MikeDacre]

I would also like to request this feature for a different reason: writing portable python 2/3 compatible code that can be used on systems with changing python environments.

A lot of scientific compute clusters make use of some sort of modules system to allow the install of multiple competing software side-by-side. Running module load x alters a suite of environmental variables to make that software run.

This creates an issue for any python code that works natively in both python2 and 3, because if the user installs the software with setuptools for both python 2 and 3, the last used interpreter is hardcoded into the shebang line. When the environment changes and a different version of python is used, those scripts will now fail to run as their parent interpreter cannot function in the new environment.

The easy way around this is to make the shebang line #!/usr/bin/env python. Please add an option to the setup.py config to set the interpretter to env.

For a full discussion of this issue, please see my stackoverflow question here: http://stackoverflow.com/questions/37874445/override-the-shebang-mangling-in-python-setuptools

[anthrotype]

Haven't tried it, but the distutils build_scripts command should accept an "executable" option.
Maybe try adding this to a setup.cfg file and see if works:

[build_scripts]
executable=/usr/bin/env python

[anthrotype]

Oh I just noticed you did try that, but didn't work.. 🙃

[ex-nerd]

Outside of the "different versions" need, this is also a huge problem point for deep virtualenvs, which end up with shebang lines too long for bash/linux to interpret, which renders them unusable. To add insult to injury, the executable flag only works for build mode, not develop.

[tedivm]

I'm shocked this hasn't been dealt with yet, as it kills continuous integration setups for anyone attempting to use jenkins and virtualenvs. There really should be some way to fix this so that broken scripts don't get generated.

[tdegeus]

Like @tedivm, I am surprised that this issue does not appear to be solved. I cannot even find a workaround. I have exactly the same problem as @MikeDacre and completely support his argumentation. There should be at least the option to manually specify the shebang. In the worst case a robust workaround should be published.

[tedivm]

Yeah, it's been about 10 months since I complained about it, and since then at least four more projects have highlighted this issue as affecting them in github issues. It's really shocking that such a big issue would be completely ignored. There are so many CI and testing systems which this bug breaks that I really can't believe that no one on the dev teams are even willing to respond to this issue.

[jaraco]

Apologies for not having responded to the issue sooner.

I think the most important thing we need to understand better is under what circumstances does this issue occur? Are these scripts generated by easy_install or by pip/distlib? If the scripts are generated by easy_install, that functionality is largely deprecated and should be replaced by pip, in which case improvements to setuptools will not help.

Can you help us replicate the issue? What commands do you run; what is the result and what is the expected result?

[njsmith]

Using env is just wrong... it breaks the guarantee that you don't have to activate a venv, you can just do path/to/venv/bin/whatever.

If you need to support longer shbangs, then I think you have to do something like:

#!/bin/sh

exec /full/path/to/venv/bin/python ...

[tedivm]

@jaraco- the issue occurs with the scripts created to wrap the entry_points defined in setup.py. When those scripts are created they set the interpreter to the python binary to whatever python instance is available.

In 90% of environments those scripts work just fine. The problem is that the the linux kernel has a limit on how long the path to the interpreter can be. If you create a venv in a directory with a long path the scripts will break. This comes up a lot in continuous integration situations- for instance, right now you essentially can not use these wrapper scripts (which also includes pip itself, as just one example) at all because Jenkins creates paths with really long names for the individual workspaces.

Changing the interpreter to !#/usr/bin/env python would solve this problem by having the system look up the appropriate python interpreter to use (and, if the venv was active, this would mean it uses that). However the solution @njsmith provided is definitely better, as it will still solve the problem but but will work in even more circumstances.

[tdegeus]

@jaraco

I can second the problem description of @tedivm. Just having the usual !#/usr/bin/env python and have the system deal with everything appears to be the simplest and most logical. The current solution of specifying the path Python explicitly seems to me to have mostly down sites (see an initial attempt to list pros and cons below, these can, however, be surely extended and refined), which would I think go against needing @njsmith's workaround (which BTW would completely mess with my editor). In any case I would find it logical to also include a mechanism in which setuptools does not mess with whatever shebang I use. I may have good reasons that were not considered in setuptools’ development.

Pros of explicitly specifying the path (current behaviour):

• You can normally guarantee the available packages.

Cons of explicitly specifying the path (current behaviour):

• Environments and paths are the usual things that one controls. This is flexible and transparent and has been so for decades. pip's will to mess with this and invent its own mechanism seems to reduce this transparency greatly.
• If for example, a user install is done the script is always stored in the same location (e.g. ~/.local/bin/). I.e. even if you want the script to 'belong' to different Python versions you would simply always end up with the last one you installed. If you switch between different environments, you are for to 'reinstall' every time.
• In many package managers (surely in conda and spack) the path is too long to fit in the shebang, completely braking the installed script.

Note: On my system (macOS) not even the right Python executable is used: the system's default is selected, while I run pip with HomeBrew's version.