Pip: 'scripts' lose executable priveleges from pip install

Created on 28 Jan 2020 · 12Comments · Source: pypa/pip

Environment

pip version: 20.0.2
Python version: 2.7.5
OS: RHEL 7.7/SELinux
setuptools version: 44.0.0

Description

pip-installing a package that contains a setup.py that specifies the scripts argument causes the resulting file to lost its executable permission(s).

Reproducible example located at:

https://github.com/bsolomon1124/pip-scripts-hostbits-mvce

Expected behavior

The source repository bin/my_script is executable (via chmox a+rx bin/my_script). It loses this permission when installed by pip. It should maintain its executable permission for all users.

How to Reproduce

$ pip install git+ssh://[email protected]/bsolomon1124/pip-scripts-hostbits-mvce.git
$ ls -la /usr/bin/my_script  # or ls -la /usr/local/bin/my_script

Output

$ ls -la /usr/bin/my_script
-rw-r--r--. 1 root root 39 Jan 28 19:10 /usr/bin/my_script

Alternate example

$ yum install gcc python-devel
$ pip install uwsgi
$ ls -la /usr/bin/uwsgi
-rw-r--r--. 1 root root 1369984 Jan 28 18:53 /usr/bin/uwsgi
$ echo $PATH
/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

auto-locked support

Source

bsolomon1124

Most helpful comment

Is /tmp mounted noexec? It may be related to #6364.

chrahunt on 29 Jan 2020

👍2

All 12 comments

This issue also appears to exist on Windows 10 Enterprise via:

py -2 -m pip install git+ssh://[email protected]/bsolomon1124/pip-scripts-hostbits-mvce.git

The script is correctly placed in c:\python27\scripts\my_script but not able to be found even though c:\python27\scripts\my_script is on PATH as well.

bsolomon1124 on 28 Jan 2020

I forked your repo, and tried this inside an alpine image:

docker run -it -v $(pwd):/app python:3.7-alpine sh
# apkg add git
# pip install git+https://github.com/niall-byrne/pip-scripts-hostbits-mvce.git

/app # ls -la /usr/local/bin/my_script
-rwxr-xr-x    1 root     root            45 Jan 28 19:24 /usr/local/bin/my_script

I tried it in python 2:

docker run -it python:2.7-alpine -v $(pwd):/app sh
# apkg add git
`# pip install git+https://github.com/bsolomon1124/pip-scripts-hostbits-mvce.git

/ # ls -la /usr/local/bin/my_script
-rwxr-xr-x    1 root     root            45 Jan 28 19:29 /usr/local/bin/my_script

Maybe it's OS specific ?

Also seems ok in:

centos/python-36-centos7:latest
- centos/python-27-centos7:latest

niall-byrne on 28 Jan 2020

Interesting. I tried via git+https rather than git+ssh and still see the same issue. Maybe it's worth noting that I have SELinux in Enforcing mode but I doubt it. And that this is not a 'fresh' instance but rather a (probably old) VM that was handed to me.

bsolomon1124 on 28 Jan 2020

I encountered this issue originally on amazon linux if I remember correctly, so I tested that too: amazonlinux:latest

I can't seem to recreate this. I'm going to close my original ticket.

niall-byrne on 28 Jan 2020

Okay. One thing I do notice is that your install location is /usr/local/bin/my_script whereas mine is /usr/bin/my_script. (Some detail on that here and here.) Feeling around in the dark here but that is one difference at least. I believe that can be overriden via [install] prefix in setup.cfg.

bsolomon1124 on 28 Jan 2020

That is relevant if I remember, we weren't having problems with this until we rebuilt the AMI we were using, and the script was being installed in local under the new image.

Hey wait a minute, we had SELinux in play too... that might be a good lead...

niall-byrne on 28 Jan 2020

setenforce 0 did not do a whole lot. But let me check altering the prefix to /usr/local.

bsolomon1124 on 28 Jan 2020

Scratch that, it looks like [install] prefix only controls where lib/ stuff goes; scripts still go to /usr/bin for me.

bsolomon1124 on 28 Jan 2020

Output of pip install -vvv, interestingly, does mention mode being changed to 755:

$ pip install -vvv git+ssh://[email protected]/bsolomon1124/pip-scripts-hostbits-mvce.git
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-KXVZnX
Created temporary directory: /tmp/pip-req-tracker-SkIvHw
Initialized build tracking at /tmp/pip-req-tracker-SkIvHw
Created build tracker: /tmp/pip-req-tracker-SkIvHw
Entered build tracker: /tmp/pip-req-tracker-SkIvHw
Created temporary directory: /tmp/pip-install-ru6Bo7
Collecting git+ssh://****@github.com/bsolomon1124/pip-scripts-hostbits-mvce.git
  Created temporary directory: /tmp/pip-req-build-QR5yxR
  Cloning ssh://****@github.com/bsolomon1124/pip-scripts-hostbits-mvce.git to /tmp/pip-req-build-QR5yxR
  Running command git clone -q 'ssh://****@github.com/bsolomon1124/pip-scripts-hostbits-mvce.git' /tmp/pip-req-build-QR5yxR
  Added git+ssh://****@github.com/bsolomon1124/pip-scripts-hostbits-mvce.git to build tracker '/tmp/pip-req-tracker-SkIvHw'
    Running setup.py (path:/tmp/pip-req-build-QR5yxR/setup.py) egg_info for package from git+ssh://****@github.com/bsolomon1124/pip-scripts-hostbits-mvce.git
    Running command python setup.py egg_info
    running egg_info
    creating /tmp/pip-req-build-QR5yxR/pip-egg-info/pip_scripts_hostbits_mvce.egg-info
    writing /tmp/pip-req-build-QR5yxR/pip-egg-info/pip_scripts_hostbits_mvce.egg-info/PKG-INFO
    writing top-level names to /tmp/pip-req-build-QR5yxR/pip-egg-info/pip_scripts_hostbits_mvce.egg-info/top_level.txt
    writing dependency_links to /tmp/pip-req-build-QR5yxR/pip-egg-info/pip_scripts_hostbits_mvce.egg-info/dependency_links.txt
    writing manifest file '/tmp/pip-req-build-QR5yxR/pip-egg-info/pip_scripts_hostbits_mvce.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-req-build-QR5yxR/pip-egg-info/pip_scripts_hostbits_mvce.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-req-build-QR5yxR/pip-egg-info/pip_scripts_hostbits_mvce.egg-info/SOURCES.txt'
  Source in /tmp/pip-req-build-QR5yxR has version 0.0.1, which satisfies requirement pip-scripts-hostbits-mvce==0.0.1 from git+ssh://****@github.com/bsolomon1124/pip-scripts-hostbits-mvce.git
  Removed pip-scripts-hostbits-mvce==0.0.1 from git+ssh://****@github.com/bsolomon1124/pip-scripts-hostbits-mvce.git from build tracker '/tmp/pip-req-tracker-SkIvHw'
Building wheels for collected packages: pip-scripts-hostbits-mvce
  Created temporary directory: /tmp/pip-wheel-yOyXLH
  Building wheel for pip-scripts-hostbits-mvce (setup.py) ...   Destination directory: /tmp/pip-wheel-yOyXLH
  Running command /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-QR5yxR/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-QR5yxR/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-yOyXLH
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/app
  copying app/__init__.py -> build/lib/app
  running build_scripts
  creating build/scripts-2.7
  copying and adjusting bin/my_script -> build/scripts-2.7
  changing mode of build/scripts-2.7/my_script from 644 to 755
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.linux-x86_64
  creating build/bdist.linux-x86_64/wheel
  creating build/bdist.linux-x86_64/wheel/app
  copying build/lib/app/__init__.py -> build/bdist.linux-x86_64/wheel/app
  running install_egg_info
  running egg_info
  creating pip_scripts_hostbits_mvce.egg-info
  writing pip_scripts_hostbits_mvce.egg-info/PKG-INFO
  writing top-level names to pip_scripts_hostbits_mvce.egg-info/top_level.txt
  writing dependency_links to pip_scripts_hostbits_mvce.egg-info/dependency_links.txt
  writing manifest file 'pip_scripts_hostbits_mvce.egg-info/SOURCES.txt'
  reading manifest file 'pip_scripts_hostbits_mvce.egg-info/SOURCES.txt'
  writing manifest file 'pip_scripts_hostbits_mvce.egg-info/SOURCES.txt'
  Copying pip_scripts_hostbits_mvce.egg-info to build/bdist.linux-x86_64/wheel/pip_scripts_hostbits_mvce-0.0.1-py2.7.egg-info
  running install_scripts
  creating build/bdist.linux-x86_64/wheel/pip_scripts_hostbits_mvce-0.0.1.data
  creating build/bdist.linux-x86_64/wheel/pip_scripts_hostbits_mvce-0.0.1.data/scripts
  copying build/scripts-2.7/my_script -> build/bdist.linux-x86_64/wheel/pip_scripts_hostbits_mvce-0.0.1.data/scripts
  changing mode of build/bdist.linux-x86_64/wheel/pip_scripts_hostbits_mvce-0.0.1.data/scripts/my_script to 755
  creating build/bdist.linux-x86_64/wheel/pip_scripts_hostbits_mvce-0.0.1.dist-info/WHEEL
  creating '/tmp/pip-wheel-yOyXLH/pip_scripts_hostbits_mvce-0.0.1-py2-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
  adding 'app/__init__.py'
  adding 'pip_scripts_hostbits_mvce-0.0.1.data/scripts/my_script'
  adding 'pip_scripts_hostbits_mvce-0.0.1.dist-info/METADATA'
  adding 'pip_scripts_hostbits_mvce-0.0.1.dist-info/WHEEL'
  adding 'pip_scripts_hostbits_mvce-0.0.1.dist-info/top_level.txt'
  adding 'pip_scripts_hostbits_mvce-0.0.1.dist-info/RECORD'
  removing build/bdist.linux-x86_64/wheel
done
  Created wheel for pip-scripts-hostbits-mvce: filename=pip_scripts_hostbits_mvce-0.0.1-py2-none-any.whl size=1655 sha256=7f7b8fd118017c1ba854387d1dd1d3ea60e12984484ea03b481260a391ddf524
  Stored in directory: /tmp/pip-ephem-wheel-cache-KXVZnX/wheels/41/67/1e/ee274760289e65d944f2dfaef31604564f0d83e66147d5c89f
Successfully built pip-scripts-hostbits-mvce
Installing collected packages: pip-scripts-hostbits-mvce
  Created temporary directory: /tmp/pip-unpacked-wheel-cxGDnd

Successfully installed pip-scripts-hostbits-mvce-0.0.1
Cleaning up...
  Removing source in /tmp/pip-req-build-QR5yxR
Removed build tracker: '/tmp/pip-req-tracker-SkIvHw'

But at some point, this is not respected:

$ ls -la /usr/bin/my_script
-rw-r--r--. 1 root root 39 Jan 28 20:07 /usr/bin/my_script

Ref: https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Lib/distutils/command/install_scripts.py#L41

And:

$ python setup.py build
running build
running build_py
creating build
creating build/lib
creating build/lib/app
copying app/__init__.py -> build/lib/app
running build_scripts
creating build/scripts-2.7
copying and adjusting bin/my_script -> build/scripts-2.7
changing mode of build/scripts-2.7/my_script from 644 to 755

$ ls -la build/scripts-2.7/my_script
-rwxr-xr-x. 1 root root 39 Jan 28 20:16 build/scripts-2.7/my_script

Installing from the local source to the current directory also works:

$ python setup.py build
running build
running build_py
creating build
creating build/lib
creating build/lib/app
copying app/__init__.py -> build/lib/app
running build_scripts
creating build/scripts-2.7
copying and adjusting bin/my_script -> build/scripts-2.7
changing mode of build/scripts-2.7/my_script from 644 to 755

$ python setup.py install
running install
running bdist_egg
running egg_info
creating pip_scripts_hostbits_mvce.egg-info
writing pip_scripts_hostbits_mvce.egg-info/PKG-INFO
writing top-level names to pip_scripts_hostbits_mvce.egg-info/top_level.txt
writing dependency_links to pip_scripts_hostbits_mvce.egg-info/dependency_links.txt
writing manifest file 'pip_scripts_hostbits_mvce.egg-info/SOURCES.txt'
reading manifest file 'pip_scripts_hostbits_mvce.egg-info/SOURCES.txt'
writing manifest file 'pip_scripts_hostbits_mvce.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/app
copying build/lib/app/__init__.py -> build/bdist.linux-x86_64/egg/app
byte-compiling build/bdist.linux-x86_64/egg/app/__init__.py to __init__.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
installing scripts to build/bdist.linux-x86_64/egg/EGG-INFO/scripts
running install_scripts
running build_scripts
creating build/bdist.linux-x86_64/egg/EGG-INFO/scripts
copying build/scripts-2.7/my_script -> build/bdist.linux-x86_64/egg/EGG-INFO/scripts
changing mode of build/bdist.linux-x86_64/egg/EGG-INFO/scripts/my_script to 755
copying pip_scripts_hostbits_mvce.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pip_scripts_hostbits_mvce.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pip_scripts_hostbits_mvce.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying pip_scripts_hostbits_mvce.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/pip_scripts_hostbits_mvce-0.0.1-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing pip_scripts_hostbits_mvce-0.0.1-py2.7.egg
Copying pip_scripts_hostbits_mvce-0.0.1-py2.7.egg to /usr/lib/python2.7/site-packages
Adding pip-scripts-hostbits-mvce 0.0.1 to easy-install.pth file
Installing my_script script to /usr/bin

Installed /usr/lib/python2.7/site-packages/pip_scripts_hostbits_mvce-0.0.1-py2.7.egg
Processing dependencies for pip-scripts-hostbits-mvce==0.0.1
Finished processing dependencies for pip-scripts-hostbits-mvce==0.0.1

$ ls -la /usr/bin/my_script
-rwxr-xr-x. 1 root root 226 Jan 28 20:23 /usr/bin/my_script

bsolomon1124 on 28 Jan 2020

Is /tmp mounted noexec? It may be related to #6364.

chrahunt on 29 Jan 2020

👍2

Yes indeed it is @chrahunt. mount shows:

```
/dev/foo/bar on /tmp type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,noquota)
````

Great find. And that would explain why installing from local source causes no issue, because that cuts /tmp out of the equation.

Is there a known solution? We are not using containers and the VM's environment is quite constrained in terms of allowed operations. For instance, is there a way to tell pip to use something other than /tmp?

bsolomon1124 on 29 Jan 2020

👍1

Related helpful discussion for others with this issue: https://github.com/pypa/pip/issues/4462.

In this case,

mkdir pip-tmp
TMPDIR='./pip-tmp' pip install package_name && rmdir pip-tmp

works like a charm.

bsolomon1124 on 29 Jan 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

New resolver cannot installs distributions that only have pre releases

sbidoul · 3Comments

Pip issues while installing requirements, OSError: [Errno 1] Operation not permitted:

GregBorrelly · 3Comments

After pip install of lark the __init__.py file is empty

gyorireka · 3Comments

Don't deprecate dependency_links or let pip use a requirements.txt file when installing from git

wrouesnel · 3Comments

pip exits with success despite incompatible constraint

cjolowicz · 3Comments