Virtualenv: Code signing changes in Sierra result in a venv python executable with a broken code signature

[dreness]

Steps To Reproduce:

1. 16A323
2. sudo python -m ensurepip
3. sudo pip install -U pip
4. sudo pip install virtualenv
5. virtualenv sierra_venv
6. codesign -v sierra_venv/bin/python

Results:

sierra_venv/bin/python: invalid Info.plist (plist or signature have been modified)

Regression:
Yes. Same steps on ElCap produce:

elcap_venv/bin/python: code object is not signed at all
In architecture: x86_64

Notes:
This is significant because at least in some cases, having no code signature is better than having a broken code signature - for example when interacting with keychain.

In OS X 10.11, the binary being copied out of the python framework bundle (per https://github.com/pypa/virtualenv/blob/master/virtualenv.py#L1303) is not signed:

vm209:~ local$ codesign -v /System/Library/Frameworks/Python.framework/Resources/Python.app/Contents/MacOS/Python 
/System/Library/Frameworks/Python.framework/Resources/Python.app/Contents/MacOS/Python: code object is not signed at all
In architecture: x86_64

... but in sierra it is:

vm301:~ local$ codesign -v /System/Library/Frameworks/Python.framework/Resources/Python.app/Contents/MacOS/Python 
vm301:~ local$ echo $?
0

When code resources are signed as part of a bundle, the code signature is stored in a single location in the bundle, instead of inside the individual binaries. However, the signed binaries also have an LC_CODE_SIGNATURE load command which is still present in the binary after it has been copied out.

vm301:~ local$ codesign -v /System/Library/Frameworks/Python.framework/Resources/Python.app/Contents/MacOS/Python
vm301:~ local$ echo $?
0
vm301:~ local$ cp /System/Library/Frameworks/Python.framework/Resources/Python.app/Contents/MacOS/Python ~/python
vm301:~ local$ codesign -v ~/python
/Users/local/python: invalid Info.plist (plist or signature have been modified)
In architecture: x86_64

I'm not sure what the best fix is here...

[dreness]

Perhaps just: codesign -f -s - to re-sign it ad hoc.

vm301:~ local$ cp /System/Library/Frameworks/Python.framework/Resources/Python.app/Contents/MacOS/Python ~/python
vm301:~ local$ codesign -v ~/python
/Users/local/python: invalid Info.plist (plist or signature have been modified)
In architecture: x86_64
vm301:~ local$ codesign -f -s - ~/python
/Users/local/python: replacing existing signature
vm301:~ local$ codesign -v ~/python
vm301:~ local$ echo $?
0
vm301:~ local$ codesign -d -vvv ~/python
Executable=/Users/local/python
Identifier=python-55554944599d1781e4da397a9311e670ed5e1dab
Format=Mach-O universal (i386 x86_64)
CodeDirectory v=20100 size=256 flags=0x2(adhoc) hashes=3+2 location=embedded
Hash type=sha256 size=32
CandidateCDHash sha256=b0fe9fbd3cc163db5788317e1a0391c09acf5934
Hash choices=sha256
CDHash=b0fe9fbd3cc163db5788317e1a0391c09acf5934
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

[dreness]

More fallout from this problem: if you try to run a GUI app out of a virtual environment, you might be told:

This program needs access to the screen.
Please run with a Framework build of python, and only when you are
logged in on the main display of your Mac.

The workaround is the same: replace bin/python in the virtualenv with /usr/bin/python

[ocxo]

This has surfaced within the context of brew, awscli, and Little Snitch (which are outside the scope of virtualenv) but it sounds what we're seeing is really a problem with virtualenv itself.

What would be involved in having a properly signed version for macOS?

[dreness]

@fromonesrc (I don't speak for pypa, but:) I guess it depends on what 'properly signed' means. If any valid signature is OK - and it generally is, unless you are trying to use entitlements granted only to specific identities - I would try the ad-hoc resign of the virtualenv python executable, as mentioned in my Oct 20 comment. If this works for you, perhaps this post-flight action could be integrated by pypa as an option when creating a virtualenv on macOS.

If instead 'properly signed' means 'signed with an Apple developer ID that is trusted by macOS by default', that is probably a much larger request - pypa would need some system for coordinating the acquisition, renewal, maintenance, and use of the signing resources (Apple developer ID cert / private key). Or more conservatively, pip could grow an option to specify existing (user-maintained) signing credentials to be used at venv creation time.

[AlexanderOMara]

I looked into this a bit, and the problem appears related to this code:

https://github.com/pypa/virtualenv/blob/e5922654e10c4c5494241112a3645079db186222/virtualenv.py#L1312-L1324

This replaces the python binary copied from /usr/bin/python earlier, with the (non-stub?) binary here:

/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

As you can see, this binary actually is in an app bundle Python.app which means it is copied away from the Info.plist and breaks the signature.

Unfortunately, I do not know of a simple solution. Perhaps instead the whole Python.app bundle needs to be copied and the main binary symlinked?

At any rate, I would love to see this fixed!

[AlexanderOMara]

Unfortunately, I do not know of a simple solution. Perhaps instead the whole Python.app bundle needs to be copied and the main binary symlinked?

Oh, that actually won't solve anything because just below that, the code modifies the binary itself, which will break the signature no matter what.

https://github.com/pypa/virtualenv/blob/e5922654e10c4c5494241112a3645079db186222/virtualenv.py#L1337-L1355

I guess the only solution then would be to remove the code signature. One option would be the undocumented codesign --remove-signature command option. Either that or an alternative way to remove the signature would be needed. With some effort I'm sure removing the code signature section could be done in just Python.

Any hope for feedback from the project maintainers?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.

[dreness]

This is becoming a popular strategy for keeping those pesky bug counts under control...

[gaborbernat]

is still an issue with latest? if so we'll reopen 😀

[gaborbernat]

If it's an issue I would like to also see a proposed solution, especially from people using/working on MacOS (aka someone from Apple). Given this we can define some deliverable for a potential PR from someone willing.

[HormyAJP]

I just tripped over the same issue. Flagged by Little Snitch, and I'm glad it was.

I personally consider this a security concern. Since you're breaking the code signature I no longer have any guarantees over the integrity of the binary I'm running. Your modifications are harmless but someone else's might not be.

An attack vector might be something like the following: Since venvs are (at least on my machine) commonly installed as my user (non elevated), any malware which ran as my user on my machine could target venv binaries and change the executable. If I'm forced to trust the unsigned binary then I'm now forced to trust the malware. You could argue that the ship has already sailed if malware is already on my system, but the trust element is important. With good firewall software I can block the malware doing nasty (network stuff) itself directly. However, I will need to trust python so I'm more exposed with this issue.

A workaround for me (I guess) is to brew install python - although frankly brew comes with a whole bunch of other security concerns in my book. The only reason I'm hitting this is that I try not to use python2 anymore. But in certain cases I'm forced to and thus rely on the system binary.

[dreness]

is still an issue with latest? if so we'll reopen 😀

It's still an issue in 10.14, anyway. Someone more brave than I can report on 10.15...

xomg% sw_vers                                             
ProductName:    Mac OS X
ProductVersion: 10.14.6
BuildVersion:   18G87

xomg% virtualenv --verbose -p /usr/bin/python testenv 
Running virtualenv with interpreter /usr/bin/python
Creating /private/tmp/testenv/lib/python2.7
Symlinking Python bootstrap modules
  Symlinking /private/tmp/testenv/lib/python2.7/lib-dynload
  Symlinking /private/tmp/testenv/lib/python2.7/config
  Symlinking /private/tmp/testenv/lib/python2.7/os.py
  Ignoring built-in bootstrap module: posix
  Symlinking /private/tmp/testenv/lib/python2.7/posixpath.py
  Cannot import bootstrap module: nt
  Symlinking /private/tmp/testenv/lib/python2.7/ntpath.py
  Symlinking /private/tmp/testenv/lib/python2.7/genericpath.py
  Symlinking /private/tmp/testenv/lib/python2.7/fnmatch.py
  Symlinking /private/tmp/testenv/lib/python2.7/locale.py
  Symlinking /private/tmp/testenv/lib/python2.7/encodings
  Symlinking /private/tmp/testenv/lib/python2.7/codecs.py
  Symlinking /private/tmp/testenv/lib/python2.7/stat.py
  Symlinking /private/tmp/testenv/lib/python2.7/UserDict.py
  Symlinking /private/tmp/testenv/lib/python2.7/copy_reg.py
  Symlinking /private/tmp/testenv/lib/python2.7/types.py
  Symlinking /private/tmp/testenv/lib/python2.7/re.py
  Symlinking /private/tmp/testenv/lib/python2.7/sre.py
  Symlinking /private/tmp/testenv/lib/python2.7/sre_parse.py
  Symlinking /private/tmp/testenv/lib/python2.7/sre_constants.py
  Symlinking /private/tmp/testenv/lib/python2.7/sre_compile.py
  Symlinking /private/tmp/testenv/lib/python2.7/warnings.py
  Symlinking /private/tmp/testenv/lib/python2.7/linecache.py
  Symlinking /private/tmp/testenv/lib/python2.7/_abcoll.py
  Symlinking /private/tmp/testenv/lib/python2.7/abc.py
  Symlinking /private/tmp/testenv/lib/python2.7/_weakrefset.py
Creating /private/tmp/testenv/lib/python2.7/site-packages
Writing /private/tmp/testenv/lib/python2.7/site.py
Writing /private/tmp/testenv/lib/python2.7/orig-prefix.txt
Writing /private/tmp/testenv/lib/python2.7/no-global-site-packages.txt
Creating parent directories for /private/tmp/testenv/include
Symlinking /private/tmp/testenv/include/python2.7
Creating /private/tmp/testenv/bin
New python executable in /private/tmp/testenv/bin/python
Changed mode of /private/tmp/testenv/bin/python to 0755
Symlinking /private/tmp/testenv/.Python
Testing executable with /private/tmp/testenv/bin/python -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
Got sys.prefix result: u'/private/tmp/testenv'
Creating /private/tmp/testenv/lib/python2.7/distutils
Writing /private/tmp/testenv/lib/python2.7/distutils/__init__.py
Writing /private/tmp/testenv/lib/python2.7/distutils/distutils.cfg
Installing setuptools, pip, wheel...
  Looking in links: /usr/local/lib/python3.7/site-packages, /usr/local/lib/python3.7/site-packages/virtualenv_support, /Library/Python/2.7/site-packages/virtualenv_support
  Collecting setuptools
    Using cached https://files.pythonhosted.org/packages/6a/9a/50fadfd53ec909e4399b67c74cc7f4e883488035cfcdb90b685758fa8b34/setuptools-41.4.0-py2.py3-none-any.whl
  Collecting pip
    Using cached https://files.pythonhosted.org/packages/4a/08/6ca123073af4ebc4c5488a5bc8a010ac57aa39ce4d3c8a931ad504de4185/pip-19.3-py2.py3-none-any.whl
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
  Installing collected packages: setuptools, pip, wheel
  Successfully installed pip-19.3 setuptools-41.4.0 wheel-0.33.6
...Installing setuptools, pip, wheel...done.
Writing /private/tmp/testenv/bin/activate
Writing /private/tmp/testenv/bin/activate.fish
Writing /private/tmp/testenv/bin/activate_this.py
Writing /private/tmp/testenv/bin/activate.csh
Writing /private/tmp/testenv/bin/python-config
Changed mode of /private/tmp/testenv/bin/python-config to 0755

xomg% codesign -v testenv/bin/python
testenv/bin/python2: invalid Info.plist (plist or signature have been modified)
In architecture: x86_64

xomg% sha512sum /usr/bin/python    
ab584b12d3ccedb04fddade3cb7e6fb57fa894881461091e03340bf2947e4e5057b6fce8d85cb98f7d1008d362f062bce5ca060f52520089bfb4624b0b3b638f  /usr/bin/python

xomg% sha512sum testenv/bin/python 
bdc76e0b05b717da8ca533c0619866241c876739b90ce96ebc06f387fd0719e0619191378f7b59e9174e0ac561dc4df65678a157047c62504b01ff9fecae429b  testenv/bin/python