pip10 segfaults when installing cryptography and pyopenssl

Created on 3 May 2018 · 14Comments · Source: pypa/pip

Pip version: 10
Python version: 2.x / 3.x
Operating system: ubuntu at least 16.04 & 18.04 / bionic + venv

Description:

pip10 segfaults when installing packages

What I've run:

docker run -ti --rm  ubuntu:18.04 bash
#
apt update -qq
apt install -y python python-dev build-essential m4 python-pip \
bash-completion python-virtualenv libexpat1-dev:amd64 \
libffi-dev:amd64 libgmp-dev:amd64 zlib1g-dev
#
virtualenv venv && . venv/bin/activate && pip install -U --ignore-installed pip \
&& pip install -U --ignore-installed cryptography pyOpenSSL\
&& pip install -U --ignore-installed cryptography pyOpenSSL

crash auto-locked bug

Source

kiorky

Most helpful comment

Okay here's the issue (huge thanks to @geofft for essentially solving this problem solely from being given an incomplete summary):

When pyOpenSSL is present in an environment pip 10's vendored urllib3 will automatically use it for generating SSLContexts. pyOpenSSL uses cryptography, which has a native shared object called _openssl.so that Python dlopens. dlopen mmaps the file and then methods are called as needed.

When using pip install --ignore-installed to install a wheel pip uses the clobber method, which ultimately calls shutil.copyfile. This function truncates the file and then writes the new data (opening a file to write truncates it). This is problematic because we've mmap'd the file and the process expects to be able to call symbols that no longer exist. When it attempts to do so (and said symbol isn't already resident in "real" memory) it segfaults.

The easy fix here is to add this to the clobber method:

+               if os.path.exists(destfile):
+                  os.unlink(destfile)
                shutil.copyfile(srcfile, destfile)

This will unlink the file (if it exists) and write a new one. Since the file was mmap'd Python will continue to be able to access the data it needs and exit normally.

reaperhulk on 12 May 2018

👍6

All 14 comments

cc @gagaro @regilero @reaperhulk

kiorky on 3 May 2018

Pinging @dstufft since he's likely the best guy for looking into this.

pradyunsg on 3 May 2018

One possible avenue of exploration for why this is occurring is that pip vendors urllib3, which looks at the environment to automatically activate alternate SSLContext support via pyopenssl (which uses cryptography's _openssl.so). When --ignore-installed is passed pip will overwrite these packages while simultaneously using them. Of course, if that's the issue then it's mysterious why pip 9 does not have the same problem.

Edit: Quick note, this isn't a Python 2.7 specific problem or an 18.04 only issue. The segfault occurs on 2.7/3.5 in Ubuntu 16.04 as well.

reaperhulk on 3 May 2018

i updated desc accordingly

kiorky on 3 May 2018

I poked at it for a few minutes. After apt install python-dbg, the traceback I get is:

(gdb) bt
#0  PyType_Ready () at ../Objects/typeobject.c:4155
#1  0x00005555555a2eff in PyObject_Hash (v=0x7ffff2fce2f0)
    at ../Objects/object.c:1121
#2  0x0000555555727abd in _PyDict_DelItemIf () at ../Objects/dictobject.c:913
#3  0x0000555555727a61 in remove_dead_weakref.lto_priv ()
    at ../Modules/_weakref.c:31
#4  0x000055555564901a in call_function (oparg=<optimized out>, 
    pp_stack=0x7fffffffc460) at ../Python/ceval.c:4372
#5  PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#6  0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#7  0x00005555556621a9 in function_call.lto_priv ()
    at ../Objects/funcobject.c:523
#8  0x000055555563aca7 in PyObject_Call (kw=0x0, arg=0x7ffff2606cd0, 
    func=0x7ffff3e5b578) at ../Objects/abstract.c:2547
#9  PyObject_CallFunctionObjArgs () at ../Objects/abstract.c:2774
#10 0x00005555556aa453 in handle_callback.lto_priv ()
    at ../Objects/weakrefobject.c:896
#11 0x0000555555682b07 in PyObject_ClearWeakRefs ()
    at ../Objects/weakrefobject.c:941
#12 0x00005555556821ec in subtype_dealloc.lto_priv ()
    at ../Objects/typeobject.c:997
#13 0x00005555556825b4 in dict_dealloc.lto_priv.332 (mp=0x7ffff2fb04b0)
    at ../Objects/dictobject.c:1086
#14 subtype_dealloc.lto_priv () at ../Objects/typeobject.c:1035
#15 0x000055555565c558 in frame_dealloc.lto_priv ()
    at ../Objects/frameobject.c:494
#16 0x000055555564dd93 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffc900, func=<optimized out>)
    at ../Python/ceval.c:4459
#17 call_function (oparg=<optimized out>, pp_stack=0x7fffffffc900)
    at ../Python/ceval.c:4392
#18 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#19 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#20 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffcae0, func=<optimized out>)
    at ../Python/ceval.c:4467
#21 call_function (oparg=<optimized out>, pp_stack=0x7fffffffcae0)
    at ../Python/ceval.c:4392
#22 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#23 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#24 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffccc0, func=<optimized out>)
    at ../Python/ceval.c:4467
#25 call_function (oparg=<optimized out>, pp_stack=0x7fffffffccc0)
---Type <return> to continue, or q <return> to quit---
    at ../Python/ceval.c:4392
#26 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#27 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#28 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffcea0, func=<optimized out>)
    at ../Python/ceval.c:4467
#29 call_function (oparg=<optimized out>, pp_stack=0x7fffffffcea0)
    at ../Python/ceval.c:4392
#30 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#31 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#32 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffd080, func=<optimized out>)
    at ../Python/ceval.c:4467
#33 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd080)
    at ../Python/ceval.c:4392
#34 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#35 0x000055555564dd72 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffd190, func=<optimized out>)
    at ../Python/ceval.c:4457
#36 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd190)
    at ../Python/ceval.c:4392
#37 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#38 0x000055555564dd72 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffd2a0, func=<optimized out>)
    at ../Python/ceval.c:4457
#39 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd2a0)
    at ../Python/ceval.c:4392
#40 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#41 0x000055555564dd72 in fast_function (nk=<optimized out>, 
    na=<optimized out>, n=1, pp_stack=0x7fffffffd3b0, func=<optimized out>)
    at ../Python/ceval.c:4457
#42 call_function (oparg=<optimized out>, pp_stack=0x7fffffffd3b0)
    at ../Python/ceval.c:4392
#43 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#44 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#45 0x00005555556621a9 in function_call.lto_priv ()
    at ../Objects/funcobject.c:523
#46 0x000055555567a78e in PyObject_Call (kw=0x0, arg=0x7ffff2452e68, 
    func=0x7ffff3b14578) at ../Objects/abstract.c:2547
#47 instancemethod_call.lto_priv () at ../Objects/classobject.c:2600
#48 0x000055555563a982 in PyObject_Call (kw=0x0, arg=0x7ffff1154c30, 
    func=0x7ffff3a12870) at ../Objects/abstract.c:2547
#49 PyObject_CallFunctionObjArgs () at ../Objects/abstract.c:2774
#50 0x000055555564b412 in PyEval_EvalFrameEx () at ../Python/ceval.c:2970
---Type <return> to continue, or q <return> to quit---
#51 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#52 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffdb20, func=<optimized out>)
    at ../Python/ceval.c:4467
#53 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdb20)
    at ../Python/ceval.c:4392
#54 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#55 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#56 0x000055555564e7d3 in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffdd00, func=<optimized out>)
    at ../Python/ceval.c:4467
#57 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdd00)
    at ../Python/ceval.c:4392
#58 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#59 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#60 0x000055555564e24e in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffdee0, func=<optimized out>)
    at ../Python/ceval.c:4467
#61 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdee0)
    at ../Python/ceval.c:4392
#62 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#63 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#64 0x000055555564c2fb in PyEval_EvalCode () at ../Python/ceval.c:669
#65 exec_statement (locals=0x7ffff7f81168, globals=0x7ffff7f81168, 
    prog=<optimized out>, f=<optimized out>) at ../Python/ceval.c:5090
#66 PyEval_EvalFrameEx () at ../Python/ceval.c:2118
#67 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#68 0x000055555564e24e in fast_function (nk=0, na=<optimized out>, 
    n=<optimized out>, pp_stack=0x7fffffffe2a0, func=<optimized out>)
    at ../Python/ceval.c:4467
#69 call_function (oparg=<optimized out>, pp_stack=0x7fffffffe2a0)
    at ../Python/ceval.c:4392
#70 PyEval_EvalFrameEx () at ../Python/ceval.c:3009
#71 0x00005555556468ca in PyEval_EvalCodeEx () at ../Python/ceval.c:3604
#72 0x00005555556621a9 in function_call.lto_priv ()
    at ../Objects/funcobject.c:523
#73 0x000055555563233e in PyObject_Call () at ../Objects/abstract.c:2547
#74 0x00005555556a3bf0 in RunModule.lto_priv.1251 () at ../Modules/main.c:197
#75 0x0000555555620406 in Py_Main () at ../Modules/main.c:592
#76 0x00007ffff7a05b97 in __libc_start_main (main=0x55555561fef0 <main>, 
    argc=8, argv=0x7fffffffe698, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffe688)
    at ../csu/libc-start.c:310
#77 0x000055555561fe0a in _start ()

(gdb) bt
#0  0x000055555562a174 in visit_decref () at ../Modules/gcmodule.c:360
#1  0x000055555562a334 in dict_traverse.lto_priv ()
    at ../Objects/dictobject.c:2198
#2  0x00005555556290e5 in subtract_refs () at ../Modules/gcmodule.c:385
#3  collect.lto_priv () at ../Modules/gcmodule.c:925
#4  0x0000555555684b61 in PyGC_Collect () at ../Modules/gcmodule.c:1440
#5  0x0000555555683e23 in Py_Finalize () at ../Python/pythonrun.c:455
#6  0x00005555556a7bf8 in Py_Exit (sts=sts@entry=0)
    at ../Python/pythonrun.c:1792
#7  0x00005555556a5341 in handle_system_exit () at ../Python/pythonrun.c:1160
#8  0x00005555556a4c83 in PyErr_PrintEx () at ../Python/pythonrun.c:1170
#9  0x00005555556a3bfd in RunModule.lto_priv.1251 () at ../Modules/main.c:199
#10 0x0000555555620406 in Py_Main () at ../Modules/main.c:592
#11 0x00007ffff7a05b97 in __libc_start_main (main=0x55555561fef0 <main>, 
    argc=8, argv=0x7fffffffe698, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffe688)
    at ../csu/libc-start.c:310
#12 0x000055555561fe0a in _start ()

So whatever's going wrong seems to be going wrong during garbage collection. Possibly there is some __del__ method doing something nasty?

I also tried running it under rr, but rr replay itself segfaulted! It looks like I need to restart the container with --cap-add=SYS_PTRACE --security-opt seccomp=unconfined, but that would require rebuilding the container and I've run out of time for this right now.

njsmith on 3 May 2018

Of course, if that's the issue then it's mysterious why pip 9 does not have the same problem.

That's not that mysterious, pip 9 explicitly patched requests/urllib3 to ignore pyopenssl etc and only use the standard library, and pip 10 enabled using pyopenssl etc on non-Windows platforms to enable people to get better TLS if possible. If that's causing issues we can probably just go back to disabling anything but stdlib TLS.

dstufft on 3 May 2018

Hello,
I have the same issue with Debian 8 on ARM.

Josue-T on 3 May 2018

BTW, it's not in general true that replacing files that are in use causes problems on Linux. So long as pip replaces files by creating new files and renameing them over the original, and pip isn't importing new extension modules afterwards, then all should be fine. (This is how apt can upgrade your system without requiring a reboot every time.) So there's something more subtle going on than just "oh well this is impossible".

njsmith on 4 May 2018

@njsmith yeah I figured it shouldn't matter (unlinking a file doesn't unmap it from memory and all), so I'm definitely only stating that this occurs when it replaces existing files. I'm not quite sure where to go in debugging this though.

reaperhulk on 4 May 2018

If that's causing issues we can probably just go back to disabling anything but stdlib TLS.

pradyunsg on 4 May 2018

👍1

Okay here's the issue (huge thanks to @geofft for essentially solving this problem solely from being given an incomplete summary):

The easy fix here is to add this to the clobber method:

+               if os.path.exists(destfile):
+                  os.unlink(destfile)
                shutil.copyfile(srcfile, destfile)

This will unlink the file (if it exists) and write a new one. Since the file was mmap'd Python will continue to be able to access the data it needs and exit normally.

reaperhulk on 12 May 2018

👍6

Might not be so rare; also seems to cause pip 9 to segfault when installing simplejson a second time, see https://github.com/simplejson/simplejson/issues/245
Verified by cherrypicking https://github.com/pypa/pip/commit/018f03aab448231462b6c013a65d60a35e06e65a.patch on top of ubuntu 18.04's pip and installing simplejson twice.

dankegel on 23 Mar 2019

FI, this bug is present on pip v9.x (occurs mainly when pip install cffi). The given patch fixes the problem.

axel3rd on 27 Mar 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

lock[bot] on 28 May 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

pip exits with success despite incompatible constraint

cjolowicz · 3Comments

Traceback running setup.py clean: ModuleNotFoundError: No module named 'setuptools'

cjerdonek · 3Comments

Don't deprecate dependency_links or let pip use a requirements.txt file when installing from git

wrouesnel · 3Comments

Pip does not update too-old dependencies when installing multiple packages

Zac-HD · 3Comments

pip fails to install some Python packages...

jiapei100 · 3Comments