Pip: Segfault while installing modules: scikit-learn, hiredis, numpy, redis, scipy, tqdm>=4.29.1

Created on 2 Feb 2019 · 13Comments · Source: pypa/pip

Environment

pip version: pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.5)
Python version: Python 3.5.3
OS: Debian GNU/Linux 9 (stretch)

Description
segfault while installing modules

Expected behavior
no segfault

How to Reproduce
I wrote a Reproducer here:
https://github.com/pippy360/python_reproducer_for_segfault

To reproduce just install the following setup file:

#!/usr/bin/env python
from setuptools import setup, find_packages

setup(
    name='lets cause a segfault',
    packages=find_packages(),
    python_requires='>=3.5',
    install_requires=[
        'scikit-learn',
        'hiredis',
        'numpy',
        'redis',
        'scipy',
        'tqdm>=4.29.1',
    ]
)

Output

user@instance-3:~/python_reproducer_for_segfault$ pip3 install .
Processing /home/user/python_reproducer_for_segfault
Collecting hiredis (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/51/df/d2a08fb767247c0acea66255908e60cdeb4cd13cf71d42a1e2ca5803a1f8/hiredis-1.0.0-cp35-cp35m-manylinux1_x86_64.whl (49kB)
    100% |████████████████████████████████| 51kB 1.7MB/s 
Collecting numpy (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/ad/15/690c13ae714e156491392cdbdbf41b485d23c285aa698239a67f7cfc9e0a/numpy-1.16.1-cp35-cp35m-manylinux1_x86_64.whl (17.2MB)
    100% |████████████████████████████████| 17.2MB 82kB/s 
Collecting redis (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/f1/19/a0282b77c23f9f9dbcc6480787a60807c78a45947593a02dbf026636c90d/redis-3.1.0-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 11.3MB/s 
Collecting scikit-learn (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/18/d9/bea927c86bf78d583d517f24cbc87606cb333bfb3a5c99cb85b547305f0f/scikit_learn-0.20.2-cp35-cp35m-manylinux1_x86_64.whl (5.3MB)
    100% |████████████████████████████████| 5.3MB 279kB/s 
Collecting scipy (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/ab/19/c0ad5b9183ef97030edd6297d1726525ff2c369a09fbb6ea52a1e616ffd6/scipy-1.2.0-cp35-cp35m-manylinux1_x86_64.whl (26.5MB)
    100% |████████████████████████████████| 26.5MB 48kB/s 
Collecting tqdm>=4.29.1 (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/76/4c/103a4d3415dafc1ddfe6a6624333971756e2d3dd8c6dc0f520152855f040/tqdm-4.30.0-py2.py3-none-any.whl (47kB)
    100% |████████████████████████████████| 51kB 10.9MB/s 
Installing collected packages: hiredis, numpy, redis, scipy, scikit-learn, tqdm, lets-cause-a-segfault
  Running setup.py install for lets-cause-a-segfault ... done
Successfully installed hiredis-1.0.0 lets-cause-a-segfault-0.0.0 numpy-1.16.1 redis-3.1.0 scikit-learn-0.20.2 scipy-1.2.0 tqdm-4.30.0
user@instance-3:~/python_reproducer_for_segfault$ ^C
user@instance-3:~/python_reproducer_for_segfault$ pip3 install .
Processing /home/user/python_reproducer_for_segfault
Collecting hiredis (from lets-cause-a-segfault==0.0.0)
  Using cached https://files.pythonhosted.org/packages/51/df/d2a08fb767247c0acea66255908e60cdeb4cd13cf71d42a1e2ca5803a1f8/hiredis-1.0.0-cp35-cp35m-manylinux1_x86_64.whl
Collecting numpy (from lets-cause-a-segfault==0.0.0)
  Using cached https://files.pythonhosted.org/packages/ad/15/690c13ae714e156491392cdbdbf41b485d23c285aa698239a67f7cfc9e0a/numpy-1.16.1-cp35-cp35m-manylinux1_x86_64.whl
Collecting redis (from lets-cause-a-segfault==0.0.0)
  Using cached https://files.pythonhosted.org/packages/f1/19/a0282b77c23f9f9dbcc6480787a60807c78a45947593a02dbf026636c90d/redis-3.1.0-py2.py3-none-any.whl
Collecting scikit-learn (from lets-cause-a-segfault==0.0.0)
  Using cached https://files.pythonhosted.org/packages/18/d9/bea927c86bf78d583d517f24cbc87606cb333bfb3a5c99cb85b547305f0f/scikit_learn-0.20.2-cp35-cp35m-manylinux1_x86_64.whl
Collecting scipy (from lets-cause-a-segfault==0.0.0)
  Using cached https://files.pythonhosted.org/packages/ab/19/c0ad5b9183ef97030edd6297d1726525ff2c369a09fbb6ea52a1e616ffd6/scipy-1.2.0-cp35-cp35m-manylinux1_x86_64.whl
Collecting tqdm>=4.29.1 (from lets-cause-a-segfault==0.0.0)
  Using cached https://files.pythonhosted.org/packages/76/4c/103a4d3415dafc1ddfe6a6624333971756e2d3dd8c6dc0f520152855f040/tqdm-4.30.0-py2.py3-none-any.whl
Installing collected packages: hiredis, numpy, redis, scipy, scikit-learn, tqdm, lets-cause-a-segfault
  Running setup.py install for lets-cause-a-segfault ... done
Successfully installed hiredis-1.0.0 lets-cause-a-segfault-0.0.0 numpy-1.16.1 redis-3.1.0 scikit-learn-0.20.2 scipy-1.2.0 tqdm-4.30.0
Segmentation fault
user@instance-3:~/python_reproducer_for_segfault$

duplicate auto-locked awaiting response bug

Source

pippy360

All 13 comments

Not reproducible with Docker image python:3.5.5-slim-stretch (I couldn’t find a 3.5.3 image, this is the earliest I can get). This might be an issue specific to Debian’s Python build, or maybe CPython fixed it?

Dockfile used:

FROM python:3.5.5-slim-stretch

RUN apt-get update -qq
RUN apt-get -qqy install git >/dev/null
RUN python -m pip install -q -U pip==9.0.1
RUN git clone https://github.com/pippy360/python_reproducer_for_segfault.git

WORKDIR python_reproducer_for_segfault
RUN pip install .

Log:

Sending build context to Docker daemon  2.048kB
Step 1/7 : FROM python:3.5.5-slim-stretch
 ---> e85d66e8ccee
Step 2/7 : RUN apt-get update -qq
 ---> Running in 594bbbe7847c
Removing intermediate container 594bbbe7847c
 ---> b88a47cd080a
Step 3/7 : RUN apt-get -qqy install git >/dev/null
 ---> Running in b47b8d8eeadc
debconf: delaying package configuration, since apt-utils is not installed
Removing intermediate container b47b8d8eeadc
 ---> 3bab98d82f39
Step 4/7 : RUN python -m pip install -q -U pip==9.0.1
 ---> Running in 63be2c38700b
Removing intermediate container 63be2c38700b
 ---> 6edb0d003c37
Step 5/7 : RUN git clone https://github.com/pippy360/python_reproducer_for_segfault.git
 ---> Running in d9c786144aa6
Cloning into 'python_reproducer_for_segfault'...
Removing intermediate container d9c786144aa6
 ---> 3a24cbdde9a9
Step 6/7 : WORKDIR python_reproducer_for_segfault
 ---> Running in 0f2c27834943
Removing intermediate container 0f2c27834943
 ---> 320adb09d298
Step 7/7 : RUN pip install .
 ---> Running in 3f1ff04ee058
Processing /python_reproducer_for_segfault
Collecting scikit-learn (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/18/d9/bea927c86bf78d583d517f24cbc87606cb333bfb3a5c99cb85b547305f0f/scikit_learn-0.20.2-cp35-cp35m-manylinux1_x86_64.whl (5.3MB)
Collecting hiredis (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/51/df/d2a08fb767247c0acea66255908e60cdeb4cd13cf71d42a1e2ca5803a1f8/hiredis-1.0.0-cp35-cp35m-manylinux1_x86_64.whl (49kB)
Collecting numpy (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/ad/15/690c13ae714e156491392cdbdbf41b485d23c285aa698239a67f7cfc9e0a/numpy-1.16.1-cp35-cp35m-manylinux1_x86_64.whl (17.2MB)
Collecting redis (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/f1/19/a0282b77c23f9f9dbcc6480787a60807c78a45947593a02dbf026636c90d/redis-3.1.0-py2.py3-none-any.whl (63kB)
Collecting scipy (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/ab/19/c0ad5b9183ef97030edd6297d1726525ff2c369a09fbb6ea52a1e616ffd6/scipy-1.2.0-cp35-cp35m-manylinux1_x86_64.whl (26.5MB)
Collecting tqdm>=4.29.1 (from lets-cause-a-segfault==0.0.0)
  Downloading https://files.pythonhosted.org/packages/76/4c/103a4d3415dafc1ddfe6a6624333971756e2d3dd8c6dc0f520152855f040/tqdm-4.30.0-py2.py3-none-any.whl (47kB)
Installing collected packages: numpy, scipy, scikit-learn, hiredis, redis, tqdm, lets-cause-a-segfault
  Running setup.py install for lets-cause-a-segfault: started
    Running setup.py install for lets-cause-a-segfault: finished with status 'done'
Successfully installed hiredis-1.0.0 lets-cause-a-segfault-0.0.0 numpy-1.16.1 redis-3.1.0 scikit-learn-0.20.2 scipy-1.2.0 tqdm-4.30.0
You are using pip version 9.0.1, however version 19.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 3f1ff04ee058
 ---> 1a8f86a1925a
Successfully built 1a8f86a1925a

Edit: Cannot reproduce with debian:stretch either.

uranusjr on 7 Feb 2019

you need to "run pip install ." twice for some reason

pippy360 on 7 Feb 2019

I can provide you with a debian vm if you'd like, just contact me on [email protected] and i'll send you the ssh details to a newly deployed vm, if you deploy a vm on google cloud this is default os used, so this could easily hit someone else

pippy360 on 7 Feb 2019

@pippy360, is this still affecting you?

chrahunt on 7 Oct 2019

For completeness, I wasn’t able to reproduce with one additional RUN pip install either. I suspect this is caused by the lock file since the segfault happens after the Successfully installed message 🤔

uranusjr on 7 Oct 2019

👍1

Yes, this is still reproducible.
If you create the fresh instance of the default VM instance on google cloud compute (Debian GNU/Linux 9 (stretch))
and run the following commands:
sudo apt-get install git -y
git clone https://github.com/pippy360/python_reproducer_for_segfault.git
cd python_reproducer_for_segfault/
sudo apt-get install python3-pip -y
pip3 install .
pip3 install .

you still get the segfault. I just tried it 20 seconds ago. If you don't want to spin up a vm instance on google cloud compute then message me at [email protected] and i'll happily provide you with a freshly spun up vm. I don't know what else to say, you can reproduce it using those steps

pippy360 on 8 Oct 2019

Emailed, I'll try to take a look tomorrow.

chrahunt on 8 Oct 2019

I have responded to your email with the ssh details for a vm

pippy360 on 8 Oct 2019

👍1

I was able to reproduce this.

The segfault occurs at Python interpreter shutdown, when trying to clean up the Reader member of the hiredis module - the type associated with this object is null, so attempting to dereference it causes a segfault.

backtrace

(gdb) p $_siginfo
$1 = {si_signo = 11, si_errno = 0, si_code = 1, _sifields = {_pad = {169, 0, -364572592,
      21963, -1818589628, 32764, -1528607782, 32732, 1, 0, 160, 0, -431765888, 21963,
      -367054240, 21963, 44, 0, -436689679, 21963, -367005680, 21963, -367054240, 21963,
      -367005680, 21963, -438187525, 21963}, _kill = {si_pid = 169, si_uid = 0}, _timer = {
      si_tid = 169, si_overrun = 0, si_sigval = {sival_int = -364572592,
        sival_ptr = 0x55cbea451050}}, _rt = {si_pid = 169, si_uid = 0, si_sigval = {
        sival_int = -364572592, sival_ptr = 0x55cbea451050}}, _sigchld = {si_pid = 169,
      si_uid = 0, si_status = -364572592, si_utime = -7810782977104783925,
      si_stime = -6565320432101064708}, _sigfault = {si_addr = 0xa9,
      _addr_lsb = 94334297116752, _addr_bnd = {_lower = 0x7ffc939a8644,
        _upper = 0x7fdca4e34bda}}, _sigpoll = {si_band = 169, si_fd = -364572592}}}
(gdb) bt
#0  0x00005555556aa054 in visit_decref () at ../Modules/gcmodule.c:373
#1  0x00005555557443c5 in dict_traverse.lto_priv () at ../Objects/dictobject.c:2570
#2  0x00005555556ae773 in subtract_refs () at ../Modules/gcmodule.c:398
#3  collect () at ../Modules/gcmodule.c:951
#4  0x00005555557a591d in collect_with_callback () at ../Modules/gcmodule.c:1119
#5  0x00005555557a5981 in PyGC_Collect () at ../Modules/gcmodule.c:1583
#6  0x00005555557ab1ac in Py_Finalize () at ../Python/pylifecycle.c:567
#7  0x00005555557ab2a8 in Py_Exit (sts=sts@entry=0) at ../Python/pylifecycle.c:1465
#8  0x00005555557ab38e in handle_system_exit () at ../Python/pythonrun.c:602
#9  0x00005555557ab3f6 in PyErr_PrintEx () at ../Python/pythonrun.c:612
#10 0x00005555557d8c7a in RunModule () at ../Modules/main.c:210
#11 0x00005555557d952f in Py_Main () at ../Modules/main.c:709
#12 0x0000555555668d71 in main () at ../Programs/python.c:65
#13 0x00007ffff6cee2e1 in __libc_start_main (main=0x555555668c90 <main>, argc=5,
    argv=0x7fffffffe648, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7fffffffe638) at ../csu/libc-start.c:291
#14 0x000055555576fa7a in _start ()

If we look at the op in dict_traverse we see that it's the module dict for hiredis

(gdb) up
#1  0x00005555557443c5 in dict_traverse.lto_priv () at ../Objects/dictobject.c:2570
2570    ../Objects/dictobject.c: No such file or directory.
(gdb) p op
$2 = {'ProtocolError': <type at remote 0x555555d40778>, 'ReplyError': <type at remote 0x555555d40c18>, '__spec__': <ModuleSpec(submodule_search_locations=['/home/chris/.local/lib/python3.5/site-packages/hiredis'], name='hiredis', origin='/home/chris/.local/lib/python3.5/site-packages/hiredis/__init__.py', _set_fileattr=True, _cached='/home/chris/.local/lib/python3.5/site-packages/hiredis/__pycache__/__init__.cpython-35.pyc', loader=<SourceFileLoader(path='/home/chris/.local/lib/python3.5/site-packages/hiredis/__init__.py', name='hiredis') at remote 0x7ffff6aa17b8>, _initializing=False, loader_state=None) at remote 0x7ffff6aa1860>, 'version': <module at remote 0x7ffff6a9de08>, 'hiredis': <module at remote 0x7ffff6a9dc78>, '__doc__': None, '__cached__': '/home/chris/.local/lib/python3.5/site-packages/hiredis/__pycache__/__init__.cpython-35.pyc', '__version__': '1.0.0', '__loader__': <...>, 'HiredisError': <type at remote 0x555555d18448>, '__package__': 'hiredis', '__file__': '/home/chris/.local/lib/python3.5/site-pa...(truncated)

And inspection of the Reader member shows that ob_type is null, but this is dereferenced in visit_decref as part of PyObject_IS_GC, leading to segfault.

(gdb) p mp->ma_keys->dk_entries[31]
$34 = {me_hash = -3403915311998756673, me_key = 'Reader',
  me_value = <unknown at remote 0x7f2a422a4940>}
(gdb) p mp->ma_keys->dk_entries[31].me_value.ob_refcnt
$36 = 1
(gdb) p mp->ma_keys->dk_entries[31].me_value.ob_type
$37 = (struct _typeobject *) 0x0

In pip 9.0.1 (the version installed by python3-pip in Debian 9) we were using CacheControl 0.11.7.

In CacheControl 0.11.7, cachecontrol.caches was importing redis.

redis imports hiredis.

For some reason importing hiredis before the processing we do to install hiredis causes the condition that leads to the segfault mentioned above. This aligns with the observation that the crash happens on the second and subsequent installs.

I've simplified this a bit by doing the following:

Installing hiredis alone - still segfaults
Installing hiredis from a pre-downloaded wheel - still segfaults
Removing the import from the CacheControl wheel and moving it into pip directly - still segfaults
Importing hiredis instead of redis - still segfaults
Moving hiredis.cpython-35m-x86_64-linux-gnu.so to hiredis.so, so that the file that gets written on install is different than the one that was imported - does not segfault

I haven't been able to reproduce this with the latest pip yet, even explicitly importing hiredis in pip itself and running with --user --ignore-installed in order to get similar behavior to 9.0.1.

chrahunt on 9 Oct 2019

👍1

As a workaround you can pip install --upgrade pip which should install under the given user, then do python3 -m pip install .... This should succeed since the latest pip uses a CacheControl that doesn't try to import redis.

chrahunt on 9 Oct 2019

This looks like a duplicate of #5366 which was resolved in #5405 (18.0).

chrahunt on 10 Oct 2019

Thanks so much for the wonderful debugging work @chrahunt!

uranusjr on 12 Oct 2019

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.