Pytest-django: Running many tests with sqlite in-memory backend fails with Segmentation fault
This happen to me a few times since I moved a large test base, of around 450 tests, to py.test (from nose), changing the django backend to either named sqlite db (test.db) or another backend does not seg fail. The failure seems to occur around the 390 test mark; even if I remove the specific tests the fault happened on.
I tried this with py.test 2.9 and latest 3.0.1 or 3.0.3.
tests.py ......
other_tests.py ...... (many more tests, ~390)
Fatal Python error: Segmentation fault
Current thread 0x00007fff73c0e000 (most recent call first):
File "venv/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 316 in execute
File "venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 62 in execute
File "venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 214 in _savepoint_rollback
File "venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254 in savepoint_rollback
File "venv/lib/python2.7/site-packages/django/db/transaction.py", line 245 in __exit__
File "venv/lib/python2.7/site-packages/django/test/testcases.py", line 941 in _rollback_atomics
File "venv/lib/python2.7/site-packages/django/test/testcases.py", line 998 in _fixture_teardown
File "venv/lib/python2.7/site-packages/django/test/testcases.py", line 853 in _post_teardown
File "venv/lib/python2.7/site-packages/django/test/testcases.py", line 189 in __call__
File "venv/lib/python2.7/site-packages/_pytest/unittest.py", line 153 in runtest
File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 104 in pytest_runtest_call
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596 in execute
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264 in __init__
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 247 in _wrapped_call
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595 in execute
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333 in <lambda>
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338 in _hookexec
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724 in __call__
File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 151 in <lambda>
File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 163 in __init__
File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 151 in call_runtest_hook
File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 133 in call_and_report
File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 79 in runtestprotocol
File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 66 in pytest_runtest_protocol
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596 in execute
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264 in __init__
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 247 in _wrapped_call
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595 in execute
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264 in __init__
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 247 in _wrapped_call
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595 in execute
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333 in <lambda>
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338 in _hookexec
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724 in __call__
File "venv/lib/python2.7/site-packages/_pytest/main.py", line 152 in pytest_runtestloop
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596 in execute
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333 in <lambda>
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338 in _hookexec
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724 in __call__
File "venv/lib/python2.7/site-packages/_pytest/main.py", line 131 in _main
File "venv/lib/python2.7/site-packages/_pytest/main.py", line 96 in wrap_session
File "venv/lib/python2.7/site-packages/_pytest/main.py", line 125 in pytest_cmdline_main
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596 in execute
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333 in <lambda>
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338 in _hookexec
File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724 in __call__
File "venv/lib/python2.7/site-packages/_pytest/config.py", line 57 in main
File "venv/bin/py.test", line 11 in <module>
/bin/sh: line 1: 23353 Segmentation fault: 11 py.test
asfaltboy
👍2
All 14 comments
I'm getting a similar error, althought I just started a project and only have one (very) basic test, and am using the spatialite plugin for SQLite. Interestingly, my DATABASES setting specifies 'NAME': ':memory:' but both a test_:memory: and test_:memory:-journal file are created in my root directory after running pytest. @asfaltboy do you experience the same behavior?
$ pytest
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.5.2, pytest-3.0.3, py-1.4.31, pluggy-0.4.0
Django settings: config.settings.test (from ini file)
rootdir: [...], inifile: pytest.ini
plugins: cov-2.4.0, django-3.0.0, faulthandler-1.3.0
collected 1 items
apps/firms/tests/test_models.py Fatal Python error: Segmentation fault
Current thread 0x00007fffce84c3c0 (most recent call first):
File ".../project/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 335 in execute
File ".../project/lib/python3.5/site-packages/django/db/backends/utils.py", line 62 in execute
File ".../project/lib/python3.5/site-packages/django/contrib/gis/db/backends/spatialite/base.py", line 75 in prepare_database
File ".../project/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 81 in handle
File ".../project/lib/python3.5/site-packages/django/core/management/base.py", line 345 in execute
File ".../project/lib/python3.5/site-packages/django/core/management/__init__.py", line 130 in call_command
File ".../project/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 70 in create_test_db
File ".../project/lib/python3.5/site-packages/django/test/runner.py", line 743 in setup_databases
File ".../project/lib/python3.5/site-packages/pytest_django/fixtures.py", line 96 in django_db_setup
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 720 in call_fixture_func
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 815 in pytest_fixture_setup
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 785 in execute
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 529 in _getfixturevalue
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 464 in _get_active_fixturedef
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 766 in execute
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 529 in _getfixturevalue
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 464 in _get_active_fixturedef
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 439 in getfixturevalue
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 447 in getfuncargvalue
File ".../project/lib/python3.5/site-packages/pytest_django/plugin.py", line 375 in _django_db_marker
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 720 in call_fixture_func
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 815 in pytest_fixture_setup
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 785 in execute
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 529 in _getfixturevalue
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 464 in _get_active_fixturedef
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 439 in getfixturevalue
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 397 in _fillfixtures
File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 252 in fillfixtures
File ".../project/lib/python3.5/site-packages/_pytest/python.py", line 1560 in setup
File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 453 in prepare
File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 100 in pytest_runtest_setup
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 151 in <lambda>
File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 163 in __init__
File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 151 in call_runtest_hook
File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 133 in call_and_report
File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 73 in runtestprotocol
File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 66 in pytest_runtest_protocol
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
File ".../project/lib/python3.5/site-packages/_pytest/main.py", line 152 in pytest_runtestloop
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
File ".../project/lib/python3.5/site-packages/_pytest/main.py", line 131 in _main
File ".../project/lib/python3.5/site-packages/_pytest/main.py", line 96 in wrap_session
File ".../project/lib/python3.5/site-packages/_pytest/main.py", line 125 in pytest_cmdline_main
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
File ".../project/lib/python3.5/site-packages/_pytest/config.py", line 56 in main
File ".../project/bin/pytest", line 11 in <module>
[1] 34950 segmentation fault pytest
jmillxyz
on 2 Nov 2016
Update: my tests now pass after installing pytest-ipdb, I'm on v0.1dev2:
pip install git+git://github.com/mverteuil/pytest-ipdb.git
It installs a fair amount of dependencies -- one of which may have fixed the problem -- but I don't have the time at the moment to dig into which one specifically. Here's the list:
Successfully installed appnope-0.1.0 decorator-4.0.10 ipdb-0.10.1 ipython-5.1.0
ipython-genutils-0.1.0 pexpect-4.2.1 pickleshare-0.7.4 prompt-toolkit-1.0.8
ptyprocess-0.5.1 pygments-2.1.3 pytest-ipdb-0.1.dev2 simplegeneric-0.8.1
traitlets-4.3.1 wcwidth-0.1.7
Hopefully related: I have a test suite running on sqlite - it passes during normal operation. However if I add --pdb then I get the segmentation fault. Could it be something to do with how the stack traces are stored which pytest-ipdb fixed?
jamescooke
on 6 Dec 2016
Not sure if it's related, but after further testing I only experienced segfaults or bus errors with SQLite > 3.13.0
jmillxyz
on 6 Dec 2016
I'm on an old sqlite 2.8.17
jamescooke
on 6 Dec 2016
If you are using systemd check out coredumpctl gdb (and then bt when inside gdb) to see if there is some more information where it crashes.
blueyed
on 6 Dec 2016
👍1
There's a bug in sqlite 3.13 or so which causes segmentation faults, and sadly it's the default version in ubuntu. You should try to update sqlite and see if it fixes your issue :
wget https://github.com/ghaering/pysqlite/archive/2.8.3.tar.gz \
&& wget https://www.sqlite.org/2016/sqlite-autoconf-3150200.tar.gz \
&& tar -xzf sqlite-autoconf-3150200.tar.gz \
&& tar -xzvf 2.8.3.tar.gz \
&& cd sqlite-autoconf-3150200 \
&& ./configure \
&& make \
&& make install \
&& cd .. \
&& cp /usr/local/lib/libsqlite3.so.0.8.6 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6 \
&& cp -av sqlite-autoconf-3150200/. pysqlite-2.8.3/ \
&& cd ./pysqlite-2.8.3 && python setup.py build_static install \
&& /opt/venv/bin/python setup.py build_static install
nMustaki
on 27 Jan 2017
👍5
@nMustaki Do you have a link/issue for that bug? I'm curious
jmillxyz
on 27 Jan 2017
This one if I recall corectly but I cannot check right now :
https://www.sqlite.org/src/info/16c9801ceba49
On Fri, 27 Jan 2017, 15:25 Jon Miller, notifications@github.com wrote:
@nMustaki https://github.com/nMustaki Do you have a link/issue for that
bug? I'm curious—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pytest-dev/pytest-django/issues/409#issuecomment-275676636,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA-fUbC400zZ_Tq89EP3tZQlUGTrPMpGks5rWf5mgaJpZM4Kg-gh
.
nMustaki
on 27 Jan 2017
👍1
Nobody provides a stacktrace?! 🤷♂️
Is it still present in the current release?
blueyed
on 28 Jan 2017
Not really a pytest-django bug though - closing.
Feel free to provide a PR if you have some workaround, and it's relevant though.
blueyed
on 28 Jan 2017
Just FYI I'm seeing a very similar backtrace with Django 2.0 and sqlite 3.8.2, Python 3.6.3 on Ubuntu (Travis CI) and not on my devel workstation which is RHEL 7 with an older sqlite3. From what I can tell the segafult happens on an UPDATE query.
I've also dropped pytest and running my tests via manage.py test still reproduces the issue. Any hints would be welcome. Thanks.
atodorov
on 29 Dec 2017
@atodorov you may want to take a look at Django ticket #24080 which seems to indicate this is likely an issue with SQLite rather than with Django.
One commenter points to related ticket on SQLite #7f7f8026eda38 which was fixed in version 3.12.1. I believe I eventually resolved my original issue by upgrading to latest SQLite as well, so I suggest you try that as well.
asfaltboy
on 29 Dec 2017
thanks for the links. I've seen the Django ticket and indeed an upgrade of sqlite solved the issue. However I don't think I've been hitting the same sqlite issue as you linked. Anyway update solves it.
atodorov
on 30 Dec 2017
Was this page helpful?
0 / 5 - 0 ratings
Related issues
WoLpH
·
7Comments
rodrigorodriguescosta
·
4Comments
aljosa
·
8Comments
koxu1996
·
3Comments
asfaltboy
·
5Comments
Most helpful comment
There's a bug in sqlite 3.13 or so which causes segmentation faults, and sadly it's the default version in ubuntu. You should try to update sqlite and see if it fixes your issue :