Hi,
I'm running the python3-pytest package of Ubuntu 18.04, v3.3.2-2, with cov-2.5.1 from a package and faulthandler-1.5.0 from pip. (I'm not using a virtual environment.) The application I am testing makes extensive use of gevent. This has worked well until now but today, when refactoring my tests to move the startup/teardown code to a module, I'm getting seg faults.
I've tried to isolate the issue, and it seems to be something around pytest teardown after the last test in a collection, ie
ie
$ pytest-3 test_app_cancel_upload.py -m crash_when_marked -v
===================================================================== test session starts ======================================================================
platform linux -- Python 3.6.5, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python3
cachedir: ../../.cache
rootdir: /home/mark/dbl-uploader-clients/dbl_dot_local, inifile: pytest.ini
plugins: faulthandler-1.5.0, cov-2.5.1
collected 5 items
test_app_cancel_upload.py::test_cancel_upload_jobs PASSED [100%]
====================================================================== 4 tests deselected ======================================================================
=========================================================== 1 passed, 4 deselected in 10.47 seconds ============================================================
Fatal Python error: Segmentation fault
Current thread 0x00007fe73c7eb740 (most recent call first):
Erreur de segmentation (core dumped)
$ pytest-3 test_app_cancel_upload.py::test_cancel_upload_jobs -v
===================================================================== test session starts ======================================================================
platform linux -- Python 3.6.5, pytest-3.3.2, py-1.5.2, pluggy-0.6.0 -- /usr/bin/python3
cachedir: ../../.cache
rootdir: /home/mark/dbl-uploader-clients/dbl_dot_local, inifile: pytest.ini
plugins: faulthandler-1.5.0, cov-2.5.1
collected 1 item
test_app_cancel_upload.py::test_cancel_upload_jobs PASSED [100%]
=================================================================== 1 passed in 8.83 seconds ===================================================================
$
The test above looks like this:
@pytest.mark.crash_when_marked
@pytest.mark.cancelUploadJobs
def test_cancel_upload_jobs(temp_config_path):
"""
Test cancelUploadJobs for host bundle
"""
config_ob = Config(prepend_real(temp_config_path))
du_app = DownUpApp(config_ob, verbose=True)
# Find an entry to download
entry_id, entry_revision = first_text_entry_ids(du_app.dbl_endpoints)
# task to download entry
tasks_xml = """<tasks>
<downloadMetadata>
<entry>{0}</entry>
<revision>{1}</revision>
<label>downloadedBundle</label>
<tasks>
<createUploadJob/>
</tasks>
</downloadMetadata>
</tasks>""".format(entry_id, entry_revision)
tasks_dom = etree.fromstring(tasks_xml)
first_bundle_id = du_app.storer.new_bundle("text", tasks_dom)
du_app.tick_while_active(60)
jobs = du_app.dbl_endpoints.entry_upload_jobs(entry_id)[3]
assert len(jobs) > 0
second_bundle_id = du_app.storer.label(first_bundle_id, "downloadedBundle")
tasks_xml = """<tasks>
<cancelUploadJobs/>
</tasks>"""
du_app.storer.add_tasks_xml(second_bundle_id, etree.fromstring(tasks_xml))
du_app.tick_while_active(60)
assert not du_app.is_active()
jobs = du_app.dbl_endpoints.entry_upload_jobs(entry_id)[3]
assert len(jobs) == 0
and two of the fixtures look like this:
@pytest.fixture()
def temp_bundle_dir():
"""
Set up a temporary bundle dir with matching settings file, delete it after use.
"""
temp_dir = tempfile.mkdtemp()
if platform.system() == "Windows":
temp_dir = re.sub(
"\\\\",
"/",
temp_dir
)
with open(prepend_test_data_dir("valid_settings.xml.template"), "r") as t_in:
template = t_in.read()
template = re.sub("%%TMP_DIR%%", temp_dir, template)
temp_bundle_dir_path = os.path.join(temp_dir, "bundles")
os.mkdir(temp_bundle_dir_path)
temp_session_bundle_dir_path = os.path.join(temp_dir, "sessions")
os.mkdir(temp_session_bundle_dir_path)
print('built {0}'.format(temp_dir))
yield temp_dir
print('cleanup {0}'.format(temp_dir))
shutil.rmtree(temp_dir)
@pytest.fixture
def temp_config_path(temp_bundle_dir):
temp_dir = temp_bundle_dir
with open(prepend_test_data_dir("valid_settings.xml.template"), "r") as t_in:
template = t_in.read()
template = re.sub("%%TMP_DIR%%", temp_dir, template)
temp_config_path = os.path.join(temp_dir, "valid_settings.xml")
with open(temp_config_path, "w") as t_out:
t_out.write(template)
print('config path: {0}'.format(temp_config_path))
return temp_config_path
This looks to me like a bug in pytest, since the test behaviour varies according to how it is invoked. But I'm quite willing to believe that there's also something evil in my code - I just have no clue how to find it.
All suggestions welcome.
GitMate.io thinks possibly related issues are https://github.com/pytest-dev/pytest/issues/3251 (Warnings not captured during test collection), https://github.com/pytest-dev/pytest/issues/2188 (Test runner hangs during collection with doctest-modules and mock.call in module), https://github.com/pytest-dev/pytest/issues/1521 (High memory usage if tests called by test case name), https://github.com/pytest-dev/pytest/issues/337 (there is no warning when tests have the same name), and https://github.com/pytest-dev/pytest/issues/1337 (Segmentation fault in tests).
Hi @mvahowe,
Unfortunately it is very hard to suggest something at this point because we can't really see all the code. It might be a bug in pytest, but I would actually guess it is some bad interaction with your code.
The only hint I can give is to try to isolate the problem in a minimal reproducible problem; that will help you find the problem in your code or isolate the bug in pytest so we can fix it.
Thanks for this. A colleague of mine got this, from OS X. (Another colleague failed to reproduce the seg fault with Windows):
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 etree.cpython-36m-darwin.so 0x0000000107f53001 __pyx_tp_dealloc_4lxml_5etree__ParserContext + 113
1 etree.cpython-36m-darwin.so 0x0000000107ef1393 __pyx_tp_dealloc_4lxml_5etree__BaseParser + 131
2 org.python.python 0x00000001066adad2 dict_dealloc + 120
3 org.python.python 0x00000001066cd26c subtype_clear + 142
4 org.python.python 0x0000000106750df7 collect + 1830
5 org.python.python 0x0000000106750641 collect_with_callback + 58
6 org.python.python 0x0000000106750694 _PyGC_CollectIfEnabled + 44
7 org.python.python 0x00000001067393bf Py_FinalizeEx + 98
8 org.python.python 0x00000001067399cd Py_Exit + 13
9 org.python.python 0x000000010673c003 handle_system_exit + 317
10 org.python.python 0x000000010673bc54 PyErr_PrintEx + 54
11 org.python.python 0x000000010673b464 PyRun_SimpleFileExFlags + 966
12 org.python.python 0x000000010674ff9a Py_Main + 3466
13 org.python.python 0x000000010666ee1d 0x10666d000 + 7709
14 libdyld.dylib 0x00007fff65b77015 start + 1
From this it looks to us like the issue is something along the lines of double deallocation within the lxml module, which uses the libxml2 library under the hood. libxml2 internals are famously murky. I'll try to come up with a simpler test case but I'm not optimistic.
What I would say is that whatever is happening depends, in a reproducible way, on how the test is called. It doesn't happen if the test is called directly in isolation, it doesn't happen if the file containing the failing test is called with other files, it does occur if all the tests in the file are run, of if the test is called in isolation via a mark decorator. This suggests to me that there's something different about how cleanup occurs. Given that there are lots of persistent gevent threads in our code, could it be something to do with how threads are shut down in different cases?
Also, I don't think sharing all our sprawling codebase here is sensible, but if someone has time to look into the problem we could give them access to our repo. We're getting identical behaviour with Linux and OS X so I'm confident the problem can be reproduced.
This suggests to me that there's something different about how cleanup occurs.
Seems like a good venue for investigation. Keep in mind that pytest itself doesn't know about gevent or attempts to close threads in anyway, so if there's threads being shutdown they are not being shutdown by pytest.
Also, I don't think sharing all our sprawling codebase here is sensible, but if someone has time to look into the problem we could give them access to our repo.
I agree, probably people won't be able to allocate the necessary time to understand the entire code base.
I had segfault too and I think it was due to updating of python 3.6.6 to 3.7.0, because application worked just fine with python 3.6.6.
Same here. Python version has no influence though. Running individual files is OK. Running them together segfaults.
Closing this for now as it seems this is not related to pytest per-se. Will be happy to reopen if more information comes to light. 馃憤
Getting same issue. I was recently running a job in Python and ctrl+c killed it. I don't know if maybe some memory allocation issue arose because of that abrupt termination? Just throwing hypotheses out there.
FWIW this seems to me to be a pytest issue because it happens in the pytest context but not in other contexts. If there are certain types of code that seg fault with pytest but not elsewhere, that's either a bug that should be fixed or a constraint that should be documented.
This code, when running under python works just fine:
from lxml import html
def test_():
pass
if __name__ == "__main__":
test_()
Run this with pytest, and it segfaults.
I can't say this is pytest issue per se, but It does feels like it.
stracing the segafault:
stat("/opt/app/venv3.7.4_try/lib/python3.7/site-packages/lxml", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/opt/app/venv3.7.4_try/lib/python3.7/site-packages/lxml/etree.cpython-37m-x86_64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=6880975, ...}) = 0
open("/opt/app/venv3.7.4_try/lib/python3.7/site-packages/lxml/etree.cpython-37m-x86_64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 9
read(9, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\216\6\0\0\0\0\0"..., 832) = 832
fstat(9, {st_mode=S_IFREG|0755, st_size=6880975, ...}) = 0
mmap(NULL, 7710904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 9, 0) = 0x7f7082b9b000
mprotect(0x7f70830a7000, 2097152, PROT_NONE) = 0
mmap(0x7f70832a7000, 278528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 9, 0x50c000) = 0x7f70832a7000
mmap(0x7f70832eb000, 43192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f70832eb000
close(9) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x2380} ---
+++ killed by SIGSEGV +++
Segmentation fault
Anyone has any insights into that ?
FWIW, my crashing code above made heavy use of lxml.
Thanks.
I have removed all my dependencies one by one, and ion my case, seems like the culprit is https://github.com/syrusakbary/snapshottest , not 100% how and why, but dropping this dependency fixed the issue for me.
I guess for future readers, while pytest might indeed be the issue, it's better to start off with a clean venv, and see if there is any specific dependency that is the issue.
Shouldn't faulthandler provide more inside? (i.e. a Python traceback).
Please also provide Python and pytest versions at least.
@AvnerCohen its possible that your segfault is only hit when certain preconditions are met,
as far as i i can tell, snapshot-test does setup/clean out sys.modules details,
its thinkable that a unexpected gc happens due to loosing/reimporting a module, thus its possible that it needs a blacklist for toplevels not to touch
I'll try to provide all the info I can here, let me know if anything is missing.
Given this basic test file:
from lxml import html
def test_():
pass
if __name__ == "__main__":
test_()
If I run this with pytest, here is the output:
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
rootdir: /opt/app
collected 1 item
moshe.py . [100%]
============================================================================================ 1 passed in 0.02s =============================================================================================
Once I also install snapshottest (pip install snapshottest), the result I see is:
$ pytest moshe.py
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
rootdir: /opt/app
plugins: snapshottest-0.5.1
collecting ... Fatal Python error: Segmentation fault
Current thread 0x00007f933f78f740 (most recent call first):
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1043 in create_module
File "<frozen importlib._bootstrap>", line 583 in module_from_spec
File "<frozen importlib._bootstrap>", line 670 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1035 in _handle_fromlist
File "/opt/app/moshe/lib/python3.7/site-packages/lxml/html/__init__.py", line 53 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 728 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1035 in _handle_fromlist
File "/opt/app/moshe.py", line 1 in <module>
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 142 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "/opt/app/moshe/lib/python3.7/site-packages/py/_path/local.py", line 701 in pyimport
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 501 in _importtestmodule
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 433 in _getobj
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 256 in obj
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 449 in _inject_setup_module_fixture
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 436 in collect
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/runner.py", line 256 in <lambda>
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/runner.py", line 229 in from_call
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/runner.py", line 256 in pytest_make_collect_report
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/runner.py", line 375 in collect_one_node
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 703 in genitems
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 478 in _perform_collect
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 439 in perform_collect
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 246 in pytest_collection
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 236 in _main
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 193 in wrap_session
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 230 in pytest_cmdline_main
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/config/__init__.py", line 90 in main
File "/opt/app/moshe/bin/pytest", line 8 in <module>
Segmentation fault
Happy to provide anything else if this can be of help.
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 142 in exec_module
Please try if it also crashes with --assert=plain.
It does.
$ pytest --assert=plain moshe.py
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.7.4, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
rootdir: /opt/app
plugins: snapshottest-0.5.1
collecting ... Fatal Python error: Segmentation fault
Current thread 0x00007f6e9e135740 (most recent call first):
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1043 in create_module
File "<frozen importlib._bootstrap>", line 583 in module_from_spec
File "<frozen importlib._bootstrap>", line 670 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1035 in _handle_fromlist
File "/opt/app/moshe/lib/python3.7/site-packages/lxml/html/__init__.py", line 53 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 728 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1035 in _handle_fromlist
File "/opt/app/moshe.py", line 1 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 728 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "/opt/app/moshe/lib/python3.7/site-packages/py/_path/local.py", line 701 in pyimport
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 501 in _importtestmodule
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 433 in _getobj
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 256 in obj
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 449 in _inject_setup_module_fixture
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/python.py", line 436 in collect
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/runner.py", line 256 in <lambda>
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/runner.py", line 229 in from_call
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/runner.py", line 256 in pytest_make_collect_report
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/runner.py", line 375 in collect_one_node
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 703 in genitems
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 478 in _perform_collect
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 439 in perform_collect
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 246 in pytest_collection
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 236 in _main
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 193 in wrap_session
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/main.py", line 230 in pytest_cmdline_main
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 86 in <lambda>
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/manager.py", line 92 in _hookexec
File "/opt/app/moshe/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
File "/opt/app/moshe/lib/python3.7/site-packages/_pytest/config/__init__.py", line 90 in main
File "/opt/app/moshe/bin/pytest", line 8 in <module>
Segmentation fault
File "/opt/app/moshe/lib/python3.7/site-packages/py/_path/local.py", line 701 in pyimport
Then try what py does there manually: https://github.com/pytest-dev/py/blob/34f716fe1fb4df9a5257b5f640ebb8a71e10aa88/py/_path/local.py#L689-L701 (to get pytest / your tests out of the equation / try to reproduce it without).
@blueyed Sorry, not sure I follow your exact suggested test here.
The recreation of this is very simple.
pip install -U lxml pytest pip snapshottestfrom lxml import html
def test_():
pass
if __name__ == "__main__":
test_()
ERROR: Could not find a version that satisfies the requirement snapshottest (from versions: none)
Not listed for Python 3.7. (have only later seen that you have it with py37)
With 3.6 I see:
platform linux -- Python 3.6.8, pytest-5.2.2, py-1.8.0, pluggy-0.13.0
rootdir: /tmp
plugins: snapshottest-0.5.1
collected 1 item
t.py . [100%]
Is the if __name__ == "__main__": part relevant for you? Is it triggered through this somehow?
Thanks for the help @blueyed
I've get a repository with a docker container that recreates this, you can see it's pretty straight forward:
https://github.com/AvnerCohen/repro-pytest-segafault
I did notice this did not occur on an alpine-python, but is present on this centos based docker.
Most helpful comment
Same here. Python version has no influence though. Running individual files is OK. Running them together segfaults.