Itk: Segmentation fault in libpthread with TBB backend
Description
I am running and antsRegistration between sets of high resolution (0.3 mm isotropic) multispectral (T1 + T2) sets of scans, consisting of a multi-level Rigid/Similarity/Affine stage and a multi-level SyN stage. (antsMultivariateTemplateConstruction2.sh)
I have compiled ANTs with the built-in ITK, against the stamped version a1a092c0612e00e94518a851ebd259c72e119474 as well as newer commits. I have built ITK_TBB against tbb2019_2019032.
I am experiencing a almost-reproducible Segmentation fault in the very late stages of 53 levels of SyN registration:
antsRegistratio[34817]: segfault at 7f278bfffa08 ip 00007f29279b8b0c sp 00007f2788bf1da8 error 4 in libpthread-2.27.so[7f29279b0000+1a000]
Switching the threading backend to the default ITK thread pool seems to avoid such Segmentation faults
Steps to Reproduce
A minimal working example is difficult to determine as it happens approximately once per level during a modelbuild, but not always with the same files.
Expected behaviour
Don't crash
Actual behaviour
antsRegistratio[34817]: segfault at 7f278bfffa08 ip 00007f29279b8b0c sp 00007f2788bf1da8 error 4 in libpthread-2.27.so[7f29279b0000+1a000]
Reproducibility
1/5 registrations with the input scans as above.
Versions
a1a092c0612e00e94518a851ebd259c72e119474
Environment
Ubuntu 18.04, ECC RAM
Bug
gdevenyi
All 14 comments
I am seeking advice as to how to build the software to enable proper debugging, so I can provide more useful details.
gdevenyi
on 6 May 2019
Have you tried RelWithDebInfo or Debug build types?
When I first added TBB threader, I had to fix some filters because they were making assumptions about the way the image would be split. TBB splits the image into much more chunks by default. Could it be that you are running into the same issue?
dzenanz
on 6 May 2019
I haven't tried any debug settings yet at all, I was looking at what I should to provide useful outputs. I will do this and report back.
gdevenyi
on 6 May 2019
I rebuilt both ANTs and ITK with the Debug Release type, unfortunately, the segfault did not produce any additional output or a core dump.
Seems Ubuntu 18.04 passes core dumps to apport which ignores non-ubuntu software.
Debugging guide, if there is one, should recommend:
> ulimit -c unlimited
> sudo sysctl -w kernel.core_pattern=/tmp/core-%e.%p.%h.%t
Running again, it'll be another day or so to trigger a crash.
gdevenyi
on 13 May 2019
👀1
👍1
Results:
gdb -c /tmp/core-antsRegistratio.12027.monster.1557767459
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
[New LWP 40616]
[New LWP 40610]
[New LWP 12027]
[New LWP 40609]
[New LWP 40613]
[New LWP 40614]
[New LWP 40615]
[New LWP 40612]
[New LWP 40611]
[New LWP 40617]
[New LWP 40639]
[New LWP 40628]
Core was generated by `/home/gdevenyi/software/ANTs-debug-build/bin/antsRegistration -d 3 --float 1 --'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fc48638cb0c in ?? ()
[Current thread is 1 (LWP 40616)]
(gdb) symbol-file ~/software/ANTs-debug-build/bin/antsRegistration
(gdb) sharedlibrary
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.27.so...done.
done.
Reading symbols from /home/gdevenyi/software/tbb2019_20190320oss/lib/intel64/gcc4.7/libtbb_debug.so.2...done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/.build-id/28/c6aade70b2d40d1f0f3d0a1a0cad1ab816448f.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.27.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.27.so...done.
done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.27.so...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.27.so...done.
done.
Reading symbols from /home/gdevenyi/software/tbb2019_20190320oss/lib/intel64/gcc4.7/libtbbmalloc_debug.so.2...done.
(gdb) bt
#0 __pthread_detach (th=140474664851200) at pthread_detach.c:49
#1 0x00007fc4865d1fbc in rml::internal::thread_monitor::detach_thread (handle=140474664851200) at ../../src/tbb/../rml/server/thread_monitor.h:232
#2 0x00007fc4865d2327 in tbb::internal::rml::private_worker::release_handle (handle=140474664851200, join=false) at ../../src/tbb/private_server.cpp:234
#3 0x00007fc4865d229a in tbb::internal::rml::private_worker::start_shutdown (this=0x7fc2f813ec80) at ../../src/tbb/private_server.cpp:253
#4 0x00007fc4865d3c69 in tbb::internal::rml::private_server::request_close_connection (this=0x7fc2fb890980) at ../../src/tbb/private_server.cpp:190
#5 0x00007fc4865dd902 in tbb::internal::market::release (this=0x7fc2fb88e080, is_public=false, blocking_terminate=false) at ../../src/tbb/market.cpp:215
#6 0x00007fc4865e18da in tbb::internal::arena::free_arena (this=0x7fc2f80a5580) at ../../src/tbb/arena.cpp:285
#7 0x00007fc4865dd4f9 in tbb::internal::market::try_destroy_arena (this=0x7fc2fb88e080, a=0x7fc2f80a5580, aba_epoch=0) at ../../src/tbb/market.cpp:348
#8 0x00007fc4865e81b5 in tbb::internal::arena::on_thread_leaving<4096u> (this=0x7fc2f80a5580) at ../../src/tbb/arena.h:390
#9 0x00007fc4865e2fe9 in tbb::internal::arena::process (this=0x7fc2f80a5580, s=...) at ../../src/tbb/arena.cpp:187
#10 0x00007fc4865dd12f in tbb::internal::market::process (this=0x7fc2fb88e080, j=...) at ../../src/tbb/market.cpp:706
#11 0x00007fc4865d2090 in tbb::internal::rml::private_worker::run (this=0x7fc2f813ed80) at ../../src/tbb/private_server.cpp:270
#12 0x00007fc4865d22ea in tbb::internal::rml::private_worker::thread_routine (arg=0x7fc2f813ed80) at ../../src/tbb/private_server.cpp:223
#13 0x00007fc48638b6db in start_thread (arg=0x7fc2c6ffb700) at pthread_create.c:463
#14 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
And stack traces for every thread
(gdb) thread apply all bt full
Thread 12 (Thread 0x7fc2cffff700 (LWP 40628)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
No locals.
#1 0x00007fc48638b45f in __deallocate_stack (pd=0x7fc2cffff700) at allocatestack.c:795
ignore1 = <optimized out>
ignore3 = <optimized out>
ignore1 = <optimized out>
ignore3 = <optimized out>
ignore2 = <optimized out>
ignore2 = <optimized out>
ignore1 = <optimized out>
ignore2 = <optimized out>
ignore3 = <optimized out>
ignore = <optimized out>
#2 __free_tcb (pd=pd@entry=0x7fc2cffff700) at pthread_create.c:367
No locals.
#3 0x00007fc48638b992 in start_thread (arg=0x7fc2cffff700) at pthread_create.c:558
pd = 0x7fc2cffff700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140474690041600, -7945600802100206697, 140474690039744, 0, 140475362437760, 140474685840688, 7943806398612251543, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#4 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
Thread 11 (Thread 0x7fc2ccbf2700 (LWP 40639)):
#0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
No locals.
#1 0x00007fc48638b45f in __deallocate_stack (pd=0x7fc2ccbf2700) at allocatestack.c:795
ignore1 = <optimized out>
ignore3 = <optimized out>
ignore1 = <optimized out>
ignore3 = <optimized out>
ignore2 = <optimized out>
ignore2 = <optimized out>
ignore1 = <optimized out>
ignore2 = <optimized out>
ignore3 = <optimized out>
ignore = <optimized out>
#2 __free_tcb (pd=pd@entry=0x7fc2ccbf2700) at pthread_create.c:367
No locals.
#3 0x00007fc48638b992 in start_thread (arg=0x7fc2ccbf2700) at pthread_create.c:558
pd = 0x7fc2ccbf2700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140474635462400, -7945600802100206697, 140474635460544, 0, 140475362435584, 140475405782320, 7943812447536817047, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#4 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
Thread 10 (LWP 40617):
#0 0x00007fc48556bab7 in munmap () at ../sysdeps/unix/syscall-template.S:78
No locals.
#1 0x00007fc48638b5e5 in __free_stacks (limit=41943040) at allocatestack.c:281
curr = <optimized out>
curr = <optimized out>
entry = <optimized out>
prev = <optimized out>
entry = <optimized out>
prev = <optimized out>
curr = <optimized out>
#2 queue_stack (stack=0x4) at allocatestack.c:305
No locals.
#3 __deallocate_stack (pd=0x4) at allocatestack.c:806
ignore1 = <optimized out>
ignore2 = <optimized out>
ignore3 = <optimized out>
ignore = <optimized out>
#4 __free_tcb (pd=pd@entry=0x7fc2cfbfe700) at pthread_create.c:367
No locals.
#5 0x00007fc48638b992 in start_thread (arg=0x7fc2cfbfe700) at pthread_create.c:558
pd = 0x7fc2cfbfe700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140474685843200, -7945600802100206697, 140474685841344, 0, 140475362438144, 140474639658288, 7943805848319566743, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#6 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
---Type <return> to continue, or q <return> to quit---
No locals.
Thread 9 (Thread 0x7fc2ce3f8700 (LWP 40611)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1 0x00007fc4865b9760 in tbb::internal::futex_wait (futex=0x7fc2f813ef2c, comparand=2) at ../../include/tbb/machine/linux_common.h:85
r = 0
e = 32706
#2 0x00007fc4865b84fc in tbb::internal::binary_semaphore::P (this=0x7fc2f813ef2c) at ../../src/tbb/semaphore.h:209
s = 2
#3 0x00007fc4865d2cd6 in rml::internal::thread_monitor::commit_wait (this=0x7fc2f813ef20, c=...) at ../../src/tbb/../rml/server/thread_monitor.h:259
do_it = true
#4 0x00007fc4865d20ff in tbb::internal::rml::private_worker::run (this=0x7fc2f813ef00) at ../../src/tbb/private_server.cpp:277
c = {my_epoch = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {my_value = 2}}, <No data fields>}, <No data fields>}}
j = @0x7fc2fb817e08: {scratch_ptr = 0x0}
#5 0x00007fc4865d22ea in tbb::internal::rml::private_worker::thread_routine (arg=0x7fc2f813ef00) at ../../src/tbb/private_server.cpp:223
self = 0x7fc2f813ef00
#6 0x00007fc48638b6db in start_thread (arg=0x7fc2ce3f8700) at pthread_create.c:463
pd = 0x7fc2ce3f8700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140474660652800, -7945600802100206697, 140474660650944, 0, 140475362438912, 140475380591920, 7943806953199903639, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#7 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
Thread 8 (Thread 0x7fc2f9a91700 (LWP 40612)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1 0x00007fc4865b9760 in tbb::internal::futex_wait (futex=0x7fc2f813ee2c, comparand=2) at ../../include/tbb/machine/linux_common.h:85
r = 0
e = 32706
#2 0x00007fc4865b84fc in tbb::internal::binary_semaphore::P (this=0x7fc2f813ee2c) at ../../src/tbb/semaphore.h:209
s = 2
#3 0x00007fc4865d2cd6 in rml::internal::thread_monitor::commit_wait (this=0x7fc2f813ee20, c=...) at ../../src/tbb/../rml/server/thread_monitor.h:259
do_it = true
#4 0x00007fc4865d20ff in tbb::internal::rml::private_worker::run (this=0x7fc2f813ee00) at ../../src/tbb/private_server.cpp:277
c = {my_epoch = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {my_value = 1}}, <No data fields>}, <No data fields>}}
j = @0x7fc2fb7b7e08: {scratch_ptr = 0x0}
#5 0x00007fc4865d22ea in tbb::internal::rml::private_worker::thread_routine (arg=0x7fc2f813ee00) at ../../src/tbb/private_server.cpp:223
self = 0x7fc2f813ee00
#6 0x00007fc48638b6db in start_thread (arg=0x7fc2f9a91700) at pthread_create.c:463
pd = 0x7fc2f9a91700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140475388991232, -7945600802100206697, 140475388989376, 0, 140475362438656, 140474660650288, 7943924751804801943, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#7 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
Thread 7 (Thread 0x7fc2f9690700 (LWP 40615)):
#0 0x00007fc485553e57 in sched_yield () at ../sysdeps/unix/syscall-template.S:78
No locals.
#1 0x00007fc4865dd145 in tbb::internal::market::process (this=0x7fc2fb88e080, j=...) at ../../src/tbb/market.cpp:714
i = 0
s = @0x7fc2fb7ebe00: {<tbb::internal::scheduler> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, _vptr.scheduler = 0x7fc48682ed90 <vtable for tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>+16>}, <rml::job> = {
scratch_ptr = 0x0}, <tbb::internal::intrusive_list_node> = {my_prev_node = 0x7fc2fb7ebe10, my_next_node = 0x7fc2fb7ebe10}, <tbb::internal::scheduler_state> = {my_arena_index = 14, my_arena_slot = 0x0, my_arena = 0x7fc2f80a5580, my_innermost_running_task = 0x7fc2fb787f40, my_inbox = {my_putter = 0x0},
my_affinity_id = 15, my_properties = {type = false, outermost = true, static worker = <optimized out>, static master = <optimized out>}, my_last_global_observer = 0x0, my_last_local_observer = 0x0, my_ref_top_priority = 0x7fc2fb88e0b0, my_ref_reload_epoch = 0x7fc2fb88e0c0},
my_stealing_threshold = 140475382695616, my_market = 0x7fc2fb88e080, my_random = {x = 3070651903, c = 3653526687, static a = <optimized out>}, my_free_list = 0x7fc2fb787640, my_dummy_task = 0x7fc2fb787f40, my_ref_count = 1, my_auto_initialized = false, my_task_node_count = 13, my_small_task_count = 14,
my_return_list = 0x7fc2fb787540, _padding1 = '\000' <repeats 111 times>, my_context_list_head = {my_prev = 0x7fc2fb7ebf30, my_next = 0x7fc2fb7ebf30},
my_context_list_mutex = {<tbb::internal::mutex_copy_deprecated_and_disabled> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, <No data fields>}, flag = 0 '\000', static is_rw_mutex = <optimized out>, static is_recursive_mutex = <optimized out>,
static is_fair_mutex = <optimized out>}, my_context_state_propagation_epoch = 0, my_local_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {
my_value = 0}}, <No data fields>}, <No data fields>}, my_offloaded_tasks = 0x0, my_offloaded_task_list_tail_link = 0x0, my_local_reload_epoch = 2, my_pool_reshuffling_pending = false,
my_nonlocal_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {my_value = 0}}, <No data fields>}, <No data fields>}, my_cilk_unwatch_thunk = {routine = 0x0, data = 0x0},
my_cilk_state = tbb::internal::generic_scheduler::cs_none, static quick_task_size = <optimized out>, static null_arena_index = <optimized out>, static min_task_pool_size = <optimized out>}
a = 0x0
#2 0x00007fc4865d2090 in tbb::internal::rml::private_worker::run (this=0x7fc2f813ee80) at ../../src/tbb/private_server.cpp:270
j = @0x7fc2fb7ebe08: {scratch_ptr = 0x0}
#3 0x00007fc4865d22ea in tbb::internal::rml::private_worker::thread_routine (arg=0x7fc2f813ee80) at ../../src/tbb/private_server.cpp:223
self = 0x7fc2f813ee80
#4 0x00007fc48638b6db in start_thread (arg=0x7fc2f9690700) at pthread_create.c:463
pd = 0x7fc2f9690700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140475384792832, -7945600802100206697, 140475384790976, 0, 140475362438784, 140474660650288, 7943924210102051735, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#5 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
---Type <return> to continue, or q <return> to quit---
Thread 6 (Thread 0x7fc2ccff3700 (LWP 40614)):
#0 0x00007fc485553e57 in sched_yield () at ../sysdeps/unix/syscall-template.S:78
No locals.
#1 0x00007fc4865dd145 in tbb::internal::market::process (this=0x7fc2fb88e080, j=...) at ../../src/tbb/market.cpp:714
i = 0
s = @0x7fc2fb77fe00: {<tbb::internal::scheduler> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, _vptr.scheduler = 0x7fc48682ed90 <vtable for tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>+16>}, <rml::job> = {
scratch_ptr = 0x0}, <tbb::internal::intrusive_list_node> = {my_prev_node = 0x7fc2fb77fe10, my_next_node = 0x7fc2fb77fe10}, <tbb::internal::scheduler_state> = {my_arena_index = 4, my_arena_slot = 0x0, my_arena = 0x7fc2f80a5580, my_innermost_running_task = 0x7fc2fb87ff40, my_inbox = {my_putter = 0x0},
my_affinity_id = 5, my_properties = {type = false, outermost = true, static worker = <optimized out>, static master = <optimized out>}, my_last_global_observer = 0x0, my_last_local_observer = 0x0, my_ref_top_priority = 0x7fc2fb88e0b0, my_ref_reload_epoch = 0x7fc2fb88e0c0},
my_stealing_threshold = 140474637563584, my_market = 0x7fc2fb88e080, my_random = {x = 1660765492, c = 2131666079, static a = <optimized out>}, my_free_list = 0x7fc2fb87f240, my_dummy_task = 0x7fc2fb87ff40, my_ref_count = 1, my_auto_initialized = false, my_task_node_count = 17, my_small_task_count = 18,
my_return_list = 0x7fc2fb87f740, _padding1 = '\000' <repeats 111 times>, my_context_list_head = {my_prev = 0x7fc2fb77ff30, my_next = 0x7fc2fb77ff30},
my_context_list_mutex = {<tbb::internal::mutex_copy_deprecated_and_disabled> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, <No data fields>}, flag = 0 '\000', static is_rw_mutex = <optimized out>, static is_recursive_mutex = <optimized out>,
static is_fair_mutex = <optimized out>}, my_context_state_propagation_epoch = 0, my_local_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {
my_value = 0}}, <No data fields>}, <No data fields>}, my_offloaded_tasks = 0x0, my_offloaded_task_list_tail_link = 0x0, my_local_reload_epoch = 2, my_pool_reshuffling_pending = false,
my_nonlocal_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {my_value = 0}}, <No data fields>}, <No data fields>}, my_cilk_unwatch_thunk = {routine = 0x0, data = 0x0},
my_cilk_state = tbb::internal::generic_scheduler::cs_none, static quick_task_size = <optimized out>, static null_arena_index = <optimized out>, static min_task_pool_size = <optimized out>}
a = 0x0
#2 0x00007fc4865d2090 in tbb::internal::rml::private_worker::run (this=0x7fc2f813ed00) at ../../src/tbb/private_server.cpp:270
j = @0x7fc2fb77fe08: {scratch_ptr = 0x0}
#3 0x00007fc4865d22ea in tbb::internal::rml::private_worker::thread_routine (arg=0x7fc2f813ed00) at ../../src/tbb/private_server.cpp:223
self = 0x7fc2f813ed00
#4 0x00007fc48638b6db in start_thread (arg=0x7fc2ccff3700) at pthread_create.c:463
pd = 0x7fc2ccff3700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140474639660800, -7945600802100206697, 140474639658944, 0, 140475362438400, 140475376393520, 7943812997829501847, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#5 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
Thread 5 (Thread 0x7fc2c7bfe700 (LWP 40613)):
#0 0x00007fc485553e57 in sched_yield () at ../sysdeps/unix/syscall-template.S:78
No locals.
#1 0x00007fc4865dd145 in tbb::internal::market::process (this=0x7fc2fb88e080, j=...) at ../../src/tbb/market.cpp:714
i = 0
s = @0x7fc2cc3bbe00: {<tbb::internal::scheduler> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, _vptr.scheduler = 0x7fc48682ed90 <vtable for tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>+16>}, <rml::job> = {
scratch_ptr = 0x0}, <tbb::internal::intrusive_list_node> = {my_prev_node = 0x7fc2cc3bbe10, my_next_node = 0x7fc2cc3bbe10}, <tbb::internal::scheduler_state> = {my_arena_index = 20, my_arena_slot = 0x0, my_arena = 0x7fc2f80a5580, my_innermost_running_task = 0x7fc2f8153f40, my_inbox = {my_putter = 0x0},
my_affinity_id = 21, my_properties = {type = false, outermost = true, static worker = <optimized out>, static master = <optimized out>}, my_last_global_observer = 0x0, my_last_local_observer = 0x0, my_ref_top_priority = 0x7fc2fb88e0b0, my_ref_reload_epoch = 0x7fc2fb88e0c0},
my_stealing_threshold = 140474549528256, my_market = 0x7fc2fb88e080, my_random = {x = 4001641785, c = 3584386207, static a = <optimized out>}, my_free_list = 0x7fc2f8153540, my_dummy_task = 0x7fc2f8153f40, my_ref_count = 1, my_auto_initialized = false, my_task_node_count = 11, my_small_task_count = 12,
my_return_list = 0x7fc2f8153a40, _padding1 = '\000' <repeats 111 times>, my_context_list_head = {my_prev = 0x7fc2cc3bbf30, my_next = 0x7fc2cc3bbf30},
my_context_list_mutex = {<tbb::internal::mutex_copy_deprecated_and_disabled> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, <No data fields>}, flag = 0 '\000', static is_rw_mutex = <optimized out>, static is_recursive_mutex = <optimized out>,
static is_fair_mutex = <optimized out>}, my_context_state_propagation_epoch = 0, my_local_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {
my_value = 0}}, <No data fields>}, <No data fields>}, my_offloaded_tasks = 0x0, my_offloaded_task_list_tail_link = 0x0, my_local_reload_epoch = 2, my_pool_reshuffling_pending = false,
my_nonlocal_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {my_value = 0}}, <No data fields>}, <No data fields>}, my_cilk_unwatch_thunk = {routine = 0x0, data = 0x0},
my_cilk_state = tbb::internal::generic_scheduler::cs_none, static quick_task_size = <optimized out>, static null_arena_index = <optimized out>, static min_task_pool_size = <optimized out>}
a = 0x0
#2 0x00007fc4865d2090 in tbb::internal::rml::private_worker::run (this=0x7fc2f813ef80) at ../../src/tbb/private_server.cpp:270
j = @0x7fc2cc3bbe08: {scratch_ptr = 0x0}
#3 0x00007fc4865d22ea in tbb::internal::rml::private_worker::thread_routine (arg=0x7fc2f813ef80) at ../../src/tbb/private_server.cpp:223
self = 0x7fc2f813ef80
#4 0x00007fc48638b6db in start_thread (arg=0x7fc2c7bfe700) at pthread_create.c:463
pd = 0x7fc2c7bfe700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140474551625472, -7945600802100206697, 140474551623616, 0, 140475362439040, 140475380591920, 7943788256133522327, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#5 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
Thread 4 (Thread 0x7fc2f928f700 (LWP 40609)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
No locals.
#1 0x00007fc4865b9760 in tbb::internal::futex_wait (futex=0x7fc2f813f02c, comparand=2) at ../../include/tbb/machine/linux_common.h:85
r = 0
e = 32706
#2 0x00007fc4865b84fc in tbb::internal::binary_semaphore::P (this=0x7fc2f813f02c) at ../../src/tbb/semaphore.h:209
s = 2
#3 0x00007fc4865d2cd6 in rml::internal::thread_monitor::commit_wait (this=0x7fc2f813f020, c=...) at ../../src/tbb/../rml/server/thread_monitor.h:259
do_it = true
#4 0x00007fc4865d20ff in tbb::internal::rml::private_worker::run (this=0x7fc2f813f000) at ../../src/tbb/private_server.cpp:277
c = {my_epoch = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {my_value = 2}}, <No data fields>}, <No data fields>}}
j = @0x7fc2f8273e08: {scratch_ptr = 0x0}
#5 0x00007fc4865d22ea in tbb::internal::rml::private_worker::thread_routine (arg=0x7fc2f813f000) at ../../src/tbb/private_server.cpp:223
self = 0x7fc2f813f000
#6 0x00007fc48638b6db in start_thread (arg=0x7fc2f928f700) at pthread_create.c:463
pd = 0x7fc2f928f700
now = <optimized out>
---Type <return> to continue, or q <return> to quit---
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140475380594432, -7945600802100206697, 140475380592576, 0, 140475362439168, 140730269108688, 7943923659809366935, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#7 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
Thread 3 (Thread 0x7fc486c32f00 (LWP 12027)):
#0 0x00005586a74ec101 in ?? ()
No symbol table info available.
#1 0x00007ffe51b319e0 in ?? ()
No symbol table info available.
#2 0x00005586a74c016e in ?? ()
No symbol table info available.
#3 0x000000000005b437 in ?? ()
No symbol table info available.
#4 0x00007ffe51b31b20 in ?? ()
No symbol table info available.
#5 0x000000000005b437 in ?? ()
No symbol table info available.
#6 0x00005586ac506390 in ?? ()
No symbol table info available.
#7 0x00007ffe51b31a10 in ?? ()
No symbol table info available.
#8 0x00005586a746cf30 in ?? ()
No symbol table info available.
#9 0x00007ffe51b31b20 in ?? ()
No symbol table info available.
#10 0x00005586adefeed0 in ?? ()
No symbol table info available.
#11 0x00005586aa6b7e08 in ?? ()
No symbol table info available.
#12 0x000000000005b437 in ?? ()
No symbol table info available.
#13 0x00007ffe51b31cb0 in ?? ()
No symbol table info available.
#14 0x00005586a75b55ea in ?? ()
No symbol table info available.
#15 0x00007ffe51b31a40 in ?? ()
No symbol table info available.
#16 0x00007ffe51b31e80 in ?? ()
No symbol table info available.
#17 0x00007ffe51b31e00 in ?? ()
No symbol table info available.
#18 0x00007ffe51b31de0 in ?? ()
No symbol table info available.
#19 0x00005586ad7316e0 in ?? ()
No symbol table info available.
#20 0x00007ffe51b31dc0 in ?? ()
No symbol table info available.
#21 0x00007ffe51b31da0 in ?? ()
No symbol table info available.
#22 0x00005586aa352fd0 in ?? ()
No symbol table info available.
#23 0x00007ffe51b31d80 in ?? ()
No symbol table info available.
#24 0x00007ffe51b31d60 in ?? ()
No symbol table info available.
#25 0x00005586aa347ba0 in ?? ()
No symbol table info available.
#26 0x00007ffe51b31d58 in ?? ()
No symbol table info available.
#27 0x0000000000000000 in ?? ()
No symbol table info available.
Thread 2 (Thread 0x7fc2f8e8e700 (LWP 40610)):
#0 0x00007fc485553e57 in sched_yield () at ../sysdeps/unix/syscall-template.S:78
No locals.
#1 0x00007fc4865dd145 in tbb::internal::market::process (this=0x7fc2fb88e080, j=...) at ../../src/tbb/market.cpp:714
i = 0
s = @0x7fc2cc373e00: {<tbb::internal::scheduler> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, _vptr.scheduler = 0x7fc48682ed90 <vtable for tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>+16>}, <rml::job> = {
scratch_ptr = 0x0}, <tbb::internal::intrusive_list_node> = {my_prev_node = 0x7fc2cc373e10, my_next_node = 0x7fc2cc373e10}, <tbb::internal::scheduler_state> = {my_arena_index = 3, my_arena_slot = 0x0, my_arena = 0x7fc2f80a5580, my_innermost_running_task = 0x7fc2f8173f40, my_inbox = {my_putter = 0x0},
my_affinity_id = 4, my_properties = {type = false, outermost = true, static worker = <optimized out>, static master = <optimized out>}, my_last_global_observer = 0x0, my_last_local_observer = 0x0, my_ref_top_priority = 0x7fc2fb88e0b0, my_ref_reload_epoch = 0x7fc2fb88e0c0},
my_stealing_threshold = 140475374298816, my_market = 0x7fc2fb88e080, my_random = {x = 4161409950, c = 1138156703, static a = <optimized out>}, my_free_list = 0x7fc2f8173b40, my_dummy_task = 0x7fc2f8173f40, my_ref_count = 1, my_auto_initialized = false, my_task_node_count = 11, my_small_task_count = 12,
my_return_list = 0x7fc2f8173640, _padding1 = '\000' <repeats 111 times>, my_context_list_head = {my_prev = 0x7fc2cc373f30, my_next = 0x7fc2cc373f30},
my_context_list_mutex = {<tbb::internal::mutex_copy_deprecated_and_disabled> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, <No data fields>}, flag = 0 '\000', static is_rw_mutex = <optimized out>, static is_recursive_mutex = <optimized out>,
static is_fair_mutex = <optimized out>}, my_context_state_propagation_epoch = 0, my_local_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {
---Type <return> to continue, or q <return> to quit---
my_value = 0}}, <No data fields>}, <No data fields>}, my_offloaded_tasks = 0x0, my_offloaded_task_list_tail_link = 0x0, my_local_reload_epoch = 2, my_pool_reshuffling_pending = false,
my_nonlocal_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {my_value = 0}}, <No data fields>}, <No data fields>}, my_cilk_unwatch_thunk = {routine = 0x0, data = 0x0},
my_cilk_state = tbb::internal::generic_scheduler::cs_none, static quick_task_size = <optimized out>, static null_arena_index = <optimized out>, static min_task_pool_size = <optimized out>}
a = 0x0
#2 0x00007fc4865d2090 in tbb::internal::rml::private_worker::run (this=0x7fc2f813f080) at ../../src/tbb/private_server.cpp:270
j = @0x7fc2cc373e08: {scratch_ptr = 0x0}
#3 0x00007fc4865d22ea in tbb::internal::rml::private_worker::thread_routine (arg=0x7fc2f813f080) at ../../src/tbb/private_server.cpp:223
self = 0x7fc2f813f080
#4 0x00007fc48638b6db in start_thread (arg=0x7fc2f8e8e700) at pthread_create.c:463
pd = 0x7fc2f8e8e700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140475376396032, -7945600802100206697, 140475376394176, 0, 140475362439296, 140730269108688, 7943927507563193239, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#5 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
Thread 1 (Thread 0x7fc2c6ffb700 (LWP 40616)):
#0 __pthread_detach (th=140474664851200) at pthread_detach.c:49
pd = 0x7fc2ce7f9700
result = 0
#1 0x00007fc4865d1fbc in rml::internal::thread_monitor::detach_thread (handle=140474664851200) at ../../src/tbb/../rml/server/thread_monitor.h:232
No locals.
#2 0x00007fc4865d2327 in tbb::internal::rml::private_worker::release_handle (handle=140474664851200, join=false) at ../../src/tbb/private_server.cpp:234
No locals.
#3 0x00007fc4865d229a in tbb::internal::rml::private_worker::start_shutdown (this=0x7fc2f813ec80) at ../../src/tbb/private_server.cpp:253
s = tbb::internal::rml::private_worker::st_normal
#4 0x00007fc4865d3c69 in tbb::internal::rml::private_server::request_close_connection (this=0x7fc2fb890980) at ../../src/tbb/private_server.cpp:190
i = 247
#5 0x00007fc4865dd902 in tbb::internal::market::release (this=0x7fc2fb88e080, is_public=false, blocking_terminate=false) at ../../src/tbb/market.cpp:215
do_release = true
#6 0x00007fc4865e18da in tbb::internal::arena::free_arena (this=0x7fc2f80a5580) at ../../src/tbb/arena.cpp:285
drained = 0
storage = 0xfffffffffffffffc
#7 0x00007fc4865dd4f9 in tbb::internal::market::try_destroy_arena (this=0x7fc2fb88e080, a=0x7fc2f80a5580, aba_epoch=0) at ../../src/tbb/market.cpp:348
pl = @0x7fc2fb88e0f0: {arenas = {<tbb::internal::intrusive_list_base<tbb::internal::intrusive_list<tbb::internal::arena>, tbb::internal::arena>> = {my_head = {my_prev_node = 0x7fc2fb88e0f0, my_next_node = 0x7fc2fb88e0f0}, my_size = 0}, <No data fields>}, next_arena = 0x7fc2fb88e0f0, workers_requested = 0,
workers_available = 39}
my_arenas = @0x7fc2fb88e0f0: {<tbb::internal::intrusive_list_base<tbb::internal::intrusive_list<tbb::internal::arena>, tbb::internal::arena>> = {my_head = {my_prev_node = 0x7fc2fb88e0f0, my_next_node = 0x7fc2fb88e0f0}, my_size = 0}, <No data fields>}
it = {<tbb::internal::intrusive_list_base<tbb::internal::intrusive_list<tbb::internal::arena>, tbb::internal::arena>::iterator_impl<tbb::internal::intrusive_list_base<tbb::internal::intrusive_list<tbb::internal::arena>, tbb::internal::arena>::iterator>> = {my_pos = 0x7fc2f80a5580}, <No data fields>}
p = 1
locked = false
#8 0x00007fc4865e81b5 in tbb::internal::arena::on_thread_leaving<4096u> (this=0x7fc2f80a5580) at ../../src/tbb/arena.h:390
aba_epoch = 0
m = 0x7fc2fb88e080
#9 0x00007fc4865e2fe9 in tbb::internal::arena::process (this=0x7fc2f80a5580, s=...) at ../../src/tbb/arena.cpp:187
index = 5
#10 0x00007fc4865dd12f in tbb::internal::market::process (this=0x7fc2fb88e080, j=...) at ../../src/tbb/market.cpp:706
i = 0
s = @0x7fc2fb7a7e00: {<tbb::internal::scheduler> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, _vptr.scheduler = 0x7fc48682ed90 <vtable for tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>+16>}, <rml::job> = {
scratch_ptr = 0x0}, <tbb::internal::intrusive_list_node> = {my_prev_node = 0x7fc2fb7a7e10, my_next_node = 0x7fc2fb7a7e10}, <tbb::internal::scheduler_state> = {my_arena_index = 5, my_arena_slot = 0x0, my_arena = 0x7fc2f80a5580, my_innermost_running_task = 0x7fc2fb7bff40, my_inbox = {my_putter = 0x0},
my_affinity_id = 6, my_properties = {type = false, outermost = true, static worker = <optimized out>, static master = <optimized out>}, my_last_global_observer = 0x0, my_last_local_observer = 0x0, my_ref_top_priority = 0x7fc2fb88e0b0, my_ref_reload_epoch = 0x7fc2fb88e0c0},
my_stealing_threshold = 140474536933056, my_market = 0x7fc2fb88e080, my_random = {x = 3233890966, c = 150152351, static a = <optimized out>}, my_free_list = 0x7fc2fb7bf540, my_dummy_task = 0x7fc2fb7bff40, my_ref_count = 1, my_auto_initialized = false, my_task_node_count = 17, my_small_task_count = 18,
my_return_list = 0x7fc2fb7bef40, _padding1 = '\000' <repeats 111 times>, my_context_list_head = {my_prev = 0x7fc2fb7a7f30, my_next = 0x7fc2fb7a7f30},
my_context_list_mutex = {<tbb::internal::mutex_copy_deprecated_and_disabled> = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, <No data fields>}, flag = 0 '\000', static is_rw_mutex = <optimized out>, static is_recursive_mutex = <optimized out>,
static is_fair_mutex = <optimized out>}, my_context_state_propagation_epoch = 0, my_local_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {
my_value = 0}}, <No data fields>}, <No data fields>}, my_offloaded_tasks = 0x0, my_offloaded_task_list_tail_link = 0x0, my_local_reload_epoch = 2, my_pool_reshuffling_pending = false,
my_nonlocal_ctx_list_update = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {my_value = 0}}, <No data fields>}, <No data fields>}, my_cilk_unwatch_thunk = {routine = 0x0, data = 0x0},
my_cilk_state = tbb::internal::generic_scheduler::cs_none, static quick_task_size = <optimized out>, static null_arena_index = <optimized out>, static min_task_pool_size = <optimized out>}
a = 0x7fc2f80a5580
#11 0x00007fc4865d2090 in tbb::internal::rml::private_worker::run (this=0x7fc2f813ed80) at ../../src/tbb/private_server.cpp:270
j = @0x7fc2fb7a7e08: {scratch_ptr = 0x0}
#12 0x00007fc4865d22ea in tbb::internal::rml::private_worker::thread_routine (arg=0x7fc2f813ed80) at ../../src/tbb/private_server.cpp:223
self = 0x7fc2f813ed80
#13 0x00007fc48638b6db in start_thread (arg=0x7fc2c6ffb700) at pthread_create.c:463
pd = 0x7fc2c6ffb700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140474539030272, -7945600802100206697, 140474539028416, 0, 140475362438528, 140475376393520, 7943791003301979031, 7940552417853449111}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#14 0x00007fc48557188f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
Was TBB compiled from source?
thewtex
on 14 May 2019
Latest binary release because their build code doesn't make the cmake files needed for itk, see my bug about the TBB module
gdevenyi
on 14 May 2019
Yes, their build process is super inconvenient for CMake users.
Now we know that the crash occurred in rml::internal::thread_monitor::detach_thread at thread_monitor.h:L232. Unfortunately, that does not ring any bells for me.
dzenanz
on 14 May 2019
Testing again with tbb2019_20190410oss_lin.tgz
I'd like to build it myself so that everything is equal, if anyone can offer any guidance as to how to get a working cmake file when doing so, I'd appreciate it.
gdevenyi
on 15 May 2019
@gdevenyi I'm perpetually disappointed with the TBB teams reluctance to follow common software distribution standards. I've worked with the team several times and slowly made improvements.
https://github.com/BRAINSia/BRAINSTools/blob/master/SuperBuild/External_TBB.cmake
https://github.com/BRAINSia/BRAINSTools/blob/master/SuperBuild/build_tbb.sh
Hans
hjmjohnson
on 15 May 2019
Followup, my tests have been running with 2019_U6 and so far no crash, so I'm leaning towards a 2019_U5 bug...
gdevenyi
on 17 May 2019
👍2
@gdevenyi thanks for the follow-up. I suspected it stems from the binary.
thewtex
on 18 May 2019
Followup again, running Release again seems the bug is back.
Going to
- build my own TBB
- use RelWithDebInfo
gdevenyi
on 24 May 2019
I'm still running into this with 2019_U9 with icc (ICC) 19.0.4.243 20190416 built from source (TBB from source) and antsRegistration https://github.com/ANTsX/ANTs/commit/b8889562341471dbe18b415d2bcefb9b8f12232a
gdevenyi
on 8 Nov 2019
Was this page helpful?
1 / 5 - 1 ratings
Related issues
fbudin69500
路
5Comments
thewtex
路
6Comments
blowekamp
路
4Comments
dzenanz
路
6Comments
seanm
路
5Comments
Most helpful comment
Followup, my tests have been running with 2019_U6 and so far no crash, so I'm leaning towards a 2019_U5 bug...