Cataclysm-dda: SIGSEGV processing items on map

Created on 15 Dec 2019 · 15Comments · Source: CleverRaven/Cataclysm-DDA

Describe the bug

Segmentation fault processing some item. I can't determine the specific cause and the reproduction case is not consistent.

Steps To Reproduce

Steps to reproduce the behavior in the attached save:

Continue riding south down the road past the map note then turn east for a few OMTs

Expected behavior

No segfaults.

Versions and configuration

OS: Arch Linux x86_64 kernel 5.4.3-arch1-1
Game Version: RELEASE=1 bindist built from e667aa09a3eccf7464e909c1e3f193fe2188449f with #36096 and #35865 applied
Graphics version: Tiles
Mods loaded: [ dda, disable npc needs, no rail stations, no fictional guns, no flaming weapons ]

Additional context

The program has crashed.
See the log file for a stack trace.
CRASH LOG FILE: ./config/crash.log
VERSION: 0.C-47852-g8603027fa9
TYPE: Signal
MESSAGE: SIGSEGV: Segmentation faultError creating SDL message box: No message system available

STACK TRACE:

    ./cataclysm-tiles(_Z21debug_write_backtraceRSo+0x39) [0x557e6a3c8939]
    ./cataclysm-tiles(+0x745976) [0x557e6a3a3976]
    ./cataclysm-tiles(+0x7455d8) [0x557e6a3a35d8]
    /usr/lib/libc.so.6(+0x3bfb0) [0x7fe9c6483fb0]
    /usr/lib/libstdc++.so.6(_ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base+0x13) [0x7fe9c684ea93]
    ./cataclysm-tiles(_ZN3map13process_itemsEbPFbR10item_stackR14safe_referenceI4itemERK8tripointRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEf16temperature_flagESG_+0x3a7) [0x557e6a9fda07]
    ./cataclysm-tiles(_ZN4game7do_turnEv+0x1467) [0x557e6a56bc27]
    ./cataclysm-tiles(main+0x1a0c) [0x557e6a9b83dc]
    /usr/lib/libc.so.6(__libc_start_main+0xf3) [0x7fe9c646f153]
    ./cataclysm-tiles(_start+0x2e) [0x557e6a01cdae]

    Attempting to repeat stack trace using debug symbols…
    debug_write_backtrace(std::ostream&)
    ??:?
    log_crash(char const*, char const*)
    ld-temp.o:?
    signal_handler(int)
    ld-temp.o:?
    ??
    ??:0
    local_Rb_tree_increment
    …/src/c++98/tree.cc:65
    std::_Rb_tree_increment(std::_Rb_tree_node_base*)
    …/src/c++98/tree.cc:85
    map::process_items(bool, bool (*)(item_stack&, safe_reference<item>&, tripoint const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, float, temperature_flag), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    ??:?
    game::do_turn()
    ??:?
    main
    ??:?
    __libc_start_main
    ??:?
    _start
    ??:?

(P1 - Critical) (S2 - Confirmed) <Bug> <Crash / Freeze>

Source

ymber

Most helpful comment

Thanks for the reliable repro case, @HarleyLorenzo . Using that, git-bisect shows the problematic commit to be @kevingranade 's 44a91c2acab89187de2243ce99b2057ac1903206 on PR #36025. Reverting this commit causes master to not crash on your repro case.

That commit doesn't appear to be particularly-important, so I'll put up a PR to revert this commit to get things working again and let @kevingranade decide how best to fix that patch.

mark7 on 18 Dec 2019

❤2

All 15 comments

Can't see a save? Been getting a similar crash log with an error like this one:

 DEBUG    : Tried to access invalid map position at grid (196589,458644,131077)

 FUNCTION : size_t map::get_nonant(const tripoint &) const
 FILE     : src/map.cpp
 LINE     : 7834

molkero on 15 Dec 2019

Yeah that's the debug message I get. Save attached.
East Northport.zip

ymber on 16 Dec 2019

Gotten similar crashes, here is my stacktrace for comparison:

The program has crashed.
See the log file for a stack trace.
CRASH LOG FILE: ./config/crash.log
VERSION: 0.D-10248-g4be222a
TYPE: Signal
MESSAGE: SIGSEGV: Segmentation fault
STACK TRACE:

./cataclysm-tiles(_Z21debug_write_backtraceRSo+0x38) [0xa100b9]
./cataclysm-tiles() [0xa008be]
./cataclysm-tiles() [0xa00a40]
/usr/lib/libc.so.6(+0x3bfb0) [0x7f8e6bfdefb0]
/usr/lib/libstdc++.so.6(_ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base+0x13) [0x7f8e6c387a93]
./cataclysm-tiles(_ZNSt23_Rb_tree_const_iteratorI8tripointEppEv+0xc) [0xc827bc]
./cataclysm-tiles(_ZN3map13process_itemsEbPFbR10item_stackR14safe_referenceI4itemERK8tripointRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEf16temperature_flagESG_+0x1e7) [0xca1859]
./cataclysm-tiles(_ZN3map20process_active_itemsEv+0x41) [0xca18ef]
./cataclysm-tiles(_ZN4game7do_turnEv+0x8c6) [0xad3c3a]
./cataclysm-tiles(main+0x12b3) [0x7ef2b9]
/usr/lib/libc.so.6(__libc_start_main+0xf3) [0x7f8e6bfca153]
./cataclysm-tiles(_start+0x29) [0x871fd9]

Attempting to repeat stack trace using debug symbols…
debug_write_backtrace(std::ostream&)
??:?
log_crash
crash.cpp:?
signal_handler
crash.cpp:?
??
??:0
local_Rb_tree_increment
…/src/c++98/tree.cc:65
std::_Rb_tree_increment(std::_Rb_tree_node_base*)
…/src/c++98/tree.cc:85
std::_Rb_tree_const_iterator<tripoint>::operator++()
??:?
map::process_items(bool, bool (*)(item_stack&, safe_reference<item>&, tripoint const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, float, temperature_flag), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
??:?
map::process_active_items()
??:?
game::do_turn()
??:?
main
??:?
__libc_start_main
??:?
_start
??:?

freshcapybara on 16 Dec 2019

And here is my save. To reproduce just go a couple of tiles south.
save.tar.gz

freshcapybara on 16 Dec 2019

On OpenBSD, I'm getting SIGBUS errors (usually indicates a use after free) when processing large batches of items (this log is from 20 units of cooked meat, crashes consistently at 62% complete)

Build is b10009 (last build I used that didn't have this problem was b9919)

GNU gdb (GDB) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-openbsd6.6".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/cataclysm-tiles...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/local/bin/cataclysm-tiles 
[New thread 229145]
[New thread 554565]
[New thread 599120]
[New thread 244176]
[New thread 207990]
[New thread 119053]
[New thread 148619]
[New thread 343007]
[New thread 566387]
[New thread 305173]

Thread 1 received signal SIGBUS, Bus error.
0x00000e735ef05493 in map::process_items(bool, bool (*)(item_stack&, safe_reference<item>&, tripoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, float, temperature_flag), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
(gdb) bt
#0  0x00000e735ef05493 in map::process_items(bool, bool (*)(item_stack&, safe_reference<item>&, tripoint const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, float, temperature_flag), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
#1  0x00000e735ef0519b in map::process_active_items() ()
#2  0x00000e735eb09283 in game::do_turn() ()
#3  0x00000e735eed06e9 in main ()

I know this isn't exactly the issue in question, but the bug is extremely similar, so I'm leaving it here for now

fossforus on 17 Dec 2019

I've got a consistent reproduction case.
East Northport.zip
Go to the last road OMT next to the tracked canoe east on the river bank. Walk around a few tiles and it reliably segfaults.

I built with debug symbols and got a more useful stack trace.

23:16:07.394 : Starting log.
23:16:07.394 INFO : Cataclysm DDA version 0.C-47852-g8603027fa9-dirty
23:16:07.423 INFO : Number of render drivers on your system: 3
23:16:07.423 INFO : Render driver: 0/opengl
23:16:07.423 INFO : Render driver: 1/opengles2
23:16:07.423 INFO : Render driver: 2/software
23:16:07.760 INFO : SDL version used during compile is 2.0.10
23:16:07.760 INFO : SDL version used during linking and in runtime is 2.0.10
23:16:07.930 INFO : Active renderer: 2/software
23:16:07.972 INFO : USE_COLOR_MODULATED_TEXTURES is set to 0
23:16:07.997 WARNING : opendir [./mods/] failed with "No such file or directory".
23:16:13.696 WARNING : opendir [./save/East Northport/mods] failed with "No such file or directory".
23:17:03.213 ERROR : (error message will follow backtrace)
    ./cataclysm-tiles(_Z21debug_write_backtraceRSo+0x39) [0x56460a502bd9]
    ./cataclysm-tiles(_Z8DebugLog10DebugLevel10DebugClass+0x11e) [0x56460a5025de]
    ./cataclysm-tiles(_Z12realDebugmsgPKcS0_S0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x5c) [0x56460a50212c]
    ./cataclysm-tiles(_Z12realDebugmsgIJRKiS1_S1_EEvPKcS3_S3_S3_DpOT_+0x7c) [0x56460a4f064c]
    ./cataclysm-tiles(_ZNK3map10get_nonantERK8tripoint+0x57) [0x56460a9de987]
    ./cataclysm-tiles(_ZNK3map18get_submap_at_gridERK8tripoint+0x9) [0x56460a9c1e39]
    ./cataclysm-tiles(_ZN3map13process_itemsEbPFbR10item_stackR14safe_referenceI4itemERK8tripointRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEf16temperature_flagESG_+0x2af) [0x56460a9d56ff]
    ./cataclysm-tiles(_ZN3map20process_active_itemsEv+0x3a) [0x56460a9d540a]
    ./cataclysm-tiles(_ZN4game7do_turnEv+0x92b) [0x56460a64a89b]
    ./cataclysm-tiles(main+0xbe0) [0x56460a9b0720]
    /usr/lib/libc.so.6(__libc_start_main+0xf3) [0x7f531131d153]
    ./cataclysm-tiles(_start+0x2e) [0x56460a1ad7ce]

    Attempting to repeat stack trace using debug symbols…
    debug_write_backtrace(std::ostream&)
    …/src/debug.cpp:620
    DebugLog(DebugLevel, DebugClass)
    …/src/debug.cpp:786
    realDebugmsg(char const*, char const*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    …/src/debug.cpp:107
    void realDebugmsg<int const&, int const&, int const&>(char const*, char const*, char const*, char const*, int const&, int const&, int const&)
    …/src/debug.h:?
    map::get_nonant(tripoint const&) const
    …/src/map.cpp:?
    map::get_submap_at_grid(tripoint const&) const
    …/src/map.cpp:7826
    map::process_items(bool, bool (*)(item_stack&, safe_reference<item>&, tripoint const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, float, temperature_flag), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    …/src/map.cpp:4385
    map::process_active_items()
    ??:?
    game::do_turn()
    …/src/game.cpp:1523
    main
    …/src/main.cpp:688
    __libc_start_main
    ??:?
    _start
    ??:?
Backtrace emission took 12 seconds.
src/map.cpp:7834 [size_t map::get_nonant(const tripoint &) const] Tried to access invalid map position at grid (-329,65335,65538)

ymber on 17 Dec 2019

I've got a reproducible too: https://drive.google.com/file/d/1aIa1YQJVyeUNoJ1bOdsfewkSWM6uL7ms/view?usp=sharing

Load game, hit shit+r to read, choose 6th book Through the Lens, then Read until you gain a level, then (I)gnore further distractions and finish twice, or thrice:

The program has crashed.
See the log file for a stack trace.
CRASH LOG FILE: ./config/crash.log
VERSION: 0.D-10214-ge667aa09a3
TYPE: Signal
MESSAGE: SIGSEGV: Segmentation fault
STACK TRACE:

    ./cataclysm-tiles(_Z21debug_write_backtraceRSo+0x39) [0x55c91eb58e59]
    ./cataclysm-tiles(+0x743ea6) [0x55c91eb33ea6]
    ./cataclysm-tiles(+0x743b08) [0x55c91eb33b08]
    /usr/lib/libc.so.6(+0x3bfb0) [0x7ff59d0cffb0]
    /usr/lib/libstdc++.so.6(_ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base+0x13) [0x7ff59d478a93]
    ./cataclysm-tiles(_ZN3map13process_itemsEbPFbR10item_stackR14safe_referenceI4itemERK8tripointRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEf16temperature_flagESG_+0x3a7) [0x55c91f18e317]
    ./cataclysm-tiles(_ZN4game7do_turnEv+0x1497) [0x55c91ecfc3f7]
    ./cataclysm-tiles(main+0x1a0c) [0x55c91f148c6c]
    /usr/lib/libc.so.6(__libc_start_main+0xf3) [0x7ff59d0bb153]
    ./cataclysm-tiles(_start+0x2e) [0x55c91e7b012e]

    Attempting to repeat stack trace using debug symbols…
    debug_write_backtrace(std::ostream&)
    ??:?
    log_crash(char const*, char const*)
    ld-temp.o:?
    signal_handler(int)
    ld-temp.o:?
    ??
    ??:0
    local_Rb_tree_increment
    …/src/c++98/tree.cc:65
    std::_Rb_tree_increment(std::_Rb_tree_node_base*)
    …/src/c++98/tree.cc:85
    map::process_items(bool, bool (*)(item_stack&, safe_reference<item>&, tripoint const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, float, temperature_flag), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    ??:?
    game::do_turn()
    ??:?
    main
    ??:?
    __libc_start_main
    ??:?
    _start
    ??:?
[1]    12250 abort (core dumped)  ./cataclysm-launcher

sztosz on 17 Dec 2019

I was trying to debug it, and here
https://github.com/CleverRaven/Cataclysm-DDA/blob/befe6dde51c87c0b89712585aa0cce1f9bca0eef/src/map.cpp#L4384

    for( const tripoint &abs_pos : submaps_with_active_items ) {
        const tripoint local_pos = abs_pos - abs_sub.xy();
        submap *const current_submap = get_submap_at_grid( local_pos );
        if( !active || !current_submap->active_items.empty() ) {
            process_items_in_submap( *current_submap, local_pos, processor, signal );
        }
    }
}

const tripoint local_pos = abs_pos - abs_sub.xy(); abs_pos get's value (65537,458753,65538) abs_sub.xy() gets value of (642,140) and that's why local_pos is (64895,458613, 65538).
But when I have breakpoint after that here if( !active || !current_submap->active_items.empty() ) {
and iterate in Clion by hand over submaps_with_active_items all the tripoints have values around (642,144,0), and since 2*16 = 65536, and abs_pos.x = 65537, and abs_pos.z = 65538 I have a crazy hunch that somwhere there tere is some unsinged 16 bit int, that get's overflowed, but it may be just conincidence. I'm gonna build it now with GCC, because it may be some obscure bug in clang :/

Sorry, it's same on GCC.

sztosz on 17 Dec 2019

@molkero what OS are you running?

My test case works consistently on Linux but I can't reproduce the crash on Windows 10.

ymber on 17 Dec 2019

@ymber Based on that https://github.com/CleverRaven/Cataclysm-DDA/issues/35719 I guess @molkero is on Manjaro (OS Version: LSB Version: n/a; Distributor ID: ManjaroLinux; Description: Manjaro Linux; )

sztosz on 17 Dec 2019

👍1

@ymber https://github.com/CleverRaven/Cataclysm-DDA/releases/tag/cdda-jenkins-b10019 Windows_x64 Tiles and I can't reproduce it either.

sztosz on 17 Dec 2019

👍1

@sztosz my guess would be a different/less pedantic version of libcxx in windows as compared to unix; just a hunch though

fossforus on 17 Dec 2019

On OpenBSD, I'm getting SIGBUS errors (usually indicates a use after free)

abs_pos get's value (65537,458753,65538)

Given that there are backtraces about RB trees in some reported crash logs, and that process_items accepts a processor function, my hunch it that a processor happens to remove/modify an item passed to it, which causes the submaps_with_active_items cache to be modified (in map::i_rem, for example) and iterators invalidated, which causes the SIGBUS errors or the garbage values. I'd suggest making a copy of the active submap cache before iterating through it to see if it fixes the crashes.

Qrox on 18 Dec 2019

Here are the logs and save of a save of mine that can trigger the bug consistantly by just walking left. (The same log and save are the same as provided in the marked-duplicate issue #36208)

crash_and_debug.zip
Save.zip

HarleyLorenzo on 18 Dec 2019

❤1

That commit doesn't appear to be particularly-important, so I'll put up a PR to revert this commit to get things working again and let @kevingranade decide how best to fix that patch.

mark7 on 18 Dec 2019

❤2

Was this page helpful?

0 / 5 - 0 ratings