Minetest: LuaJIT "not enough memory" error despite enough memory available

Created on 1 Aug 2015  ·  60Comments  ·  Source: minetest/minetest

VanessaE reported a fatal server error with the message "not enough memory", which originates from the Lua interpreter. Obviously there is still a lot of memory left on the system at the time of this error, suggesting the problem is with some code somewhere attempting to allocate too much memory.

We need better diagnostics to determine where this originated from first, and then replicate the error a second time to find the actual cause.

2015-07-15 17:42:19: ACTION[ServerThread]: awesomegirl leaves game. List of players: solime 
2015-07-15 17:42:59: ACTION[ServerThread]: solime times out. List of players: 
2015-07-15 17:49:23: ACTION[ServerThread]: Server: Player with the name "Admin" tried to connect from 178.120.209.181 but it was disallowed for the following reason: That is a clearly false, misleading, or otherwise disallowed username. Please choose a unique username and try again.
2015-07-15 17:49:50: ACTION[ServerThread]: Server: Player with the name "Admin" tried to connect from 24.33.145.158 but it was disallowed for the following reason: That is a clearly false, misleading, or otherwise disallowed username. Please choose a unique username and try again.
2015-07-15 18:04:40: ACTION[ServerThread]: User VanessaE from IRC logs in as VanessaEzekowitz
2015-07-15 18:07:14: ACTION[ServerThread]: Server: meiko1 supplied empty password
2015-07-15 18:08:46: ACTION[ServerThread]: Moving untaker to static spawnpoint at (-219,10,-475)
2015-07-15 18:08:46: ACTION[ServerThread]: Moving untaker to static_spawnpoint at (-219,10,-475)
2015-07-15 18:08:46: ACTION[ServerThread]: untaker [73.212.126.39] joins game. 
2015-07-15 18:08:46: ACTION[ServerThread]: untaker joins game. List of players: untaker
Created new entry for `73.212.126.39'
2015-07-15 18:09:28: ERROR[main]: UNRECOVERABLE error occurred. Stopping server. Please fix the following error:
2015-07-15 18:09:28: ERROR[main]: not enough memory

In thread 7ffff7fe0760:
/home/minetest/minetest_core/src/server.cpp:505: void Server::step(float): A fatal error occurred: not enough memory
Debug stacks:
DEBUG STACK FOR THREAD 7ffff0920700:
#0  virtual void* EmergeThread::Thread()
(Leftover data: #1  MapBlock* ServerMap::loadBlock(v3s16))
(Leftover data: #2  void ServerMap::loadBlock(std::string*, v3s16, MapSector*, bool))
(Leftover data: #3  void ItemStack::deSerialize(std::istream&, IItemDefManager*))
DEBUG STACK FOR THREAD 7ffff1533700:
#0  virtual void* CurlFetchThread::Thread()
DEBUG STACK FOR THREAD 7ffff1d34700:
#0  virtual void* ServerThread::Thread()
#1  void Server::Receive()
(Leftover data: #2  void Server::SendBlocks(float))
(Leftover data: #3  void RemoteClient::GetNextBlocks(ServerEnvironment*, EmergeManager*, float, std::vector<PrioritySortedBlockTransfer>&))
(Leftover data: #4  void ItemStack::serialize(std::ostream&) const)
(Leftover data: #5  bool getCraftingResult(Inventory*, ItemStack&, std::vector<ItemStack>&, bool, IGameDef*))
(Leftover data: #6  void ItemStack::deSerialize(std::istream&, IItemDefManager*))
DEBUG STACK FOR THREAD 7ffff7fe0760:
#0  int main(int, char**)
#1  Dedicated server branch
#2  void dedicated_server_loop(Server&, bool&)
#3  void Server::step(float)
[New Thread 0x7ffff0920700 (LWP 19112)]
[New Thread 0x7fffeabd2700 (LWP 12570)]

Program received signal SIGABRT, Aborted.
0x00007ffff5e0e165 in raise () from /lib/x86_64-linux-gnu/libc.so.6

Thread 7 (Thread 0x7fffeabd2700 (LWP 12570)):
#0  0x00007ffff7294344 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x00007ffff6e72163 in ?? () from /usr/lib/x86_64-linux-gnu/libleveldb.so.1
No symbol table info available.
#2  0x00007ffff728fb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3  0x00007ffff5eb795d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#4  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 6 (Thread 0x7ffff0920700 (LWP 19112)):
#0  0x00007ffff7296490 in sem_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x00000000005eff66 in Event::wait (this=0xc35d80) at /home/minetest/minetest_core/src/jthread/pthread/jevent.cpp:58
        sem_wait_retval = 0
        __PRETTY_FUNCTION__ = "void Event::wait()"
#2  0x000000000073eb16 in EmergeThread::Thread (this=0xc35ce0) at /home/minetest/minetest_core/src/emerge.cpp:442
        allow_generate = true
        data = {vmanip = 0x0, seed = 0, blockpos_min = {X = 0, Y = 0, Z = 0}, blockpos_max = {X = 0, Y = 0, Z = 0}, blockpos_requested = {X = 0, Y = 0, Z = 0}, transforming_liquid = {m_set = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<irr::core::vector3d<short> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<irr::core::vector3d<short> > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<irr::core::vector3d<short>, irr::core::vector3d<short>, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x0, _M_left = 0x7ffff091fb10, _M_right = 0x7ffff091fb10}, _M_node_count = 0}}}, m_queue = {c = {<std::_Deque_base<irr::core::vector3d<short>, std::allocator<irr::core::vector3d<short> > >> = {_M_impl = {<std::allocator<irr::core::vector3d<short> >> = {<__gnu_cxx::new_allocator<irr::core::vector3d<short> >> = {<No data fields>}, <No data fields>}, _M_map = 0x176ea50, _M_map_size = 8, _M_start = {_M_cur = 0x1875540, _M_first = 0x1875540, _M_last = 0x187573e, _M_node = 0x176ea68}, _M_finish = {_M_cur = 0x1875540, _M_first = 0x1875540, _M_last = 0x187573e, _M_node = 0x176ea68}}}, <No data fields>}}}, nodedef = 0x0}
        block = 0x1876470
        modified_blocks = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<irr::core::vector3d<short> const, MapBlock*> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<irr::core::vector3d<short> const, MapBlock*> > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<irr::core::vector3d<short>, irr::core::vector3d<short>, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x1876bf0, _M_left = 0x1876bf0, _M_right = 0x1876bf0}, _M_node_count = 1}}}
        __debug_stacker = {m_stack = 0xcc6210, m_overflowed = false}
        __PRETTY_FUNCTION__ = "virtual void* EmergeThread::Thread()"
        last_tried_pos = {X = -11, Y = 3, Z = -26}
        flags = 1 '\001'
        p = {X = -11, Y = 3, Z = -26}
#3  0x00000000005f0815 in JThread::TheThread (param=0xc35ce0) at /home/minetest/minetest_core/src/jthread/pthread/jthread.cpp:157
        jthread = 0xc35ce0
#4  0x00007ffff728fb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5  0x00007ffff5eb795d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 5 (Thread 0x7ffff1533700 (LWP 18719)):
#0  0x00007ffff7296581 in sem_timedwait () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x00000000005f032d in JSemaphore::Wait (this=0xc2fa18, time_ms=100000000) at /home/minetest/minetest_core/src/jthread/pthread/jsemaphore.cpp:126
        waittime = {tv_sec = 1437076468, tv_nsec = 537330000}
        now = {tv_sec = 1436976468, tv_usec = 537330}
        __PRETTY_FUNCTION__ = "bool JSemaphore::Wait(unsigned int)"
        sem_wait_retval = 32767
#2  0x0000000000761c1b in MutexedQueue<CurlFetchThread::Request>::pop_front (this=0xc2f9a0, wait_time_max_ms=100000000) at /home/minetest/minetest_core/src/util/container.h:231
No locals.
#3  0x00000000007608a2 in CurlFetchThread::waitForRequest (this=0xc2f920, timeout=100000000) at /home/minetest/minetest_core/src/httpfetch.cpp:548
        req = {type = 3960159088, fetch_request = {url = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc2fa40 "p;\v\354\377\177"}}, caller = 140737153547128, request_id = 140737242148048, timeout = 140737153547128, connect_timeout = 7740376, multipart = 248, post_fields = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = 4048760064, _M_parent = 0x7630e8, _M_left = 0x7ffff1532d20, _M_right = 0x7ffff1532d30}, _M_node_count = 140737153547120}}}, post_data = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7ffff1533700 ""}}, extra_headers = {<std::_Vector_base<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {_M_impl = {<std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, _M_start = 0x7ffff1532d40, _M_finish = 0x761db5, _M_end_of_storage = 0x7fffec0b3b70}}, <No data fields>}, useragent = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc2fa40 "p;\v\354\377\177"}}}, event = 0x7fffec0b3b70}
#4  0x0000000000760e22 in CurlFetchThread::Thread (this=0xc2f920) at /home/minetest/minetest_core/src/httpfetch.cpp:679
        still_ongoing = 0
        __debug_stacker = {m_stack = 0xcc0ae0, m_overflowed = false}
        __PRETTY_FUNCTION__ = "virtual void* CurlFetchThread::Thread()"
        pool = {handles = {<std::_List_base<void*, std::allocator<void*> >> = {_M_impl = {<std::allocator<std::_List_node<void*> >> = {<__gnu_cxx::new_allocator<std::_List_node<void*> >> = {<No data fields>}, <No data fields>}, _M_node = {_M_next = 0x7fffecb74630, _M_prev = 0x7fffecb74630}}}, <No data fields>}}
        mres = CURLM_OK
#5  0x00000000005f0815 in JThread::TheThread (param=0xc2f920) at /home/minetest/minetest_core/src/jthread/pthread/jthread.cpp:157
        jthread = 0xc2f920
#6  0x00007ffff728fb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#7  0x00007ffff5eb795d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#8  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 4 (Thread 0x7ffff1d34700 (LWP 18713)):
#0  0x00007ffff7296581 in sem_timedwait () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x00000000005f032d in JSemaphore::Wait (this=0x7fffffffe2a0, time_ms=30) at /home/minetest/minetest_core/src/jthread/pthread/jsemaphore.cpp:126
        waittime = {tv_sec = 1436976568, tv_nsec = 705360000}
        now = {tv_sec = 1436976568, tv_usec = 675360}
        __PRETTY_FUNCTION__ = "bool JSemaphore::Wait(unsigned int)"
        sem_wait_retval = 0
#2  0x0000000000606e39 in MutexedQueue<con::ConnectionEvent>::pop_front (this=0x7fffffffe228, wait_time_max_ms=30) at /home/minetest/minetest_core/src/util/container.h:231
No locals.
#3  0x00000000005ffd83 in con::Connection::waitEvent (this=0x7fffffffe180, timeout_ms=30) at /home/minetest/minetest_core/src/network/connection.cpp:2833
No locals.
#4  0x0000000000600175 in con::Connection::Receive (this=0x7fffffffe180, pkt=0x7ffff1d33d00) at /home/minetest/minetest_core/src/network/connection.cpp:2890
        e = {type = 4294959448, peer_id = 32767, data = {data = 0xbda520 "", m_size = 13357104}, timeout = false, address = {m_addr_family = 0, m_address = {ipv4 = {sin_family = 13824, sin_port = 61907, sin_addr = {s_addr = 32767}, sin_zero = "y\036`\000\000\000\000"}, ipv6 = {sin6_family = 13824, sin6_port = 61907, sin6_flowinfo = 32767, sin6_addr = {__in6_u = {__u6_addr8 = "y\036`\000\000\000\000\000 \245\275\000\000\000\000", __u6_addr16 = {7801, 96, 0, 0, 42272, 189, 0, 0}, __u6_addr32 = {6299257, 0, 12428576, 0}}}, sin6_scope_id = 4057151024}}, m_port = 32767}}
#5  0x0000000000804ceb in Server::Receive (this=0x7fffffffe050) at /home/minetest/minetest_core/src/server.cpp:1036
        pkt = {m_data = {<std::_Vector_base<unsigned char, std::allocator<unsigned char> >> = {_M_impl = {<std::allocator<unsigned char>> = {<__gnu_cxx::new_allocator<unsigned char>> = {<No data fields>}, <No data fields>}, _M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}}, <No data fields>}, m_datasize = 0, m_read_offset = 0, m_command = 0, m_peer_id = 0}
        __debug_stacker = {m_stack = 0xcbd030, m_overflowed = false}
        __PRETTY_FUNCTION__ = "void Server::Receive()"
        data = {data = 0x0, m_size = 0, refcount = 0x7fffe4562b50}
        peer_id = 0
#6  0x00000000007febd7 in ServerThread::Thread (this=0xc35620) at /home/minetest/minetest_core/src/server.cpp:109
        __debug_stacker = {m_stack = 0xcbd030, m_overflowed = false}
        __PRETTY_FUNCTION__ = "virtual void* ServerThread::Thread()"
#7  0x00000000005f0815 in JThread::TheThread (param=0xc35620) at /home/minetest/minetest_core/src/jthread/pthread/jthread.cpp:157
        jthread = 0xc35620
#8  0x00007ffff728fb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#9  0x00007ffff5eb795d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#10 0x0000000000000000 in ?? ()
No symbol table info available.

Thread 3 (Thread 0x7ffff2945700 (LWP 18697)):
#0  0x00007ffff5eb1453 in select () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000000000083753a in UDPSocket::WaitData (this=0x7fffffffe180, timeout_ms=50) at /home/minetest/minetest_core/src/socket.cpp:557
        readset = {fds_bits = {1024, 0 <repeats 15 times>}}
        result = 32767
        tv = {tv_sec = 0, tv_usec = 48930}
#2  0x00000000005fc4f7 in con::ConnectionReceiveThread::receive (this=0x7fffffffe438) at /home/minetest/minetest_core/src/network/connection.cpp:2108
        packet_maxsize = 1500
        packetdata = {data = 0x1875540 "", m_size = 1500, refcount = 0x176d9f0}
        packet_queued = true
        loop_count = 0
#3  0x00000000005fb917 in con::ConnectionReceiveThread::Thread (this=0x7fffffffe438) at /home/minetest/minetest_core/src/network/connection.cpp:2032
        sp = {m_profiler = 0xbfb380, m_name = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x1876c08 "ConnectionReceive: [con(10/0)]"}}, m_timer = 0x1876090, m_type = SPT_AVG}
        __PRETTY_FUNCTION__ = "virtual void* con::ConnectionReceiveThread::Thread()"
        ThreadIdentifier = <incomplete type>
#4  0x00000000005f0815 in JThread::TheThread (param=0x7fffffffe438) at /home/minetest/minetest_core/src/jthread/pthread/jthread.cpp:157
        jthread = 0x7fffffffe438
#5  0x00007ffff728fb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#6  0x00007ffff5eb795d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#7  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 2 (Thread 0x7ffff3146700 (LWP 18694)):
#0  0x00007ffff7296581 in sem_timedwait () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1  0x00000000005f032d in JSemaphore::Wait (this=0x7fffffffe408, time_ms=50) at /home/minetest/minetest_core/src/jthread/pthread/jsemaphore.cpp:126
        waittime = {tv_sec = 1436976568, tv_nsec = 695461000}
        now = {tv_sec = 1436976568, tv_usec = 645461}
        __PRETTY_FUNCTION__ = "bool JSemaphore::Wait(unsigned int)"
        sem_wait_retval = 32767
#2  0x00000000005f5ebc in con::ConnectionSendThread::Thread (this=0x7fffffffe330) at /home/minetest/minetest_core/src/network/connection.cpp:1275
        sp = {m_profiler = 0xbfb380, m_name = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7fffec6a4ca8 "ConnectionSend: [con(10/0)]"}}, m_timer = 0x7fffec2e8220, m_type = SPT_AVG}
        dtime = 0.0390000008
        c = {type = con::CONNCMD_NONE, address = {m_addr_family = 0, m_address = {ipv4 = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, ipv6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, m_port = 0}, peer_id = 0, channelnum = 0 '\000', data = {data = 0x0, m_size = 0}, reliable = false, raw = false}
        __PRETTY_FUNCTION__ = "virtual void* con::ConnectionSendThread::Thread()"
        curtime = 2457491781
        lasttime = 2457491742
        ThreadIdentifier = <incomplete type>
#3  0x00000000005f0815 in JThread::TheThread (param=0x7fffffffe330) at /home/minetest/minetest_core/src/jthread/pthread/jthread.cpp:157
        jthread = 0x7fffffffe330
#4  0x00007ffff728fb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5  0x00007ffff5eb795d in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#6  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 1 (Thread 0x7ffff7fe0760 (LWP 18682)):
#0  0x00007ffff5e0e165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00007ffff5e113e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2  0x00000000007282bc in fatal_error_fn (msg=0x7fffe4c54628 "not enough memory", file=0x8f57e0 "/home/minetest/minetest_core/src/server.cpp", line=505, function=0x8f9330 "void Server::step(float)") at /home/minetest/minetest_core/src/debug.cpp:165
No locals.
#3  0x0000000000801e43 in Server::step (this=0x7fffffffe050, dtime=0.100000001) at /home/minetest/minetest_core/src/server.cpp:505
        __debug_stacker = {m_stack = 0xc0e760, m_overflowed = false}
        __PRETTY_FUNCTION__ = "void Server::step(float)"
        async_err = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7fffe4c54628 "not enough memory"}}
#4  0x0000000000811659 in dedicated_server_loop (server=..., kill=@0xbfb320: false) at /home/minetest/minetest_core/src/server.cpp:3413
        steplen = 0.100000001
        profiler_print_interval = 0
        __debug_stacker = {m_stack = 0xc0e760, m_overflowed = false}
        __PRETTY_FUNCTION__ = "void dedicated_server_loop(Server&, bool&)"
        m_profiler_interval = {m_accumulator = 0}
#5  0x0000000000782dee in run_dedicated_server (game_params=..., cmd_args=...) at /home/minetest/minetest_core/src/main.cpp:846
        __debug_stacker = {m_stack = 0xc0e760, m_overflowed = false}
        bind_addr = {m_addr_family = 2, m_address = {ipv4 = {sin_family = 2, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, ipv6 = {sin6_family = 2, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, m_port = 30002}
        kill = @0xbfb320: false
        bind_str = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xbd7158 ""}}
        server = {<con::PeerHandler> = {_vptr.PeerHandler = 0x8f94b0}, <MapEventReceiver> = {_vptr.MapEventReceiver = 0x8f9550}, <InventoryManager> = {_vptr.InventoryManager = 0x8f9568}, <IGameDef> = {_vptr.IGameDef = 0x8f95a0}, m_bind_addr = {m_addr_family = 2, m_address = {ipv4 = {sin_family = 2, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, ipv6 = {sin6_family = 2, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, m_port = 30002}, m_path_world = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc30ed8 "/home/minetest/.minetest/worlds/Nostalgia_World"}}, m_gamespec = {id = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc0db78 "mt_nostalgia"}}, path = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc30fd8 "/home/minetest/.minetest/games/mt_nostalgia"}}, gamemods_path = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc30dc8 "/home/minetest/.minetest/games/mt_nostalgia/mods"}}, addon_mods_paths = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0xc20f10, _M_left = 0xc20f10, _M_right = 0xc20f10}, _M_node_count = 1}}}, name = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc2fc98 "Minetest Nostalgia"}}, menuicon_path = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xbd7158 ""}}}, m_simple_singleplayer_mode = false, m_async_fatal_error = {m_value = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7fffe4c54628 "not enough memory"}}, m_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}}, m_liquid_transform_timer = 0.10255108, m_liquid_transform_every = 1, m_print_info_timer = 0, m_masterserver_timer = 100.109047, m_objectdata_timer = 0, m_emergethread_trigger_timer = 0.100000001, m_savemap_timer = 100.099045, m_map_timer_and_unload_interval = {m_accumulator = 2.33476329}, m_env = 0x7fffec00a1e0, m_env_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, m_con = {m_udpSocket = {m_handle = 10, m_timeout_ms = 5, m_addr_family = 2}, m_command_queue = {m_queue = {<std::_Deque_base<con::ConnectionCommand, std::allocator<con::ConnectionCommand> >> = {_M_impl = {<std::allocator<con::ConnectionCommand>> = {<__gnu_cxx::new_allocator<con::ConnectionCommand>> = {<No data fields>}, <No data fields>}, _M_map = 0xc1f760, _M_map_size = 8, _M_start = {_M_cur = 0x7fffe4bb8160, _M_first = 0x7fffe4bb80d0, _M_last = 0x7fffe4bb82c8, _M_node = 0xc1f798}, _M_finish = {_M_cur = 0x7fffe4bb8160, _M_first = 0x7fffe4bb80d0, _M_last = 0x7fffe4bb82c8, _M_node = 0xc1f798}}}, <No data fields>}, m_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, m_size = {m_semaphore = {__size = "\000\000\000\000\200", '\000' <repeats 11 times>, "y\036`\000\000\000\000\000\020\215\303\000\000\000\000", __align = 549755813888}}}, m_event_queue = {m_queue = {<std::_Deque_base<con::ConnectionEvent, std::allocator<con::ConnectionEvent> >> = {_M_impl = {<std::allocator<con::ConnectionEvent>> = {<__gnu_cxx::new_allocator<con::ConnectionEvent>> = {<No data fields>}, <No data fields>}, _M_map = 0xc31030, _M_map_size = 8, _M_start = {_M_cur = 0x1875e80, _M_first = 0x1875d40, _M_last = 0x1875f40, _M_node = 0xc31068}, _M_finish = {_M_cur = 0x1875e80, _M_first = 0x1875d40, _M_last = 0x1875f40, _M_node = 0xc31068}}}, <No data fields>}, m_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, m_size = {m_semaphore = {__size = "\000\000\000\000\200\000\000\000\001\000\000\000\000\000\000\000\330\017\303", '\000' <repeats 12 times>, __align = 549755813888}}}, m_peer_id = 1, m_protocol_id = 1329951747, m_peers = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<unsigned short const, con::Peer*> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<unsigned short const, con::Peer*> > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<unsigned short, unsigned short, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x7fffec1465e0, _M_left = 0x7fffec1465e0, _M_right = 0x7fffec1465e0}, _M_node_count = 1}}}, m_peer_ids = {<std::_List_base<unsigned short, std::allocator<unsigned short> >> = {_M_impl = {<std::allocator<std::_List_node<unsigned short> >> = {<__gnu_cxx::new_allocator<std::_List_node<unsigned short> >> = {<No data fields>}, <No data fields>}, _M_node = {_M_next = 0x7fffecd5d060, _M_prev = 0x7fffecd5d060}}}, <No data fields>}, m_peers_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, m_sendThread = {<JThread> = {_vptr.JThread = 0x86b1d0, threadid = 140737271588608, started = true, retval = 0x0, running = true, requeststop = false, continuemutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, continuemutex2 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}}, m_connection = 0x7fffffffe180, m_max_packet_size = 512, m_timeout = 30, m_outgoing_queue = {c = {<std::_Deque_base<con::OutgoingPacket, std::allocator<con::OutgoingPacket> >> = {_M_impl = {<std::allocator<con::OutgoingPacket>> = {<__gnu_cxx::new_allocator<con::OutgoingPacket>> = {<No data fields>}, <No data fields>}, _M_map = 0xc30f10, _M_map_size = 8, _M_start = {_M_cur = 0x7fffeca6d0f8, _M_first = 0x7fffeca6cf40, _M_last = 0x7fffeca6d120, _M_node = 0xc30f48}, _M_finish = {_M_cur = 0x7fffeca6d0f8, _M_first = 0x7fffeca6cf40, _M_last = 0x7fffeca6d120, _M_node = 0xc30f48}}}, <No data fields>}}, m_send_sleep_semaphore = {m_semaphore = {__size = "\000\000\000\000\200\000\000\000\001\000\000\000\000\000\000\000}0f\366\377\177\000\000\240\344\377\377\377\177\000", __align = 549755813888}}, m_iteration_packets_avaialble = 1024, m_max_commands_per_iteration = 1, m_max_data_packets_per_iteration = 1024, m_max_packets_requeued = 256}, m_receiveThread = {<JThread> = {_vptr.JThread = 0x86b190, threadid = 140737263195904, started = true, retval = 0x0, running = true, requeststop = false, continuemutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, continuemutex2 = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}}, m_connection = 0x7fffffffe180}, m_info_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, m_bc_peerhandler = 0x7fffffffe050, m_bc_receive_timeout = 30, m_shutting_down = false, m_next_remote_peer_id = 123}, m_banmanager = 0xc35e00, m_rollback = 0x7fffec00b900, m_enable_rollback_recording = true, m_emerge = 0xc356b0, m_script = 0xc57a10, m_itemdef = 0xc311b0, m_nodedef = 0xc320c0, m_craftdef = 0xc34e50, m_event = 0xc34a50, m_mods = {<std::_Vector_base<ModSpec, std::allocator<ModSpec> >> = {_M_impl = {<std::allocator<ModSpec>> = {<__gnu_cxx::new_allocator<ModSpec>> = {<No data fields>}, <No data fields>}, _M_start = 0xc5b4a0, _M_finish = 0xc5c730, _M_end_of_storage = 0xc5c730}}, <No data fields>}, m_step_dtime = 0.100000001, m_step_dtime_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, m_lag = 0.100999758, m_thread = 0xc35620, m_time_of_day_send_timer = 5, m_uptime = {m_value = 21400.100319348276, m_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}}, m_clients = {m_con = 0x7fffffffe180, m_clients_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, m_clients = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<unsigned short const, RemoteClient*> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<unsigned short const, RemoteClient*> > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<unsigned short, unsigned short, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x7fffe44e50f0, _M_left = 0x7fffe44e50f0, _M_right = 0x7fffe44e50f0}, _M_node_count = 1}}}, m_clients_names = {<std::_Vector_base<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {_M_impl = {<std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, _M_start = 0x7fffeca8bc90, _M_finish = 0x7fffeca8bc98, _M_end_of_storage = 0x7fffeca8bcb0}}, <No data fields>}, m_env = 0x7fffec00a1e0, m_env_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, m_print_info_timer = 10.1000023, static statenames = 0xbd6bc0}, m_peer_change_queue = {c = {<std::_Deque_base<con::PeerChange, std::allocator<con::PeerChange> >> = {_M_impl = {<std::allocator<con::PeerChange>> = {<__gnu_cxx::new_allocator<con::PeerChange>> = {<No data fields>}, <No data fields>}, _M_map = 0xc35160, _M_map_size = 8, _M_start = {_M_cur = 0x1117988, _M_first = 0x1117800, _M_last = 0x1117a00, _M_node = 0xc35190}, _M_finish = {_M_cur = 0x1117988, _M_first = 0x1117800, _M_last = 0x1117a00, _M_node = 0xc35190}}}, <No data fields>}}, m_shutdown_requested = false, m_unsent_map_edit_queue = {c = {<std::_Deque_base<MapEditEvent*, std::allocator<MapEditEvent*> >> = {_M_impl = {<std::allocator<MapEditEvent*>> = {<__gnu_cxx::new_allocator<MapEditEvent*>> = {<No data fields>}, <No data fields>}, _M_map = 0xc353c0, _M_map_size = 8, _M_start = {_M_cur = 0x7fffe450e738, _M_first = 0x7fffe450e570, _M_last = 0x7fffe450e770, _M_node = 0xc353e0}, _M_finish = {_M_cur = 0x7fffe450e738, _M_first = 0x7fffe450e570, _M_last = 0x7fffe450e770, _M_node = 0xc353e0}}}, <No data fields>}}, m_ignore_map_edit_events = false, m_ignore_map_edit_events_area = {MinEdge = {X = 1, Y = 1, Z = 1}, MaxEdge = {X = 0, Y = 0, Z = 0}}, m_ignore_map_edit_events_peer_id = 0, m_media = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, MediaInfo> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, MediaInfo> > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x7fffec00ebe0, _M_left = 0x7fffec019780, _M_right = 0x7fffec00c5c0}, _M_node_count = 173}}}, m_playing_sounds = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<int const, ServerPlayingSound> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<int const, ServerPlayingSound> > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<int, int, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x0, _M_left = 0x7fffffffe768, _M_right = 0x7fffffffe768}, _M_node_count = 0}}}, m_next_sound_id = 0, m_detached_inventories = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Inventory*> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, Inventory*> > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x7fffec0090e0, _M_left = 0xc8fef0, _M_right = 0x7fffec005ff0}, _M_node_count = 3}}}, m_particlespawner_ids = {<std::_Vector_base<unsigned int, std::allocator<unsigned int> >> = {_M_impl = {<std::allocator<unsigned int>> = {<__gnu_cxx::new_allocator<unsigned int>> = {<No data fields>}, <No data fields>}, _M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}}, <No data fields>}}
#6  0x000000000077de27 in main (argc=11, argv=0x7fffffffec08) at /home/minetest/minetest_core/src/main.cpp:233
        retval = 8811197
        cmd_args = {m_settings = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, SettingsEntry> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, SettingsEntry> > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0xc0dbc0, _M_left = 0xc0db10, _M_right = 0xc0da30}, _M_node_count = 5}}}, m_defaults = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, SettingsEntry> > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, SettingsEntry> > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x0, _M_left = 0x7fffffffe958, _M_right = 0x7fffffffe958}, _M_node_count = 0}}}, m_callbacks = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<std::pair<void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void*), void*>, std::allocator<std::pair<void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void*), void*> > > > > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<std::pair<void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void*), void*>, std::allocator<std::pair<void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void*), void*> > > > > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x0, _M_left = 0x7fffffffe988, _M_right = 0x7fffffffe988}, _M_node_count = 0}}}, m_callbackMutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, m_mutex = {mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}}
        cmd_args_ok = true
        __debug_stacker = {m_stack = 0xc0e760, m_overflowed = false}
        __PRETTY_FUNCTION__ = "int main(int, char**)"
        game_params = {socket_port = 30002, world_path = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc30ed8 "/home/minetest/.minetest/worlds/Nostalgia_World"}}, game_spec = {id = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc0db78 "mt_nostalgia"}}, path = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc30fd8 "/home/minetest/.minetest/games/mt_nostalgia"}}, gamemods_path = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc30dc8 "/home/minetest/.minetest/games/mt_nostalgia/mods"}}, addon_mods_paths = {_M_t = {_M_impl = {<std::allocator<std::_Rb_tree_node<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<__gnu_cxx::new_allocator<std::_Rb_tree_node<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> = {<No data fields>}, <No data fields>}, _M_key_compare = {<std::binary_function<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0xc31080, _M_left = 0xc31080, _M_right = 0xc31080}, _M_node_count = 1}}}, name = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc2fc98 "Minetest Nostalgia"}}, menuicon_path = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xbd7158 ""}}}, is_dedicated_server = true, log_level = 2}
A debugging session is active.

    Inferior 1 [process 18682] will be killed.
Bug Can't fix Upstream issue
Source
picture kwolekr

Most helpful comment

@paramat As of now, LuaJIT is the only option for fast enough mods, and FFI is the only option to escape the memory limits of LuaJIT. In my eyes, "we should not expect or force mods to use it as a way around the memory issue" is to want to eat your cake and have it.

Maybe I'm wrong. If you have a suggestion for another way to allow mods to run fast and use all available memory, I'm all ears.

It matters little how well or bad maintained LuaJIT is, because it's the only option speed wise for some people, but it has that annoying problem of not being able to use more than 4 Gb space (1 Gb in many cases). As of now, there's no other alternative for using all available memory than allocating it through FFI.

Not all mods will need this. Only mods that have big storage requirements.

picture ghost on 5 May 2018
👍2

All 60 comments

Getting the same error, even when there's quite a bit of unused system memory:

2015-07-31 19:53:11: ACTION[ServerThread]: Marlingirl places node default:junglewood at (1385,6,-465)
2015-07-31 19:53:12: ERROR[main]: UNRECOVERABLE error occurred. Stopping server. Please fix the following error:
2015-07-31 19:53:12: ERROR[main]: not enough memory

In thread 7f70d39cb780:
/home/craig/minetest/minetest-tmp/minetest/src/server.cpp:522: void Server::step(float): A fatal error occurred: not enough memory
Debug stacks:
DEBUG STACK FOR THREAD 7f70c92cc700:
#0  virtual void* EmergeThread::Thread()
(Leftover data: #1  MapBlock* ServerMap::loadBlock(v3s16))
(Leftover data: #2  void ServerMap::loadBlock(std::string*, v3s16, MapSector*, bool))
(Leftover data: #3  void ItemStack::deSerialize(std::istream&, IItemDefManager*))
DEBUG STACK FOR THREAD 7f70c9ccd700:
#0  virtual void* CurlFetchThread::Thread()
DEBUG STACK FOR THREAD 7f70ca6ce700:
#0  virtual void* ServerThread::Thread()
#1  void Server::Receive()
#2  void Server::ProcessData(NetworkPacket*)
(Leftover data: #3  void RemoteClient::GetNextBlocks(ServerEnvironment*, EmergeManager*, float, std::vector<PrioritySortedBlockTransfer>&))
(Leftover data: #4  void ItemStack::serialize(std::ostream&) const)
(Leftover data: #5  bool getCraftingResult(Inventory*, ItemStack&, std::vector<ItemStack>&, bool, IGameDef*))
(Leftover data: #6  void ItemStack::deSerialize(std::istream&, IItemDefManager*))
DEBUG STACK FOR THREAD 7f70d39cb780:
#0  int main(int, char**)
#1  Dedicated server branch
#2  void dedicated_server_loop(Server&, bool&)
#3  void Server::step(float)
(Leftover data: #4  void Server::SendAccessDenied_Legacy(irr::u16, const wstring&))
davisonio picture davisonio on 2 Aug 2015

Unfortunately this is going to be very hard to track down.

In order to get a backtrace of the problem, debug.traceback needs to be called while still in context of the failure. However, in the case of LUA_ERRMEM, the error handler is not called - after lua_pcall returns, the execution stack has already been unwound.

The best we can do (without modifying lua/luajit) is tracking which callback the OOM event happens within.

kwolekr picture kwolekr on 3 Aug 2015

does this memory limit in LuaJIT still apply?
http://lua-users.org/lists/lua-l/2010-11/msg00233.html

twoelk picture twoelk on 5 Aug 2015

@twoelk: Seems so. Lua has no problem consuming 2GiB, but LuaJIT bails at 1GiB. It's also worth noting that Lua (5.3.1) runs this code much faster than LuaJIT (2.0.4) (2.07s Vs. 15.35s).
Test code:

local s = string.rep(" ", 1024)
local t = {}
for i = 1, 1024 * 1024 do
    -- Concatenate i to the string to break Lua's string interning.
    t[i] = s..i
end
ShadowNinja picture ShadowNinja on 7 Aug 2015

Since my last comment I've added some extra diagnostics to help us gain more insight into this problem: https://github.com/minetest/minetest/commit/18cfd89a86af550b3c4663def77a5fac46e895ae https://github.com/minetest/minetest/commit/2b04ab874d75711bc021a0cd8dc7fca68f4e6929 https://github.com/minetest/minetest/commit/bcf47bc67cf3e1c2f410a81e26ceab1bdab06b4a

This should help trace down the problem where it does happen.
I've personally observed the hard LuaJIT memory limits, bailing out usually around 1925 MB and sometimes as low as 513MB. Nevertheless this shouldn't happen while doing something as trivial as displaying a formspec which is what happened in one instance VanessaE reported.

I'm keeping this issue open
1). to decide the fate of LuaJIT, especially seeing as how the author is retiring and it is at risk of becoming a dead project
2). because there might be another issue with an API's implementation that attempts to allocate too many Lua resources on accident

kwolekr picture kwolekr on 15 Aug 2015

Perhaps a few more suggestions for additional diagnostics:

Adds a debug.getsize() function which returns the size in bytes of a Lua object

This toolkit provides various gdb extension commands for analyzing core dump files for nginx and/or luajit.

Other custom C processes with LuaJIT embedded can also be analyzed by this tool as long as the target C program saves the main Lua VM state (lua_State) pointer in a global C variable named globalL, just as in the standard luajit command-line utility program.

t4im picture t4im on 8 Sep 2015

I had the similar problem, and went looking for a solution. I found the following, and tried, it, and it works great. I had a few mapgen mods loaded, and when I flew around to make it work, it would fail, but with this library, it does not run out of memory any longer:

git://github.com/Neopallium/mmap_lowmem.git

It pauses when it starts up, so patch it like this first, or it won't seem to be working if you run it full-screen:

diff --git a/mmap_lowmem.c b/mmap_lowmem.c
index 69e6fa1..ee1ecb4 100644
--- a/mmap_lowmem.c
+++ b/mmap_lowmem.c
@@ -34,7 +34,7 @@ static pthread_mutex_t page_alloc_lock = PTHREAD_MUTEX_INITIALIZER;
 #define MBYTE (KBYTE * 1024)
 #define GBYTE (MBYTE * 1024)

-#define ENABLE_VERBOSE 1
+//#define ENABLE_VERBOSE 1

 #if (ENABLE_VERBOSE != 1)
 #define printf(...)

Then run "make" (and sudo make install if you like)

Run minetest with:

LD_PRELOAD=/usr/lib/libmmap_lowmem_mt.so minetest [args]

I did this:

sudo dpkg-divert --local --divert /usr/bin/minetest.minetest --rename --add /usr/bin/minetest

And /usr/bin/minetest is now a wrapper script:

$ cat =minetest
#!/bin/sh
LD_PRELOAD=/usr/lib/libmmap_lowmem_mt.so
export LD_PRELOAD
exec /usr/bin/minetest.minetest "$@"
KarlHegbloom picture KarlHegbloom on 26 Nov 2015

PS: It says something regarding compiling "luajit" as a position independent binary, so that this thing can move it to the upper end of the low_mem, to maximize the amount of memory that it has available. I have not experimented with recompiling "minecraft" like that. We'll have to look and see how big Minecraft is, I think, to be sure that it fits or whatever, because I'm expecting that it's bigger than the "luajit' binary this library's author is discussing. Also, I wonder if it can be set up such that it's linked in, and only wraps the malloc calls coming from the luajit library, so that the rest of "minetest" can get memory from another area??? If that matters? I don't really understand it very well, so somebody who knows more about it will need to take a look.

KarlHegbloom picture KarlHegbloom on 26 Nov 2015

http://www.freelists.org/post/luajit/Extending-LuaJITs-memory-limit-to-4Gbytes

KarlHegbloom picture KarlHegbloom on 26 Nov 2015

I'm using the ppa minetestdevs-ubuntu-daily-builds. I enabled the deb-src line in /etc/apt/sources.list.d/minetestdevs-ubuntu-daily-builds-wily.list, did apt-get update, and then apt-get source minetest. In debian/rules, above the override_dh_auto_configure: target, I added:

export CFLAGS := -fPIC -ggdb
export CXXFLAGS := -fPIC -ggdb
export LDFLAGS := -pie

... and then ran 'fakeroot debian/rules clean binary" and installed the resulting .deb. It's running great, using the LD_PRELOAD in the aforementioned wrapper script. Using the same world that would crash it with an out of memory error when I pushed it beyond the edge of the map, it now stays running. It does slow down there, but not too bad, considering I'm flying with fast move enabled. I think just walking around it won't be very noticable, and without all that extra code running for the mapgen mods I've got tossed into this world to play with them and see what they do, it won't run any different than usual. Maybe it's the lua garbage collection taking that long with so much RAM? Anyway, it runs.

There's a lot of information on line about the luajit memory limit and it's garbage collector. There's a proposed spec for a better gc for luajit-3... I suppose that will also eliminate this lowmem allocation thing too?

KarlHegbloom picture KarlHegbloom on 26 Nov 2015

http://bayesanalytic.com/access-extra-memory-from-lua-jit/

KarlHegbloom picture KarlHegbloom on 26 Nov 2015

Notifying @kwolekr perhaps the above posts are helpful.

paramat picture paramat on 28 Nov 2015

This is also a modder's competence problem (not entirely). At some point they should try the Lua's garbage collection for their utterly greedy mods : http://luatut.com/collectgarbage.html

Thus the blocker labelling is quite questionable.

kilbith picture kilbith on 18 Feb 2016

This is also a modder's competence problem
Thus the blocker labelling is quite questionable.

It is also present in default minetest game, I've experienced it after running MT straight for ~8hrs or so (walking in singleplayer).

Fixer-007 picture Fixer-007 on 18 Feb 2016

I wouldn't give it blocker label. Its mostly an upstream bug, and not our fault, nor something we can fix. From what I've read the boundary was introduced due to some addressing problems and because the GC doesn't really work on too large data sets either (it gets slower and slower). So its more something luajit has to fix, and not us.

est31 picture est31 on 18 Feb 2016

Ok i was unsure about the label anyway.

paramat picture paramat on 18 Feb 2016

I just ran into this error using dreambuilder on Windows / 64bit. The 32 bit build does not show this problem.

PatrickKursawe picture PatrickKursawe on 5 Mar 2016

If this "known bug" is hear to stay it certainly needs some extensive documentation on the dev-wiki. Modders should be warned to go near this "place" and shown recomended "detours" to avoid falling into this trap.

twoelk picture twoelk on 12 Mar 2016

@twoelk: There's nothing to do but minimize memory usage, which mods should be doing already anyway.

ShadowNinja picture ShadowNinja on 12 Mar 2016

Or get a degree in computer science and perform the work on luajit to fix
it. If wishes were houses... who will let somebody sleep under a desk?

On Sat, Mar 12, 2016, 12:43 ShadowNinja [email protected] wrote:

@twoelk https://github.com/twoelk: There's nothing to do but minimize
memory usage, which mods should be doing already anyway.

—
Reply to this email directly or view it on GitHub
https://github.com/minetest/minetest/issues/2988#issuecomment-195795679.

KarlHegbloom picture KarlHegbloom on 12 Mar 2016

Bumping this issue again, I have a feel that more and more people encounter this after private discussions and this https://forum.minetest.net/viewtopic.php?f=6&t=13826, including me (I have a custom subgame for testing just this, at game start 64bit mt.exe eats 1.7gb and issues OOM error right away.

Here it is:

2016-05-25 16:11:28: ERROR[Main]: ServerError: Lua: OOM error from mod 'darkage' in callback environment_OnGenerated(): not enough memory
2016-05-25 16:11:28: ERROR[Main]: Current Lua memory usage: 22 MB

Interestingly, another person posted same error about OnGenerated() but from other mod

2016-05-19 12:01:27: ERROR[Main]: ServerError: Lua: OOM error from mod 'tsm_mines' in callback environment_OnGenerated(): not enough memory
2016-05-19 12:01:27: ERROR[Main]: Current Lua memory usage: 12 MB

Any clues from this two errors? Bad mods? Engine problems? Why lua memory reading is so low?

Another thing that multiple people noted is... by using 32 bit builds you can avoid some of this lua OOM errors, so some people use 32 bit editions of minetest (kinda awkward to hear this).

Don't ignore this.

Fixer-007 picture Fixer-007 on 25 May 2016

Have you tried using the wrapper library that I posted about?

On Wed, May 25, 2016, 07:21 Fixer [email protected] wrote:

Bumping this issue again, I have a feel that more and more people
encounter this after private discussions and this
https://forum.minetest.net/viewtopic.php?f=6&t=13826, including me (I
have a custom subgame for testing just this, at game start 64bit mt.exe
eats 1.7gb and issues OOM error right away.

Here it is:

2016-05-25 16:11:28: ERROR[Main]: ServerError: Lua: OOM error from mod 'darkage' in callback environment_OnGenerated(): not enough memory
2016-05-25 16:11:28: ERROR[Main]: Current Lua memory usage: 22 MB

Interestingly, another person posted same error about OnGenerated() but
from other mod

2016-05-19 12:01:27: ERROR[Main]: ServerError: Lua: OOM error from mod 'tsm_mines' in callback environment_OnGenerated(): not enough memory
2016-05-19 12:01:27: ERROR[Main]: Current Lua memory usage: 12 MB

Any clues from this two errors? Coincidence? Don't think so. Bad mods?
Engine problems?

Another thing that multiple people noted is... by using 32 bit builds you
can avoid some of this lua OOM errors, so some people use 32 bit editions
of minetest (kinda awkward to hear this).

Don't ignore this.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/minetest/minetest/issues/2988#issuecomment-221573719

KarlHegbloom picture KarlHegbloom on 25 May 2016

@Fixer-007 It's interesting that those both are using mapgen callbacks. Are they doing vmanip stuff? I wonder if it leaks.

sofar picture sofar on 25 May 2016

As far as i know it's partly the lua tables used to store perlin noise maps, 3D noise has 500,000 values per chunk. A complex lua mapgen like some of mine with many 3D noises can use 1+GB of memory that way and cause OOM errors with LuaJIT.

Note there is now a 'noise buffer' option that prevents new lua tables being created for noise, the named table is re-used instead, see usage here https://github.com/paramat/moonrealm/blob/master/init.lua#L243 highly recommended that these buffers are used.

There is also a new way to use 3D noise by storing and using it it in thin slices instead, hmmmm says the optimum usage is to calculate 80x80x2 slices at a time. I'm still too lazy to use this method.

Duane-r uses lua garbage collector code in his lua mapgens.

paramat picture paramat on 26 May 2016

@KarlHegbloom No I don't since I'm on windows os.
@sofar Let me look at the code...
darkage bundled with it has this:

minetest.register_on_generated(function(minp, maxp, seed)
    -- Generate stratus
    local t1 = os.clock()
    minetest.log("info", "[darkage] Generate...")

    local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
    local data = vm:get_data()
    local area = VoxelArea:new{MinEdge=emin, MaxEdge=emax}

--  generate_ore("darkage:desert_stone_with_iron", "default:desert_stone", minp, maxp, seed+0, 1/7/7/7, 3, 5, -15, 40)
    generate_claylike("darkage:mud", minp, maxp, seed+1, 4, 0, 2, 0)
    generate_claylike("darkage:silt", minp, maxp, seed+2, 4, -1, 1, 1)

    generate_strati(minp, maxp, seed, data, area)

    vm:set_data(data)
    vm:write_to_map()

    minetest.log("info", string.format("[darkage] finished after: %.2fs", os.clock() - t1))
end)

tsm_mines has this:

minetest.register_on_generated(function(minp, maxp, seed)
    if minp.y > MINE_DEEP_MIN or minp.y < MINE_DEEP_MAX then
        return
    end
    cnt = cnt+1
    if cnt < 8/MINE_FACTOR then return end
    cnt = 0

    local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
    local data = vm:get_data()
    local va = VoxelArea:new{ MinEdge = emin, MaxEdge = emax }
    local mpos = find_cave(emin,emax,data,va)
    if mpos == nil then return end
    local mpos2 = {x=mpos.x+math.random(0,3),y=mpos.y-1,z=mpos.z}
    local mpos3 = {x=mpos.x,y=mpos.y-2,z=mpos.z+math.random(0,3)}
    data = make_mine(mpos,mpos2,mpos3, data, va, 0)
    vm:set_data(data)
    vm:calc_lighting(emin,emax)
    vm:update_liquids()
    vm:write_to_map()
end)
Fixer-007 picture Fixer-007 on 26 May 2016

I wonder if it's possible to generate the noise using GPU and if that's
fast enough so that less has to be cached?

On Wed, May 25, 2016, 16:54 Fixer [email protected] wrote:

@KarlHegbloom https://github.com/KarlHegbloom No I don't since I'm on
windows os.
@sofar https://github.com/sofar Let me look at the code...
darkage bundled with it has this:

minetest.register_on_generated(function(minp, maxp, seed)
-- Generate stratus
local t1 = os.clock()
minetest.log("info", "[darkage] Generate...")

local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
local data = vm:get_data()
local area = VoxelArea:new{MinEdge=emin, MaxEdge=emax}

-- generate_ore("darkage:desert_stone_with_iron", "default:desert_stone", minp, maxp, seed+0, 1/7/7/7, 3, 5, -15, 40)
generate_claylike("darkage:mud", minp, maxp, seed+1, 4, 0, 2, 0)
generate_claylike("darkage:silt", minp, maxp, seed+2, 4, -1, 1, 1)

generate_strati(minp, maxp, seed, data, area)

vm:set_data(data)
vm:write_to_map()

minetest.log("info", string.format("[darkage] finished after: %.2fs", os.clock() - t1))

end)

tsm_mines has this:

minetest.register_on_generated(function(minp, maxp, seed)
if minp.y > MINE_DEEP_MIN or minp.y < MINE_DEEP_MAX then
return
end
cnt = cnt+1
if cnt < 8/MINE_FACTOR then return end
cnt = 0

local vm, emin, emax = minetest.get_mapgen_object("voxelmanip")
local data = vm:get_data()
local va = VoxelArea:new{ MinEdge = emin, MaxEdge = emax }
local mpos = find_cave(emin,emax,data,va)
if mpos == nil then return end
local mpos2 = {x=mpos.x+math.random(0,3),y=mpos.y-1,z=mpos.z}
local mpos3 = {x=mpos.x,y=mpos.y-2,z=mpos.z+math.random(0,3)}
data = make_mine(mpos,mpos2,mpos3, data, va, 0)
vm:set_data(data)
vm:calc_lighting(emin,emax)
vm:update_liquids()
vm:write_to_map()

end)

—
You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub
https://github.com/minetest/minetest/issues/2988#issuecomment-221731839

KarlHegbloom picture KarlHegbloom on 26 May 2016

The problem is lua JIT. Creating the noise in the GPU is not that productive IMO as it needs to be copied back to RAM, and according to @kwolekr mapgen bottleneck is about copying stuff around, not the noise itself.

If mapgen is prone to problems like this, we should explore to give the ability to do mods in different languages than lua.

est31 picture est31 on 26 May 2016

Ok, so maybe the mapgen noise stuff ought to be stored in RAM owned by C++,
rather than by luajit, opaque to luajit, which can see it only via accessor
methods. That data must not contain lua objects that can be gc'd...

Or just rewrite luajit, right? ;-)

On Thu, May 26, 2016, 12:17 est31 [email protected] wrote:

The problem is lua JIT. Creating the noise in the GPU is not that
productive IMO as it needs to be copied back to RAM, and according to
@kwolekr https://github.com/kwolekr mapgen bottleneck is about copying
stuff around, not the noise itself.

If mapgen is prone to problems like this, we should explore to give the
ability to do mods in different languages than lua.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/minetest/minetest/issues/2988#issuecomment-221951758

KarlHegbloom picture KarlHegbloom on 26 May 2016
😄1

http://askubuntu.com/a/426013
Lua also has a garbage collector, limited memory fixes that ↑, doesn't it?

HybridDog picture HybridDog on 15 Aug 2016

I did some lua mem benchmarks with collectgarbage("count") on default minetest_game 0.4.14-dev and mem usage was really low, like 5 MB at most, but I clearly remember OOM error after testing it much earlier in 0.4.13 for straight 8 hours (or even less) with just walking... I have no idea how it can shoot up from 5 megabyte up to limit :/ back then.

Fixer-007 picture Fixer-007 on 15 Aug 2016

I have no idea how it can shoot up from 5 megabyte up to limit :/ back then.

Maybe it's possible to dump the lua(jit) memory.

HybridDog picture HybridDog on 16 Aug 2016

More weirdness from my minetest with luajit (64bit) on windows 7 sp1 (64bit).

  • @VanessaE practically has no OOMs with her dreambuilder on linux with luajit, but I have them in 5-10 minutes of play (and faster after each game restart after OOM error)
  • Illuna server subgame has darkage mod - no problems on linux, crashes with OOM nearly instantly to me on windows.

In both cases it is

OOM error from mod 'hahahaha' in callback environment_OnGenerated(): not enough memory

Mostly technic-worldgen, darkage, etc
Happens on luajit 2.0.1 and 2.1.0-beta

Those mods are unplayable to me
Lua mem usage was <50-100mb in both cases...

Fixer-007 picture Fixer-007 on 1 Jan 2017

@Fixer-007: There are some changes after 2.1.0-beta2 that might fix this for you. You can try compiling from the git repo on the v2.1 branch. Here's a GitHub mirror.

ShadowNinja picture ShadowNinja on 2 Jan 2017

@ShadowNinja
You mean LJ_GC64 mode? It can be compiled with XCFLAGS=-DLUAJIT_ENABLE_GC64

Useful info:
https://github.com/LuaJIT/LuaJIT/issues/25
https://github.com/LuaJIT/LuaJIT/issues/25#issuecomment-183660706
https://github.com/LuaJIT/LuaJIT/pull/149
https://github.com/LuaJIT/LuaJIT/issues/225
https://github.com/LuaJIT/LuaJIT/commit/c94b921f924c1b37fea52e34f4e01ba8b37d77d0
https://gist.github.com/hami-jp/3e37eafcea43546fa27b8b5d99dd7381
http://dev.minetest.net/Build_Win32_minetest_including_all_required_libraries
https://github.com/LuaJIT/LuaJIT/blob/v2.1/doc/install.html
https://forum.minetest.net/viewtopic.php?pid=76627
Fixer-007 picture Fixer-007 on 8 Mar 2017

How I compiled libluajit.a with msys2/mingw64 on windows 7 (remainder for myself):

pacman -Syu
pacman -Su
pacman -S base-devel git
pacman -S mingw-w64-x86_64-toolchain

diff -Naur LuaJIT-2.1/src/lib_package.c LuaJIT-2.1-old//src/lib_package.c
--- LuaJIT-2.0.2/src/lib_package.c  2013-10-20 16:11:52 +0000
+++ LuaJIT-2.0.2-old//src/lib_package.c 2013-10-20 16:13:28 +0000
@@ -76,9 +76,8 @@
 #ifndef GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS
 #define GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS  4
 #define GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT  2
-BOOL WINAPI GetModuleHandleExA(DWORD, LPCSTR, HMODULE*);
 #endif
-
+WINBASEAPI BOOL WINAPI GetModuleHandleExA(DWORD, LPCSTR, HMODULE*);
 #undef setprogdir

 static void setprogdir(lua_State *L)
  • In /src/Makefile comment BUILDMODE=mixed and uncomment BUILDMODE=static (or use "make BUILDMODE=static" (?)
  • In mingw64 shell run:
    make XCFLAGS=-DLUAJIT_ENABLE_GC64 PREFIX=/opt/mingw64/luajit-2.1.0-git
  • then do:
    make install PREFIX=/opt/mingw64/luajit-2.1.0-git
  • copy libluajit.a in /opt/mingw64/luajit-2.1.0-git/bin from src/somewhere if it was not there
  • create /opt/mingw64/luajit-2.1.0-git/bin/lua/jit directory and copy into files from original src\jit directory.
  • minetest buildbot script looks into libs/luajit, so I put here newly compiled files:
    /include <-- probably not needed
    /lua/jit
    libluajit.a
    luajit.exe <-- probably not needed
  • minetest will pick up libluajit.a and compile with it (and even work)
Fixer-007 picture Fixer-007 on 8 Mar 2017

It seems luajit in this "LJ_GJ64 mode" allows to go past 1-2GB limit, on minetest-dev people suggested something like this to try:
a = {}; for i = 1,3000000 do a[i] = table.copy(minetest.registered_nodes); end
While on simple minetest it gave me instant lua OOM, on luajit with lj_gj64 mode it run with collectgarbage("count")/1024 reporting ~4800 mb of used ram.
It works for some time and then crashes, here is two example backtraces with gdb:

EDIT (ShadowNinja): Moved massive backtraces to gist.

Fixer-007 picture Fixer-007 on 8 Mar 2017

@Fixer-007 here's the crash I observe:

sfan5 picture sfan5 on 15 Mar 2017

Here is minetest 0.4.16 stable with luajit-2.1.0-beta3 GC64 enabled compiled by @sfan5
DL: https://kitsunemimi.pw/tmp/minetest-0.4.16-gc64-win64.7z

It seems to be working for me, I did test it in WE (up to 65 million nodes selected, up to 4.5gb mem use by minetest), Dreambuilder modpack straight fly test for over 1hr 40 min was success (without GC64 I have OOM in 5-10 minutes).

People who have OOMs should definitely test it.

Update:

It seems luajit in this "LJ_GJ64 mode" allows to go past 1-2GB limit, on minetest-dev people suggested something like this to try:
a = {}; for i = 1,3000000 do a[i] = table.copy(minetest.registered_nodes); end
While on simple minetest it gave me instant lua OOM, on luajit with lj_gj64 mode it run with collectgarbage("count")/1024 reporting ~4800 mb of used ram.
It works for some time and then crashes, here is two example backtraces with gdb:

No longer crashes for me on sfan5 build (newer luajit).

Fixer-007 picture Fixer-007 on 8 Aug 2017

The build above worked perfectly for me and a couple of people I have told.
Minetest as a whole also ran much more smoothly than I have ever gotten.

I don't think this information is shared enough - this issue has been around for 2 years now.
Perhaps someone should start informing people which provide 64bit windows builds to start using this method...

neinwhal picture neinwhal on 26 Oct 2017

informing people which provide 64bit windows builds

hello that's me

to start using this method...

gc64 is rather "recent" (5 months) and I haven't had time to verify in person that gc64 is stable enough as a default on Windows
(I actually experienced the opposite on linux, gc64 mode segfaults once enough memory is used)

sfan5 picture sfan5 on 26 Oct 2017

Seems to work for me too, crashes have stopped.

nvrsbr picture nvrsbr on 26 Oct 2017

I don't think this information is shared enough - this issue has been around for 2 years now.
Perhaps someone should start informing people which provide 64bit windows builds to start using this method...

I've created a topic on forum some time ago right here: https://forum.minetest.net/viewtopic.php?f=3&t=18263 (maybe it needs to be moved to News section). I've added link to GC64 version to my forum signature. More and more people should try it and see how stable it is on both win and nix, I had good experience with that build so far on windows.

Fixer-007 picture Fixer-007 on 27 Oct 2017
👍1 
Ok, so maybe the mapgen noise stuff ought to be stored in RAM owned by C++,
rather than by luajit, opaque to luajit, which can see it only via accessor
methods. That data must not contain lua objects that can be gc'd...

@KarlHegbloom done that and it was too slow, https://github.com/asl97/minetest/pull/2.

I believe any solution that should be accepted is one that doesn't slow it down too much.

I do have two other solution:

  1. a new api which unify common stuff (for small light lua mapgen, most of the time is spend fetching + setting the data, writing to map and doing light, liquid update)

it should save 15MB per stuff that each mod uses

...
local data = vm:get_data()
for mg in registed_mg do
  mg(minp, maxp, data, param2)
done
vm:set_data(data)
vm:set_param2_data(param2)
vm:calc_lighting()
vm:update_liquids()
vm:write_to_map()
  1. smaller chunksize(really), smaller chunk means smaller table (data + noise) and more generated event, meaning more chance for gc to do it's stuff before it run out of memory.

There are of course other possible solution, like an iteration (generator) data object implemented in c++ to reduce the heavy 'on demand' lookup and an set data that takes a coroutine which removes the heavy 'on-demand' sets.

asl97 picture asl97 on 8 Nov 2017

There is also a new way to use 3D noise by storing and using it it in thin slices instead, hmmmm says the optimum usage is to calculate 80x80x2 slices at a time. I'm still too lazy to use this method.

There are also memory use optimisations that i detailed here https://forum.minetest.net/viewtopic.php?f=18&t=16043
Latest 'riverdev' mod uses these 3 (but not slices).
Many mods still don't use these optimisations and should.

Smaller chunksize: no, not as a default, for many reasons. However those having problems already have the option of using it.

It's a LuaJIT issue that LuaJIT might solve itself, there is AFAIK already a solution for those using Windows. If LuaJIT is so poorly maintained or development has stopped then relying on it is not a good idea anyway. So i don't think radical solutions are justified, especially when mods should be using the memory use optimisations and already have the option of getting noise data in slices.

paramat picture paramat on 8 Nov 2017

Let's start with why LuaJIT, answer: Speed.

Yes, there are those optimization and such but those wouldn't fix the current (un-updated) mods.
They don't make simple mapgen that don't use noise faster ether.

Like you said, many mods still don't use these optimisations.
The idea is fixing the current api so that today mods works as it is without OOM while maintaining similar speed4.

There only need to be 6 mods with simple mapgen1 to be as heavy as riverdev in terms of memory and takes quarter to half the time even if it does nothing to the data2 (from my calculation).

Radical solutions shouldn't be justified because of some stupid OOM due to some outdated implementation which can't uses more than 1GB.

However, LuaJIT doesn't make getting, setting, writing to map, lighting and liquid faster.
Those are c++ api in minetest, be it lua or luajit, the speed should be the same.

Those are the lower boundary2 which can't be surpass without radical solutions.
Minetest is stuck with an additional 250ms-500ms + 15mb per full lua mapgen at chunksize 5.

Is it slow? not really, it's playable for one player.
Is it fast enough? no way if 15-20 (or 100) clients is the goal.3.

Is it enough to justify radical solutions if it bring lua mapgen from an O(W+L) to an O(1+L) where L is the actual mapgen logic and W is all the unnecessary wasted processing?2

The current minetest.register_on_generated has two kind of lua mapgen, the one with minetest.get_mapgen_object("voxelmanip") and the one without.

A LuaVoxelManip with deferring function when it is_mapgen_vm shouldn't be hard to made.
However it wouldn't be backward compatible with mapgen which doesn't use LuaVoxelManip.

If those two mapgen kind are splitted up, there are a number of optimization that can be done in the engine.

  1. does nothing other than fetching data, setting data, writing to map, lighting and liquid
  2. about 450ms per chunk base on the result from the pr with luajit, krock says around 230ms (+ maybe 100ms for get set data), http://irc.minetest.net/minetest/2017-11-07#i_5130817
  3. #6485
  4. #6617

ps: Smaller chunksize wouldn't be default for a lot of reason but it's still a solution.

asl97 picture asl97 on 12 Nov 2017

Although with drawbacks (only for trusted mods, needs modders to update the code), #6863 will alleviate this.

ghost picture ghost on 6 Jan 2018

This can be resolved by making APIs store lots of data C++ side and use the FFI for fast access

rubenwardy picture rubenwardy on 28 Apr 2018

@rubenwardy If you're using FFI to access the data, you can use FFI to allocate the data as well, as outlined in https://github.com/minetest/minetest/issues/5442#issuecomment-354044118 which avoids any OOM issues in 64 bits.

ghost picture ghost on 29 Apr 2018

@rubenwardy #6863 deals with exactly that. Care to review it?

ghost picture ghost on 5 May 2018

This can be resolved by making APIs store lots of data C++ side and use the FFI for fast access

I'm uncomfortable with this use of 'resolved'. It's true that this works, but it's not a suitable 'resolution' to this issue. LuaJIT needs improving. Depending on LuaJIT is already fairly crazy anyway due to how it is not well maintained and not compatible with more recent versions of Lua.
The FFI feature may be added and may be useful, but we should not expect or force mods to use it as a way around the memory issue, it should be an optional thing that is not essential.
We should not let this issue force us into a false resolution of relying on FFI.

paramat picture paramat on 5 May 2018

@paramat As of now, LuaJIT is the only option for fast enough mods, and FFI is the only option to escape the memory limits of LuaJIT. In my eyes, "we should not expect or force mods to use it as a way around the memory issue" is to want to eat your cake and have it.

Maybe I'm wrong. If you have a suggestion for another way to allow mods to run fast and use all available memory, I'm all ears.

It matters little how well or bad maintained LuaJIT is, because it's the only option speed wise for some people, but it has that annoying problem of not being able to use more than 4 Gb space (1 Gb in many cases). As of now, there's no other alternative for using all available memory than allocating it through FFI.

Not all mods will need this. Only mods that have big storage requirements.

ghost picture ghost on 5 May 2018
👍2

I don't think mods should directly use something like FFI for to improve performance or space usage.
But now and then a function like allocate_table(size) which creates a lua table with its array part size set to contain size elements (e.g. w * h * z elements for a w x h x z cube of nodes (table pointers)) could be useful for performance and/or space improvements. The function can be overridden to simply return an usual empty lua table if it's not supported.

HybridDog picture HybridDog on 5 May 2018

LuaJIT is the only option for fast enough mods

No. It shouldn't be portrayed as essential, often it is used as compensation for overcomplex and poorly coded mods. Being faster means modders have made their mods even more complex and intensive, which of course leads to relying on LuaJIT.

The real issue is that people have unfortunately started to use and rely on something that is actually poorly maintained and has problems. Maybe they weren't aware of the issues so it's usually not their fault.
I was lucky in that i have never bothered to use it despite having released one of the highest number of mods of any modder. My approach has been to use the basic and well maintained version of Lua, code well and not let my mods get overcomplex.

It's no surprise some feel they need LuaJIT if they write or use a nightmare mod like Moretrees, which is insanely and unnecessarily overintensive and unfortunately heavily promoted and present on almost all servers.
There are many other nightmare mods which are poorly coded and overintensive, and which are very common on servers, and servers tend to add hundreds of mods. The real issue on servers is actually bad mods and too many of them, for which people try to compensate for by using LuaJIT.

The issue is that people have got used to something that has turned out to be a mess. The best thing to do is to try to move away from it instead of keeping the mess and trying to patch it up with stuff we wouldn't normally use, this will just lead to more mess and pain in future.

FFI is the only option to escape the memory limits of LuaJIT

There's GC64 and maybe a version of LuaJIT will appear that is better.
I can eat my cake and have it in my stomach :)

It matters little how well or bad maintained LuaJIT is, because it's the only option speed wise for some people,

That makes no sense, it's critical how well LuaJIT is maintained, that is the problem here.

paramat picture paramat on 5 May 2018

@paramat

No. It shouldn't be portrayed as essential, often it is used as compensation for overcomplex and poorly coded mods.

Often, and often not. Some tasks just require the bigger speed; you wouldn't do numeric crunching in pure Lua, for example. Server owners rely on its speed in order to run more mods simultaneously, and they will, because they can, so it's essential for them.

Besides, you can't tell modders "look how easy Lua is to write mods, anyone can do it" and at the same time "But you need to be an expert in order to avoid bottlenecks and use state-of-the-art algorithms for them to run fast".

My approach has been to use the basic and well maintained version of Lua, code well and not let my mods get overcomplex.

Your approach is that of a CSist, which most modders aren't.

The real issue on servers is actually bad mods and too many of them, for which people try to compensate for by using LuaJIT.

Yes, that's the real issue. It's how reality works. Real people rely on real software to do real tasks, without looking into whether it will bite them in the ass in future. The solution to this would have been to not allow LuaJIT at all since the very beginning, but that ship sailed very long ago, and now it's what we have. On the other hand, many of the servers currently existing, do exist because of it, and they would have been much duller without it.

The best thing to do is to try to move away from it instead of keeping the mess and trying to patch it up with stuff we wouldn't normally use, this will just lead to more mess and pain in future.

That approach is not feasible. Backing up on LuaJIT support now, would cause a chaos. It's like telling people not to use their cellphones any longer because they are a mess. People would probably continue to use them because they have become an important part of their lives, and only a low percentage would obey and switch to dropping their cell and using only fixed phones.

There's GC64 and maybe a version of LuaJIT will appear that is better.
I can eat my cake and have it in my stomach :)

OK; I misunderstood what GC64 accomplished. Yes, promoting that would be a solution. How about embedding a LuaJIT which has GC64 support in MT, then?

As for "maybe a version will appear that is better", yes, maybe... but servers are crashing as we speak.

That makes no sense, it's critical how well LuaJIT is maintained, that is the problem here.

No it is not. Even If LuaJIT maintenance was dropped two years ago, it would still be used because it's essential for some.

Anyway, the last LuaJIT commit is from April, so development is slow (rings any bells?) but not stalled.

ghost picture ghost on 6 May 2018

What is everyone's CPU model? Especially those that get crashes < 1GB of usage

rubenwardy picture rubenwardy on 4 Jul 2018

Oh erm, some of the statements in my last comment were overly harsh, sorry. I don't blame people for using LuaJIT and the situation we're in is unfortunate.

paramat picture paramat on 4 Jul 2018

The solution seems simple to me: bundle LuaJIT with Minetest, configured to build with GC64 enabled. Sure, it can go out of sync with upstream, so update it from time to time.

VanessaE picture VanessaE on 4 Jul 2018

The issue is that gc64 is experimental

rubenwardy picture rubenwardy on 4 Jul 2018

So mark it as experimental, problem solved, people will also test it extensively.

Fixer-007 picture Fixer-007 on 5 Jul 2018

Some updated information for people reading this thread:
Since 5.0, the official 64-bit Windows releases are compiled with a LuaJIT build with GC64 enabled.
I haven't had any bug reports (even with previous testing on a smaller scale), so I can say with confidence that GC64 works perfectly fine on Windows.

If you're running Linux or anything else, it is also definitely worth giving LuaJIT 2.1.0-beta3 a try (gc64 needs to be enabled manually!) and see if it solves the issue.

sfan5 picture sfan5 on 6 Apr 2020
❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

tenplus1 picture tenplus1  ·  188Comments
paramat picture paramat  ·  160Comments
nerzhul picture nerzhul  ·  57Comments
rubenwardy picture rubenwardy  ·  53Comments
tacotexmex picture tacotexmex  ·  83Comments