Rclcpp: Segmentation fault on wait_for_work

Created on 13 Nov 2020  路  14Comments  路  Source: ros2/rclcpp

Bug report

As a workaround for https://github.com/ros2/rclcpp/issues/1454 I tried to create a utility to avoid a hard coded sleep, that blocks until the spin() is active and a callback has been called, before returning.
It causes a segmentation fault very unfrequently.

Required Info:

  • Operating System: Ubuntu 20.04
  • Installation type: Binaries
  • Version or commit hash: 5.0.0-1focal.20201007.211227
  • DDS implementation: Fast DDS
  • Client library (if applicable): rclcpp

Steps to reproduce issue

std::future<void> spin_executor_async(std::shared_ptr<rclcpp::Node> node, rclcpp::Executor &executor)
{
  // We create a timer that will be called as part of the spin
  // this function won't return until the executor is spinning and the timer is called
  std::atomic<bool> timer_called{false};
  std::function<void()> timer_called_lambda = [&](){std::cout << " timer called " << std::endl; timer_called = true;};
  auto timer = node->create_wall_timer(std::chrono::nanoseconds(1), timer_called_lambda);
  executor.add_node(node);
  auto executor_spin_future = std::async(
    std::launch::async, [&executor]() -> void {
      executor.spin();
    });

  while (!timer_called)
  {
    std::this_thread::sleep_for(1ns);
  }
  timer.reset();
  executor.remove_node(node);
  return executor_spin_future;
}

TEST(Foo, bar2) {
  auto node = std::make_shared<rclcpp::Node>("test");
  auto executor = std::make_shared<rclcpp::executors::SingleThreadedExecutor>();

  auto executor_spin_future = spin_executor_async(node, *executor);
  executor->cancel();
  return;
}

Expected behavior

That the test doesn't hang, as a workaround for #1454

Actual behavior

Doesn't hang, but very unfrequently 1% of runs it segfaults.
Because the thread is stuck on executor->spin() (wait_for_work exactly)

Additional information

The segfault happens on thread 2 (the executor spin thread), here's the callstack:

Thread 2 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 9204.9221]
0x00007fc84fcd007b in rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator<void> >::collect_entities(std::map<std::weak_ptr<rclcpp::CallbackGroup>, std::weak_ptr<rclcpp::node_interfaces::NodeBaseInterface>, std::owner_less<std::weak_ptr<rclcpp::CallbackGroup> >, std::allocator<std::pair<std::weak_ptr<rclcpp::CallbackGroup> const, std::weak_ptr<rclcpp::node_interfaces::NodeBaseInterface> > > > const&) () from /opt/ros/rolling/lib/librclcpp.so
(rr) bt
#0  0x00007fc84fcd007b in rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy<std::allocator<void> >::collect_entities(std::map<std::weak_ptr<rclcpp::CallbackGroup>, std::weak_ptr<rclcpp::node_interfaces::NodeBaseInterface>, std::owner_less<std::weak_ptr<rclcpp::CallbackGroup> >, std::allocator<std::pair<std::weak_ptr<rclcpp::CallbackGroup> const, std::weak_ptr<rclcpp::node_interfaces::NodeBaseInterface> > > > const&) () from /opt/ros/rolling/lib/librclcpp.so
#1  0x00007fc84fcb93b6 in rclcpp::Executor::wait_for_work(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /opt/ros/rolling/lib/librclcpp.so
#2  0x00007fc84fcb9da3 in rclcpp::Executor::get_next_executable(rclcpp::AnyExecutable&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >) () from /opt/ros/rolling/lib/librclcpp.so
#3  0x00007fc84fcbf00d in rclcpp::executors::SingleThreadedExecutor::spin() () from /opt/ros/rolling/lib/librclcpp.so
#4  0x0000563c231efd7d in <lambda()>::operator()(void) const (__closure=0x563c23c088e8) at /root/control/src/ros2_control/controller_manager/test/test_controller_manager.cpp:143
#5  0x0000563c231f2e13 in std::__invoke_impl<void, spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...)
    at /usr/include/c++/9/bits/invoke.h:60
#6  0x0000563c231f2db4 in std::__invoke<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#7  0x0000563c231f2d44 in std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x563c23c088e8)
    at /usr/include/c++/9/thread:244
#8  0x0000563c231f2cd1 in std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >::operator()(void) (this=0x563c23c088e8)
    at /usr/include/c++/9/thread:251
#9  0x0000563c231f2a74 in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void>::operator()(void) const (this=0x7fc8496485d0) at /usr/include/c++/9/future:1362
#10 0x0000563c231f2777 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/9/bits/std_function.h:286
#11 0x0000563c231f65d0 in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (this=0x7fc8496485d0)
    at /usr/include/c++/9/bits/std_function.h:688
#12 0x0000563c231f4ab2 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (
    this=0x563c23c088b0, __f=0x7fc8496485d0, __did_set=0x7fc84964852f) at /usr/include/c++/9/future:561
#13 0x0000563c231fcaf5 in std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__f=
    @0x7fc849648570: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x563c231f4a78 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, 
    __t=@0x7fc849648540: 0x563c23c088b0) at /usr/include/c++/9/bits/invoke.h:73
#14 0x0000563c231f956e in std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__fn=
    @0x7fc849648570: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x563c231f4a78 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>)
    at /usr/include/c++/9/bits/invoke.h:95
#15 0x0000563c231f6384 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (this=0x7fc8496484c0) at /usr/include/c++/9/mutex:671
#16 0x0000563c231f63b3 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (this=0x0) at /usr/include/c++/9/mutex:676
#17 0x0000563c231f63c8 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() () at /usr/include/c++/9/mutex:676
#18 0x00007fc84fbbc47f in __pthread_once_slow (once_control=0x563c23c088c8, init_routine=0x7fc84f771b80 <__once_proxy>) at pthread_once.c:116
#19 0x0000563c231ed57a in __gthread_once (__once=0x563c23c088c8, __func=0x7fc84f771b80 <__once_proxy>) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:700
#20 0x0000563c231f6472 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::--Type <RET> for more, q to quit, c to continue without paging--c
*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__once=..., __f=@0x7fc849648570: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x563c231f4a78 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>) at /usr/include/c++/9/mutex:683
#21 0x0000563c231f4835 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x563c23c088b0, __res=..., __ignore_failure=false) at /usr/include/c++/9/future:401
#22 0x0000563c231f21ee in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void>::<lambda()>::operator()(void) const (this=0x563c23c088b0) at /usr/include/c++/9/future:1662
#23 0x0000563c231f397b in std::__invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> >(std::__invoke_other, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void>::<lambda()> &&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#24 0x0000563c231f391c in std::__invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> >(std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void>::<lambda()> &&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#25 0x0000563c231f385e in std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x563c23c08da8) at /usr/include/c++/9/thread:244
#26 0x0000563c231f379f in std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> > >::operator()(void) (this=0x563c23c08da8) at /usr/include/c++/9/thread:251
#27 0x0000563c231f32c6 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> > > >::_M_run(void) (this=0x563c23c08da0) at /usr/include/c++/9/thread:195
#28 0x00007fc84f772d84 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#29 0x00007fc84fbb3609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#30 0x00007fc84f5af293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

The callstack for thread 1:

#0  __lll_lock_wait (futex=futex@entry=0x563c23b17598, private=0) at lowlevellock.c:52
#1  0x00007fc84fbb60a3 in __GI___pthread_mutex_lock (mutex=0x563c23b17598) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007fc84fcba3a1 in rclcpp::Executor::remove_callback_group_from_map(std::shared_ptr<rclcpp::CallbackGroup>, std::map<std::weak_ptr<rclcpp::CallbackGroup>, std::weak_ptr<rclcpp::node_interfaces::NodeBaseInterface>, std::owner_less<std::weak_ptr<rclcpp::CallbackGroup> >, std::allocator<std::pair<std::weak_ptr<rclcpp::CallbackGroup> const, std::weak_ptr<rclcpp::node_interfaces::NodeBaseInterface> > > >&, bool) ()
   from /opt/ros/rolling/lib/librclcpp.so
#3  0x00007fc84fcb8f4d in rclcpp::Executor::remove_node(std::shared_ptr<rclcpp::node_interfaces::NodeBaseInterface>, bool) () from /opt/ros/rolling/lib/librclcpp.so
#4  0x00007fc84fcb42e4 in rclcpp::Executor::remove_node(std::shared_ptr<rclcpp::Node>, bool) () from /opt/ros/rolling/lib/librclcpp.so
#5  0x0000563c231eff4c in spin_executor_async (node=std::shared_ptr<class rclcpp::Node> (use count 3, weak count 1) = {...}, executor=...)
    at /root/control/src/ros2_control/controller_manager/test/test_controller_manager.cpp:151
#6  0x0000563c231f00ad in TestControllerManager_controller_parameters_Test::TestBody (this=0x563c23d26bd0) at /root/control/src/ros2_control/controller_manager/test/test_controller_manager.cpp:177
#7  0x0000563c23241c34 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x563c23d26bd0, method=&virtual testing::Test::TestBody(), 
    location=0x563c2326d73b "the test body") at /opt/ros/rolling/src/gtest_vendor/src/gtest.cc:2447
#8  0x0000563c2323a9f9 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x563c23d26bd0, method=&virtual testing::Test::TestBody(), location=0x563c2326d73b "the test body")
    at /opt/ros/rolling/src/gtest_vendor/src/gtest.cc:2483
#9  0x0000563c23216646 in testing::Test::Run (this=0x563c23d26bd0) at /opt/ros/rolling/src/gtest_vendor/src/gtest.cc:2522
#10 0x0000563c2321703f in testing::TestInfo::Run (this=0x563c238d4f30) at /opt/ros/rolling/src/gtest_vendor/src/gtest.cc:2703
#11 0x0000563c23217730 in testing::TestCase::Run (this=0x563c238d4db0) at /opt/ros/rolling/src/gtest_vendor/src/gtest.cc:2825
#12 0x0000563c23222e19 in testing::internal::UnitTestImpl::RunAllTests (this=0x563c238d4a40) at /opt/ros/rolling/src/gtest_vendor/src/gtest.cc:5216
#13 0x0000563c2324312e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x563c238d4a40, 
    method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x563c23222b70 <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x563c2326e140 "auxiliary test code (environments or event listeners)") at /opt/ros/rolling/src/gtest_vendor/src/gtest.cc:2447
#14 0x0000563c2323bb0d in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x563c238d4a40, 
    method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x563c23222b70 <testing::internal::UnitTestImpl::RunAllTests()>, 
    location=0x563c2326e140 "auxiliary test code (environments or event listeners)") at /opt/ros/rolling/src/gtest_vendor/src/gtest.cc:2483
#15 0x0000563c232217f8 in testing::UnitTest::Run (this=0x563c232b3520 <testing::UnitTest::GetInstance()::instance>) at /opt/ros/rolling/src/gtest_vendor/src/gtest.cc:4824
#16 0x0000563c2320e295 in RUN_ALL_TESTS () at /opt/ros/rolling/src/gtest_vendor/include/gtest/gtest.h:2370
#17 0x0000563c2320e217 in main (argc=1, argv=0x7fff2d24dfa8) at /opt/ros/rolling/src/gmock_vendor/src/gmock_main.cc:53
bug
Source
picture v-lopez

All 14 comments

My guess is that removing the timer while spinning (timer.reset();) might not be thread safe, but I'm not sure if that's the issue.

@v-lopez could you try building with debug info to get more information in the backtrace? (i.e., line numbers), thanks!

ivanpauno picture ivanpauno on 13 Nov 2020

Adding debug callstack from rclcpp master (dd0f97f1790de0384caeeb94a7c1c49e4b9f08ed):

Thread 2:

Thread 2 received signal SIGSEGV, Segmentation fault.
[Switching to Thread 18773.18784]
0x00007f15ee02fb12 in rclcpp::Executor::add_callback_groups_from_nodes_associated_to_executor (this=0x55cc2bc42600) at /root/rclcpp/src/ros2/rclcpp/rclcpp/src/rclcpp/executor.cpp:175
175       auto group_ptrs = node->get_callback_groups();
(rr) bt
#0  0x00007f15ee02fb12 in rclcpp::Executor::add_callback_groups_from_nodes_associated_to_executor (this=0x55cc2bc42600) at /root/rclcpp/src/ros2/rclcpp/rclcpp/src/rclcpp/executor.cpp:175
#1  0x00007f15ee033bae in rclcpp::Executor::wait_for_work (this=0x55cc2bc42600, timeout=...) at /root/rclcpp/src/ros2/rclcpp/rclcpp/src/rclcpp/executor.cpp:664
#2  0x00007f15ee034d52 in rclcpp::Executor::get_next_executable (this=0x55cc2bc42600, any_executable=..., timeout=...) at /root/rclcpp/src/ros2/rclcpp/rclcpp/src/rclcpp/executor.cpp:871
#3  0x00007f15ee045459 in rclcpp::executors::SingleThreadedExecutor::spin (this=0x55cc2bc42600) at /root/rclcpp/src/ros2/rclcpp/rclcpp/src/rclcpp/executors/single_threaded_executor.cpp:35
#4  0x000055cc2b5e9f6b in <lambda()>::operator()(void) const (__closure=0x55cc2bc424f8) at /root/rclcpp/src/ros2/rclcpp/rclcpp/test/rclcpp/test_duration.cpp:278
#5  0x000055cc2b5eca27 in std::__invoke_impl<void, spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> >(std::__invoke_other, <lambda()> &&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#6  0x000055cc2b5ec9c8 in std::__invoke<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> >(<lambda()> &&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#7  0x000055cc2b5ec958 in std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x55cc2bc424f8) at /usr/include/c++/9/thread:244
#8  0x000055cc2b5ec8e5 in std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >::operator()(void) (this=0x55cc2bc424f8) at /usr/include/c++/9/thread:251
#9  0x000055cc2b5ec688 in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void>::operator()(void) const (this=0x7f15e4c4b5d0) at /usr/include/c++/9/future:1362
#10 0x000055cc2b5ec38b in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/9/bits/std_function.h:286
#11 0x000055cc2b5f06d4 in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (this=0x7f15e4c4b5d0) at /usr/include/c++/9/bits/std_function.h:688
#12 0x000055cc2b5ef0be in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (this=0x55cc2bc424c0, __f=0x7f15e4c4b5d0, __did_set=0x7f15e4c4b52f)
    at /usr/include/c++/9/future:561
#13 0x000055cc2b5f569d in std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__f=
    @0x7f15e4c4b570: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x55cc2b5ef084 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __t=@0x7f15e4c4b540: 0x55cc2bc424c0) at /usr/include/c++/9/bits/invoke.h:73
#14 0x000055cc2b5f32ae in std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__fn=
    @0x7f15e4c4b570: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x55cc2b5ef084 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>) at /usr/include/c++/9/bits/invoke.h:95
#15 0x000055cc2b5f0488 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (this=0x7f15e4c4b4c0) at /usr/include/c++/9/mutex:671
#16 0x000055cc2b5f04b7 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (this=0x0) at /usr/include/c++/9/mutex:676
#17 0x000055cc2b5f04cc in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() () at /usr/include/c++/9/mutex:676
#18 0x00007f15eda9547f in __pthread_once_slow (once_control=0x55cc2bc424d8, init_routine=0x7f15ed910b80 <__once_proxy>) at pthread_once.c:116
#19 0x000055cc2b5e344a in __gthread_once (__once=0x55cc2bc424d8, __func=0x7f15ed910b80 <__once_proxy>) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:700
#20 0x000055cc2b5f0576 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__once=..., __f=
    @0x7f15e4c4b570: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x55cc2b5ef084 <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>) at /usr/include/c++/9/mutex:683
#21 0x000055cc2b5eee41 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (this=0x55cc2bc424c0, __res=..., __ignore_failure=false)
    at /usr/include/c++/9/future:401
#22 0x000055cc2b5ebe02 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void>::<lambda()>::operator()(void) const (this=0x55cc2bc424c0)
    at /usr/include/c++/9/future:1662
#23 0x000055cc2b5edf2b in std::__invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> >(std::__invoke_other, std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void>::<lambda()> &&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#24 0x000055cc2b5edecc in std::__invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> >(std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >, void>::<lambda()> &&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#25 0x000055cc2b5ede0e in std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x55cc2bc42af8) at /usr/include/c++/9/thread:244
#26 0x000055cc2b5edd4f in std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> > >::operator()(void) (this=0x55cc2bc42af8) at /usr/include/c++/9/thread:251
#27 0x000055cc2b5ed876 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<spin_executor_async(std::shared_ptr<rclcpp::Node>, rclcpp::Executor&)::<lambda()> > >; _Res = void]::<lambda()> > > >::_M_run(void) (this=0x55cc2bc42af0) at /usr/include/c++/9/thread:195
#28 0x00007f15ed911d84 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#29 0x00007f15eda8c609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#30 0x00007f15ed750293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1

[Switching to thread 1 (Thread 18773.18773)]
#0  __lll_lock_wait (futex=futex@entry=0x55cc2bc42688, private=0) at lowlevellock.c:52
52  lowlevellock.c: No such file or directory.
(rr) bt
#0  __lll_lock_wait (futex=futex@entry=0x55cc2bc42688, private=0) at lowlevellock.c:52
#1  0x00007f15eda8f0a3 in __GI___pthread_mutex_lock (mutex=0x55cc2bc42688) at ../nptl/pthread_mutex_lock.c:80
#2  0x00007f15ee0077b4 in __gthread_mutex_lock (__mutex=0x55cc2bc42688) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:749
#3  0x00007f15ee007884 in std::mutex::lock (this=0x55cc2bc42688) at /usr/include/c++/9/bits/std_mutex.h:100
#4  0x00007f15ee01c1a5 in std::unique_lock<std::mutex>::lock (this=0x7ffdbf4e9990) at /usr/include/c++/9/bits/unique_lock.h:141
#5  0x00007f15ee01a2b1 in std::unique_lock<std::mutex>::unique_lock (this=0x7ffdbf4e9990, __m=...) at /usr/include/c++/9/bits/unique_lock.h:71
#6  0x00007f15ee030a9c in rclcpp::Executor::remove_callback_group_from_map (this=0x55cc2bc42600, group_ptr=std::shared_ptr<class rclcpp::CallbackGroup> (use count 3, weak count 2) = {...}, Traceback (most recent call last):
  File "/lib/x86_64-linux-gnu/../../share/gcc-10/python/libstdcxx/v6/printers.py", line 714, in children
    node = lookup_node_type('_Rb_tree_node', self.val.type).pointer()
AttributeError: 'NoneType' object has no attribute 'pointer'
weak_groups_to_nodes=
std::map with 0 elements, notify=true) at /root/rclcpp/src/ros2/rclcpp/rclcpp/src/rclcpp/executor.cpp:303
#7  0x00007f15ee031106 in rclcpp::Executor::remove_node (this=0x55cc2bc42600, node_ptr=std::shared_ptr<class rclcpp::node_interfaces::NodeBaseInterface> (use count 7, weak count 2) = {...}, notify=true) at /root/rclcpp/src/ros2/rclcpp/rclcpp/src/rclcpp/executor.cpp:357
#8  0x00007f15ee03127c in rclcpp::Executor::remove_node (this=0x55cc2bc42600, node_ptr=std::shared_ptr<class rclcpp::Node> (use count 3, weak count 1) = {...}, notify=true) at /root/rclcpp/src/ros2/rclcpp/rclcpp/src/rclcpp/executor.cpp:371
#9  0x000055cc2b5ea13a in spin_executor_async (node=std::shared_ptr<class rclcpp::Node> (use count 3, weak count 1) = {...}, executor=...) at /root/rclcpp/src/ros2/rclcpp/rclcpp/test/rclcpp/test_duration.cpp:286
#10 0x000055cc2b5ea310 in Foo_bar2_Test::TestBody (this=0x55cc2ba18ac0) at /root/rclcpp/src/ros2/rclcpp/rclcpp/test/rclcpp/test_duration.cpp:295
#11 0x000055cc2b632830 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (object=0x55cc2ba18ac0, method=&virtual testing::Test::TestBody(), location=0x55cc2b649a83 "the test body")
    at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2433
#12 0x000055cc2b62b431 in testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=0x55cc2ba18ac0, method=&virtual testing::Test::TestBody(), location=0x55cc2b649a83 "the test body")
    at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2469
#13 0x000055cc2b606524 in testing::Test::Run (this=0x55cc2ba18ac0) at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2508
#14 0x000055cc2b606f1d in testing::TestInfo::Run (this=0x55cc2ba17a60) at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2684
#15 0x000055cc2b607662 in testing::TestSuite::Run (this=0x55cc2ba17bd0) at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2816
#16 0x000055cc2b613748 in testing::internal::UnitTestImpl::RunAllTests (this=0x55cc2ba165d0) at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5338
#17 0x000055cc2b633d5e in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55cc2ba165d0, 
    method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x55cc2b613340 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55cc2b64a4c0 "auxiliary test code (environments or event listeners)")
    at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2433
#18 0x000055cc2b62c66f in testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x55cc2ba165d0, 
    method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x55cc2b613340 <testing::internal::UnitTestImpl::RunAllTests()>, location=0x55cc2b64a4c0 "auxiliary test code (environments or event listeners)")
    at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2469
#19 0x000055cc2b611f64 in testing::UnitTest::Run (this=0x55cc2b67c560 <testing::UnitTest::GetInstance()::instance>) at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4925
#20 0x000055cc2b5fdfa4 in RUN_ALL_TESTS () at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2473
#21 0x000055cc2b5fdf26 in main (argc=1, argv=0x7ffdbf4ea218) at /root/rclcpp/install/gtest_vendor/src/gtest_vendor/src/gtest_main.cc:45
v-lopez picture v-lopez on 16 Nov 2020
👍1

Thanks for the additional information @v-lopez !

This seems to have been introduced in https://github.com/ros2/rclcpp/pull/1218.
It might be the same issue than in https://github.com/ros2/rclcpp/issues/1402, or a different one.

@wjwwood FYI

ivanpauno picture ivanpauno on 16 Nov 2020

I'll have a look ASAP. Thanks @ivanpauno

wjwwood picture wjwwood on 18 Nov 2020

according to the stack trace, I think this is because weak_nodes_ is not protected by mutex.

fujitatomoya picture fujitatomoya on 18 Nov 2020

1402 has been resolved. But I think this issue is different. Does the issue still occur with the most recent commits?

brawner picture brawner on 2 Dec 2020

Yes it does still crash as of ea0ee50318a9609843cd7295c4542c42a7ed339c

v-lopez picture v-lopez on 2 Dec 2020

@v-lopez I would expect https://github.com/ros2/rclcpp/pull/1505 to fix the issue, could you give it a try?

ivanpauno picture ivanpauno on 8 Jan 2021

Before I would get a crash in a few seconds of attempts, now it took over a minute but I still got a segfault.

Running it with valgrind to see if I can catch it.

v-lopez picture v-lopez on 8 Jan 2021
👍1

@v-lopez I tried your example locally and I think that after last #1505 commit I got it right.
(I'm still going to run the reproducer for some minutes more, as in my system it was hard to make the example segfault)

ivanpauno picture ivanpauno on 8 Jan 2021

Without these latest changes, it took me 4h of gdb to catch a crash, but I'm attaching the callstack.

callstack.txt

I'm running now with the latest changes and so far no crashes. I'd consider it fixed.

v-lopez picture v-lopez on 8 Jan 2021
❤1

I'm running now with the latest changes and so far no crashes. I'd consider it fixed.

I'm confident that it's fixed now as that callstack is the same I got before the latest commit (segmentation fault while copy constructing), so we only need to wait for a review of #1505 :tada: .

ivanpauno picture ivanpauno on 11 Jan 2021

@ivanpauno With #1505 now merged, can we close this?

clalancette picture clalancette on 19 Jan 2021
👍1

@ivanpauno With #1505 now merged, can we close this?

Yes, closing

ivanpauno picture ivanpauno on 19 Jan 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

LouisChen1905 picture LouisChen1905  路  14Comments
Karsten1987 picture Karsten1987  路  10Comments
Myzhar picture Myzhar  路  8Comments
tylerjw picture tylerjw  路  8Comments
schornakj picture schornakj  路  4Comments