Rclcpp: lock-order-inversion (potential deadlock) in rclcpp::Client::wait_for_service, detected by ThreadSanitizer

[yuriy-universal-ivanov]

Bug report

Required Info:

• Operating System:
  
  
  
  • Ubuntu 18.04
  
  
• Installation type:
  
  
  
  • Binaries, with apt-get
  
  
• Version or commit hash:
  
  
  
  • eloquent, from deb http://packages.ros.org/ros2/ubuntu bionic main
  
  
• DDS implementation:
  
  
  
  • Default (Fast-RTPS)
  
  
• Client library (if applicable):
  
  
  
  • rclcpp
  
  

Steps to reproduce issue

Reproduced when using rclcpp::executors::MultiThreadedExecutor::spin() in one thread and a rclcpp::Client waiting for a service in our own worker thread.

Expected behavior

Mutexes are always locked in the same order in rclcpp to avoid deadlocks.

Actual behavior

Some mutexes are locked in different order:
1) When rclcpp::graph_listener::GraphListener::run_loop() -> rclcpp::node_interfaces::NodeGraph::notify_graph_change()
2) When rclcpp::node_interfaces::NodeGraph::get_graph_event() -> undetected by TSan call in rclcpp: <null> <null> (librclcpp.so+0x9fb1c)

Additional information

In the worker thread:

    template <typename ResponseMessageType>
    ResponseMessageType FutureResponse<ResponseMessageType>::Impl::request(std::shared_ptr<RclcppClient> rclcpp_client,
                                                                           std::vector<uint8_t> request_message)
    {
      constexpr std::chrono::milliseconds single_spin_limit{100};

      while (!stop && rclcpp::ok() && !rclcpp_client->wait_for_service(single_spin_limit))
      {
      }

where rclcpp_client is std::shared_ptr<rclcpp::Client<our service type>> and stop is an std::atomic<bool>.

ThreadSanitizer detects a lock order inversion, which is a real threat of a deadlock:

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=9849)
  Cycle in lock order graph: M470198622835870888 (0x000000000000) => M471324660181503496 (0x000000000000) => M470198622835870888

  Mutex M471324660181503496 acquired here while holding mutex M470198622835870888 in thread T159:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x3fadb)
    #1 <null> <null> (librclcpp.so+0x9fb1c)
    #2 our_project::FutureResponse<our_project::BinarySerializable<void> >::Impl::request(std::shared_ptr<rclcpp::Client<component_wrapper_ros2_interfaces::srv::BinaryBuffers> >, std::vector<unsigned char, std::allocator<unsigned char> >) /our_project/components/component-wrapper/include/./component-wrapper/future_response.h:124 (OurProjectComponentWrapper_test+0x5d82e)
    #3 our_project::FutureResponse<our_project::BinarySerializable<void> >::Impl::Impl(std::shared_ptr<rclcpp::Client<component_wrapper_ros2_interfaces::srv::BinaryBuffers> >, std::vector<unsigned char, std::allocator<unsigned char> >)::{lambda()#1}::operator()() const /our_project/components/component-wrapper/include/./component-wrapper/future_response.h:106 (OurProjectComponentWrapper_test+0x5b31c)
<...>

  Mutex M470198622835870888 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x3fadb)
    #1 rclcpp::node_interfaces::NodeGraph::get_graph_event() <null> (librclcpp.so+0xc1d38)
    #2 our_project::FutureResponse<our_project::BinarySerializable<void> >::Impl::request(std::shared_ptr<rclcpp::Client<component_wrapper_ros2_interfaces::srv::BinaryBuffers> >, std::vector<unsigned char, std::allocator<unsigned char> >) /our_project/components/component-wrapper/include/./component-wrapper/future_response.h:124 (OurProjectComponentWrapper_test+0x5d82e)
<...>

  Mutex M470198622835870888 acquired here while holding mutex M471324660181503496 in thread T161:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x3fadb)
    #1 rclcpp::node_interfaces::NodeGraph::notify_graph_change() <null> (librclcpp.so+0xc067a)

  Mutex M471324660181503496 previously acquired by the same thread here:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x3fadb)
    #1 rclcpp::graph_listener::GraphListener::run_loop() <null> (librclcpp.so+0x9ffa3)

future_response.h:124 contains !rclcpp_client->wait_for_service(single_spin_limit)

[ivanpauno]

@yuriy-universal-ivanov can you share a complete example with reproduction steps (including how to get the tsan issue)?

From the available information, it's not obvious what's the problem.

[yuriy-universal-ivanov]

Sure.

rclcpp_1121_repro.zip

The archive contains two files: rclcpp_1121_repro.cpp and CMakeLists.txt to build it with CMake (it enables TSan). Target platform to test at - Linux (I tested at Ubuntu 18.04 with ROS2 eloquent). ROS2 should be sourced prior to invocation of CMake build so it succeeds. TSan from GCC 7+ or Clang 6+ is required to work correctly in modern Linux kernels.

When built (Debug configuration is recommended for human readable symbols in TSan report), it produces binary executable rclcpp_1121_repro. Execute it. On finishing, TSan reports from 1 to 4 warnings. Among them there is always the one this bug is about (lock-order-inversion), the rest depend on whether you're lucky or not: they are race conditions on destroying rclcpp service and client without stopping multithreaded executors, and it's a separate problem, which may have been already fixed. I checked the history of changes of appropriate code in ROS2, so I'm waiting for Foxy to be released to check it - if it's not fixed there still, I'll also report another bug. This one is only about the lock-order-inversion.

The report will look like:

==================
WARNING: ThreadSanitizer: data race (pid=19756)
  Atomic read of size 1 at 0x7b0c00003270 by thread T7 (mutexes: write M826545943351435664):
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x3fadb)
    #1 eprosima::fastrtps::rtps::PDP::assert_remote_participant_liveliness(eprosima::fastrtps::rtps::GuidPrefix_t const&) <null> (libfastrtps.so.1+0x33a27a)

  Previous write of size 8 at 0x7b0c00003270 by thread T3:
    #0 operator new(unsigned long) <null> (libtsan.so.0+0x73dda)
    #1 eprosima::fastrtps::rtps::PDP::PDP(eprosima::fastrtps::rtps::BuiltinProtocols*, eprosima::fastrtps::rtps::RTPSParticipantAllocationAttributes const&) <null> (libfastrtps.so.1+0x33ff1a)
    #2 void std::allocator_traits<std::allocator<rclcpp::Node> >::construct<rclcpp::Node, char const (&) [30]>(std::allocator<rclcpp::Node>&, rclcpp::Node*, char const (&) [30]) /usr/include/c++/7/bits/alloc_traits.h:475 (rclcpp_1121_repro+0x309e7)
    #3 std::_Sp_counted_ptr_inplace<rclcpp::Node, std::allocator<rclcpp::Node>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [30]>(std::allocator<rclcpp::Node>, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:526 (rclcpp_1121_repro+0x2f86c)
    #4 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, rclcpp::Node*, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:637 (rclcpp_1121_repro+0x2dd38)
    #5 std::__shared_ptr<rclcpp::Node, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:1295 (rclcpp_1121_repro+0x2bce9)
    #6 std::shared_ptr<rclcpp::Node>::shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:344 (rclcpp_1121_repro+0x2a1da)
    #7 std::shared_ptr<rclcpp::Node> std::allocate_shared<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:691 (rclcpp_1121_repro+0x28b18)
    #8 std::shared_ptr<rclcpp::Node> std::make_shared<rclcpp::Node, char const (&) [30]>(char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:707 (rclcpp_1121_repro+0x2617e)
    #9 std::shared_ptr<rclcpp::Node> rclcpp::Node::make_shared<char const (&) [30]>(char const (&) [30]) /opt/ros/eloquent/include/rclcpp/node.hpp:75 (rclcpp_1121_repro+0x23ca9)
    #10 request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::atomic<bool>&) /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:46 (rclcpp_1121_repro+0xa4c6)
    #11 operator() /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:114 (rclcpp_1121_repro+0xa99c)
    #12 __invoke_impl<void, main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0xd419)
    #13 __invoke<main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0xbfcb)
    #14 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1b82c)
    #15 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1b5f2)
    #16 operator() /usr/include/c++/7/future:1362 (rclcpp_1121_repro+0x1aea1)
    #17 _M_invoke /usr/include/c++/7/bits/std_function.h:302 (rclcpp_1121_repro+0x19935)
    #18 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706 (rclcpp_1121_repro+0x22340)
    #19 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561 (rclcpp_1121_repro+0x20594)
    #20 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73 (rclcpp_1121_repro+0x2787a)
    #21 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x24844)
    #22 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672 (rclcpp_1121_repro+0x21e31)
    #23 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21ed6)
    #24 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21f16)
    #25 pthread_once <null> (libtsan.so.0+0x2dc39)
    #26 __gthread_once /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:699 (rclcpp_1121_repro+0x9826)
    #27 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/mutex:684 (rclcpp_1121_repro+0x2205e)
    #28 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/7/future:401 (rclcpp_1121_repro+0x2023f)
    #29 operator() /usr/include/c++/7/future:1667 (rclcpp_1121_repro+0x174e1)
    #30 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0x19c5f)
    #31 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x184c9)
    #32 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1e58e)
    #33 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1df0c)
    #34 _M_run /usr/include/c++/7/thread:186 (rclcpp_1121_repro+0x1c2a0)
    #35 <null> <null> (libstdc++.so.6+0xbd6de)

  Location is heap block of size 40 at 0x7b0c00003270 allocated by thread T3:
    #0 operator new(unsigned long) <null> (libtsan.so.0+0x73dda)
    #1 eprosima::fastrtps::rtps::PDP::PDP(eprosima::fastrtps::rtps::BuiltinProtocols*, eprosima::fastrtps::rtps::RTPSParticipantAllocationAttributes const&) <null> (libfastrtps.so.1+0x33ff1a)
    #2 void std::allocator_traits<std::allocator<rclcpp::Node> >::construct<rclcpp::Node, char const (&) [30]>(std::allocator<rclcpp::Node>&, rclcpp::Node*, char const (&) [30]) /usr/include/c++/7/bits/alloc_traits.h:475 (rclcpp_1121_repro+0x309e7)
    #3 std::_Sp_counted_ptr_inplace<rclcpp::Node, std::allocator<rclcpp::Node>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [30]>(std::allocator<rclcpp::Node>, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:526 (rclcpp_1121_repro+0x2f86c)
    #4 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, rclcpp::Node*, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:637 (rclcpp_1121_repro+0x2dd38)
    #5 std::__shared_ptr<rclcpp::Node, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:1295 (rclcpp_1121_repro+0x2bce9)
    #6 std::shared_ptr<rclcpp::Node>::shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:344 (rclcpp_1121_repro+0x2a1da)
    #7 std::shared_ptr<rclcpp::Node> std::allocate_shared<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:691 (rclcpp_1121_repro+0x28b18)
    #8 std::shared_ptr<rclcpp::Node> std::make_shared<rclcpp::Node, char const (&) [30]>(char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:707 (rclcpp_1121_repro+0x2617e)
    #9 std::shared_ptr<rclcpp::Node> rclcpp::Node::make_shared<char const (&) [30]>(char const (&) [30]) /opt/ros/eloquent/include/rclcpp/node.hpp:75 (rclcpp_1121_repro+0x23ca9)
    #10 request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::atomic<bool>&) /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:46 (rclcpp_1121_repro+0xa4c6)
    #11 operator() /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:114 (rclcpp_1121_repro+0xa99c)
    #12 __invoke_impl<void, main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0xd419)
    #13 __invoke<main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0xbfcb)
    #14 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1b82c)
    #15 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1b5f2)
    #16 operator() /usr/include/c++/7/future:1362 (rclcpp_1121_repro+0x1aea1)
    #17 _M_invoke /usr/include/c++/7/bits/std_function.h:302 (rclcpp_1121_repro+0x19935)
    #18 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706 (rclcpp_1121_repro+0x22340)
    #19 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561 (rclcpp_1121_repro+0x20594)
    #20 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73 (rclcpp_1121_repro+0x2787a)
    #21 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x24844)
    #22 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672 (rclcpp_1121_repro+0x21e31)
    #23 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21ed6)
    #24 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21f16)
    #25 pthread_once <null> (libtsan.so.0+0x2dc39)
    #26 __gthread_once /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:699 (rclcpp_1121_repro+0x9826)
    #27 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/mutex:684 (rclcpp_1121_repro+0x2205e)
    #28 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/7/future:401 (rclcpp_1121_repro+0x2023f)
    #29 operator() /usr/include/c++/7/future:1667 (rclcpp_1121_repro+0x174e1)
    #30 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0x19c5f)
    #31 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x184c9)
    #32 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1e58e)
    #33 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1df0c)
    #34 _M_run /usr/include/c++/7/thread:186 (rclcpp_1121_repro+0x1c2a0)
    #35 <null> <null> (libstdc++.so.6+0xbd6de)

  Mutex M826545943351435664 is already destroyed.

  Thread T7 (tid=19764, running) created by thread T3 at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xbd994)
    #2 void std::allocator_traits<std::allocator<rclcpp::Node> >::construct<rclcpp::Node, char const (&) [30]>(std::allocator<rclcpp::Node>&, rclcpp::Node*, char const (&) [30]) /usr/include/c++/7/bits/alloc_traits.h:475 (rclcpp_1121_repro+0x309e7)
    #3 std::_Sp_counted_ptr_inplace<rclcpp::Node, std::allocator<rclcpp::Node>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [30]>(std::allocator<rclcpp::Node>, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:526 (rclcpp_1121_repro+0x2f86c)
    #4 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, rclcpp::Node*, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:637 (rclcpp_1121_repro+0x2dd38)
    #5 std::__shared_ptr<rclcpp::Node, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:1295 (rclcpp_1121_repro+0x2bce9)
    #6 std::shared_ptr<rclcpp::Node>::shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:344 (rclcpp_1121_repro+0x2a1da)
    #7 std::shared_ptr<rclcpp::Node> std::allocate_shared<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:691 (rclcpp_1121_repro+0x28b18)
    #8 std::shared_ptr<rclcpp::Node> std::make_shared<rclcpp::Node, char const (&) [30]>(char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:707 (rclcpp_1121_repro+0x2617e)
    #9 std::shared_ptr<rclcpp::Node> rclcpp::Node::make_shared<char const (&) [30]>(char const (&) [30]) /opt/ros/eloquent/include/rclcpp/node.hpp:75 (rclcpp_1121_repro+0x23ca9)
    #10 request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::atomic<bool>&) /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:46 (rclcpp_1121_repro+0xa4c6)
    #11 operator() /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:114 (rclcpp_1121_repro+0xa99c)
    #12 __invoke_impl<void, main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0xd419)
    #13 __invoke<main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0xbfcb)
    #14 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1b82c)
    #15 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1b5f2)
    #16 operator() /usr/include/c++/7/future:1362 (rclcpp_1121_repro+0x1aea1)
    #17 _M_invoke /usr/include/c++/7/bits/std_function.h:302 (rclcpp_1121_repro+0x19935)
    #18 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706 (rclcpp_1121_repro+0x22340)
    #19 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561 (rclcpp_1121_repro+0x20594)
    #20 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73 (rclcpp_1121_repro+0x2787a)
    #21 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x24844)
    #22 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672 (rclcpp_1121_repro+0x21e31)
    #23 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21ed6)
    #24 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21f16)
    #25 pthread_once <null> (libtsan.so.0+0x2dc39)
    #26 __gthread_once /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:699 (rclcpp_1121_repro+0x9826)
    #27 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/mutex:684 (rclcpp_1121_repro+0x2205e)
    #28 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/7/future:401 (rclcpp_1121_repro+0x2023f)
    #29 operator() /usr/include/c++/7/future:1667 (rclcpp_1121_repro+0x174e1)
    #30 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0x19c5f)
    #31 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x184c9)
    #32 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1e58e)
    #33 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1df0c)
    #34 _M_run /usr/include/c++/7/thread:186 (rclcpp_1121_repro+0x1c2a0)
    #35 <null> <null> (libstdc++.so.6+0xbd6de)

  Thread T3 (tid=19760, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xbd994)
    #2 _Async_state_impl /usr/include/c++/7/future:1664 (rclcpp_1121_repro+0x176f1)
    #3 construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/ext/new_allocator.h:136 (rclcpp_1121_repro+0x15f7d)
    #4 construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/alloc_traits.h:475 (rclcpp_1121_repro+0x14ddf)
    #5 _Sp_counted_ptr_inplace<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr_base.h:526 (rclcpp_1121_repro+0x1371e)
    #6 __shared_count<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void> >, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr_base.h:637 (rclcpp_1121_repro+0x115b8)
    #7 __shared_ptr<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void> >, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr_base.h:1295 (rclcpp_1121_repro+0x104d7)
    #8 shared_ptr<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void> >, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr.h:344 (rclcpp_1121_repro+0xfb28)
    #9 allocate_shared<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void> >, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr.h:691 (rclcpp_1121_repro+0xf196)
    #10 make_shared<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr.h:707 (rclcpp_1121_repro+0xe289)
    #11 _S_make_async_state<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/future:1705 (rclcpp_1121_repro+0xd4ef)
    #12 async<main(int, char**)::<lambda()> > /usr/include/c++/7/future:1719 (rclcpp_1121_repro+0xc0b2)
    #13 async<main(int, char**)::<lambda()> > /usr/include/c++/7/future:1749 (rclcpp_1121_repro+0xb1a4)
    #14 main /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:114 (rclcpp_1121_repro+0xab4f)

SUMMARY: ThreadSanitizer: data race (/usr/lib/x86_64-linux-gnu/libtsan.so.0+0x3fadb) in __interceptor_pthread_mutex_lock
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=19756)
  Cycle in lock order graph: M923091860363544392 (0x000000000000) => M924217897709013128 (0x000000000000) => M923091860363544392

  Mutex M924217897709013128 acquired here while holding mutex M923091860363544392 in thread T25:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x3fadb)
    #1 <null> <null> (librclcpp.so+0x9fb1c)
    #2 operator() /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:60 (rclcpp_1121_repro+0xa139)
    #3 __invoke_impl<std::__cxx11::basic_string<char>, request(const string&, std::atomic<bool>&)::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0xcc24)
    #4 __invoke<request(const string&, std::atomic<bool>&)::<lambda()> > /usr/include/c++/7/bits/invoke.h:96 (rclcpp_1121_repro+0xba80)
    #5 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1b765)
    #6 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1b40f)
    #7 operator() /usr/include/c++/7/future:1339 (rclcpp_1121_repro+0x1a662)
    #8 _M_invoke /usr/include/c++/7/bits/std_function.h:302 (rclcpp_1121_repro+0x18ff1)
    #9 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706 (rclcpp_1121_repro+0x22340)
    #10 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561 (rclcpp_1121_repro+0x20594)
    #11 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73 (rclcpp_1121_repro+0x2787a)
    #12 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x24844)
    #13 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672 (rclcpp_1121_repro+0x21e31)
    #14 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21ed6)
    #15 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21f16)
    #16 pthread_once <null> (libtsan.so.0+0x2dc39)
    #17 __gthread_once /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:699 (rclcpp_1121_repro+0x9826)
    #18 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/mutex:684 (rclcpp_1121_repro+0x2205e)
    #19 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/7/future:401 (rclcpp_1121_repro+0x2023f)
    #20 operator() /usr/include/c++/7/future:1667 (rclcpp_1121_repro+0x16be5)
    #21 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >; _Res = std::__cxx11::basic_string<char>]::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0x1931b)
    #22 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >; _Res = std::__cxx11::basic_string<char>]::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x17f7d)
    #23 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1e63e)
    #24 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1dff0)
    #25 _M_run /usr/include/c++/7/thread:186 (rclcpp_1121_repro+0x1c334)
    #26 <null> <null> (libstdc++.so.6+0xbd6de)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M923091860363544392 acquired here while holding mutex M924217897709013128 in thread T27:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x3fadb)
    #1 rclcpp::node_interfaces::NodeGraph::notify_graph_change() <null> (librclcpp.so+0xc067a)

  Thread T25 (tid=19782, running) created by thread T3 at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xbd994)
    #2 _Async_state_impl /usr/include/c++/7/future:1664 (rclcpp_1121_repro+0x16dca)
    #3 construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >, std::__cxx11::basic_string<char> >, std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > > > /usr/include/c++/7/ext/new_allocator.h:136 (rclcpp_1121_repro+0x15881)
    #4 construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >, std::__cxx11::basic_string<char> >, std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > > > /usr/include/c++/7/bits/alloc_traits.h:475 (rclcpp_1121_repro+0x14587)
    #5 _Sp_counted_ptr_inplace<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr_base.h:526 (rclcpp_1121_repro+0x12926)
    #6 __shared_count<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >, std::__cxx11::basic_string<char> >, std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >, std::__cxx11::basic_string<char> > >, std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr_base.h:637 (rclcpp_1121_repro+0x10eb0)
    #7 __shared_ptr<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >, std::__cxx11::basic_string<char> > >, std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr_base.h:1295 (rclcpp_1121_repro+0x1012f)
    #8 shared_ptr<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >, std::__cxx11::basic_string<char> > >, std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr.h:344 (rclcpp_1121_repro+0xf900)
    #9 allocate_shared<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >, std::__cxx11::basic_string<char> >, std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >, std::__cxx11::basic_string<char> > >, std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr.h:691 (rclcpp_1121_repro+0xeb9e)
    #10 make_shared<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >, std::__cxx11::basic_string<char> >, std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr.h:707 (rclcpp_1121_repro+0xdd41)
    #11 _S_make_async_state<std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > > > /usr/include/c++/7/future:1705 (rclcpp_1121_repro+0xcd13)
    #12 async<request(const string&, std::atomic<bool>&)::<lambda()> > /usr/include/c++/7/future:1719 (rclcpp_1121_repro+0xbb6a)
    #13 async<request(const string&, std::atomic<bool>&)::<lambda()> > /usr/include/c++/7/future:1749 (rclcpp_1121_repro+0xb05a)
    #14 request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::atomic<bool>&) /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:86 (rclcpp_1121_repro+0xa673)
    #15 operator() /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:114 (rclcpp_1121_repro+0xa99c)
    #16 __invoke_impl<void, main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0xd419)
    #17 __invoke<main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0xbfcb)
    #18 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1b82c)
    #19 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1b5f2)
    #20 operator() /usr/include/c++/7/future:1362 (rclcpp_1121_repro+0x1aea1)
    #21 _M_invoke /usr/include/c++/7/bits/std_function.h:302 (rclcpp_1121_repro+0x19935)
    #22 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706 (rclcpp_1121_repro+0x22340)
    #23 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561 (rclcpp_1121_repro+0x20594)
    #24 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73 (rclcpp_1121_repro+0x2787a)
    #25 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x24844)
    #26 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672 (rclcpp_1121_repro+0x21e31)
    #27 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21ed6)
    #28 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21f16)
    #29 pthread_once <null> (libtsan.so.0+0x2dc39)
    #30 __gthread_once /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:699 (rclcpp_1121_repro+0x9826)
    #31 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/mutex:684 (rclcpp_1121_repro+0x2205e)
    #32 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/7/future:401 (rclcpp_1121_repro+0x2023f)
    #33 operator() /usr/include/c++/7/future:1667 (rclcpp_1121_repro+0x174e1)
    #34 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0x19c5f)
    #35 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x184c9)
    #36 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1e58e)
    #37 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1df0c)
    #38 _M_run /usr/include/c++/7/thread:186 (rclcpp_1121_repro+0x1c2a0)
    #39 <null> <null> (libstdc++.so.6+0xbd6de)

  Thread T27 (tid=19784, running) created by thread T25 at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xbd994)
    #2 operator() /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:60 (rclcpp_1121_repro+0xa139)
    #3 __invoke_impl<std::__cxx11::basic_string<char>, request(const string&, std::atomic<bool>&)::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0xcc24)
    #4 __invoke<request(const string&, std::atomic<bool>&)::<lambda()> > /usr/include/c++/7/bits/invoke.h:96 (rclcpp_1121_repro+0xba80)
    #5 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1b765)
    #6 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1b40f)
    #7 operator() /usr/include/c++/7/future:1339 (rclcpp_1121_repro+0x1a662)
    #8 _M_invoke /usr/include/c++/7/bits/std_function.h:302 (rclcpp_1121_repro+0x18ff1)
    #9 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706 (rclcpp_1121_repro+0x22340)
    #10 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561 (rclcpp_1121_repro+0x20594)
    #11 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73 (rclcpp_1121_repro+0x2787a)
    #12 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x24844)
    #13 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672 (rclcpp_1121_repro+0x21e31)
    #14 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21ed6)
    #15 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21f16)
    #16 pthread_once <null> (libtsan.so.0+0x2dc39)
    #17 __gthread_once /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:699 (rclcpp_1121_repro+0x9826)
    #18 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/mutex:684 (rclcpp_1121_repro+0x2205e)
    #19 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/7/future:401 (rclcpp_1121_repro+0x2023f)
    #20 operator() /usr/include/c++/7/future:1667 (rclcpp_1121_repro+0x16be5)
    #21 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >; _Res = std::__cxx11::basic_string<char>]::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0x1931b)
    #22 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<request(const string&, std::atomic<bool>&)::<lambda()> > >; _Res = std::__cxx11::basic_string<char>]::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x17f7d)
    #23 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1e63e)
    #24 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1dff0)
    #25 _M_run /usr/include/c++/7/thread:186 (rclcpp_1121_repro+0x1c334)
    #26 <null> <null> (libstdc++.so.6+0xbd6de)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/usr/lib/x86_64-linux-gnu/libtsan.so.0+0x3fadb) in __interceptor_pthread_mutex_lock
==================
Response received timely: Hi from the service
==================
WARNING: ThreadSanitizer: data race (pid=19756)
  Write of size 8 at 0x7ba0000000b0 by thread T3 (mutexes: write M839216801595638304):
    #0 close <null> (libtsan.so.0+0x2f437)
    #1 eprosima::fastrtps::rtps::UDPChannelResource::release() <null> (libfastrtps.so.1+0x236bb5)
    #2 void std::allocator_traits<std::allocator<rclcpp::Node> >::destroy<rclcpp::Node>(std::allocator<rclcpp::Node>&, rclcpp::Node*) /usr/include/c++/7/bits/alloc_traits.h:487 (rclcpp_1121_repro+0x3974f)
    #3 std::_Sp_counted_ptr_inplace<rclcpp::Node, std::allocator<rclcpp::Node>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /usr/include/c++/7/bits/shared_ptr_base.h:535 (rclcpp_1121_repro+0x37e54)
    #4 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/7/bits/shared_ptr_base.h:154 (rclcpp_1121_repro+0x245e8)
    #5 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/7/bits/shared_ptr_base.h:684 (rclcpp_1121_repro+0x21afc)
    #6 std::__shared_ptr<rclcpp::Node, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/7/bits/shared_ptr_base.h:1123 (rclcpp_1121_repro+0x213ed)
    #7 std::shared_ptr<rclcpp::Node>::~shared_ptr() /usr/include/c++/7/bits/shared_ptr.h:93 (rclcpp_1121_repro+0x21419)
    #8 request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::atomic<bool>&) /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:46 (rclcpp_1121_repro+0xa7e8)
    #9 operator() /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:114 (rclcpp_1121_repro+0xa99c)
    #10 __invoke_impl<void, main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0xd419)
    #11 __invoke<main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0xbfcb)
    #12 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1b82c)
    #13 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1b5f2)
    #14 operator() /usr/include/c++/7/future:1362 (rclcpp_1121_repro+0x1aea1)
    #15 _M_invoke /usr/include/c++/7/bits/std_function.h:302 (rclcpp_1121_repro+0x19935)
    #16 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706 (rclcpp_1121_repro+0x22340)
    #17 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561 (rclcpp_1121_repro+0x20594)
    #18 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73 (rclcpp_1121_repro+0x2787a)
    #19 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x24844)
    #20 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672 (rclcpp_1121_repro+0x21e31)
    #21 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21ed6)
    #22 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21f16)
    #23 pthread_once <null> (libtsan.so.0+0x2dc39)
    #24 __gthread_once /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:699 (rclcpp_1121_repro+0x9826)
    #25 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/mutex:684 (rclcpp_1121_repro+0x2205e)
    #26 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/7/future:401 (rclcpp_1121_repro+0x2023f)
    #27 operator() /usr/include/c++/7/future:1667 (rclcpp_1121_repro+0x174e1)
    #28 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0x19c5f)
    #29 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x184c9)
    #30 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1e58e)
    #31 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1df0c)
    #32 _M_run /usr/include/c++/7/thread:186 (rclcpp_1121_repro+0x1c2a0)
    #33 <null> <null> (libstdc++.so.6+0xbd6de)

  Previous read of size 8 at 0x7ba0000000b0 by thread T7:
    #0 recvmsg <null> (libtsan.so.0+0x3b47b)
    #1 eprosima::fastrtps::rtps::UDPChannelResource::Receive(unsigned char*, unsigned int, unsigned int&, eprosima::fastrtps::rtps::Locator_t&) <null> (libfastrtps.so.1+0x235a26)

  Location is file descriptor 11 created by thread T3 at:
    #0 socket <null> (libtsan.so.0+0x2ee73)
    #1 eprosima::fastrtps::rtps::UDPv4Transport::OpenAndBindInputSocket(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned short, bool) <null> (libfastrtps.so.1+0x2553e2)
    #2 void std::allocator_traits<std::allocator<rclcpp::Node> >::construct<rclcpp::Node, char const (&) [30]>(std::allocator<rclcpp::Node>&, rclcpp::Node*, char const (&) [30]) /usr/include/c++/7/bits/alloc_traits.h:475 (rclcpp_1121_repro+0x309e7)
    #3 std::_Sp_counted_ptr_inplace<rclcpp::Node, std::allocator<rclcpp::Node>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [30]>(std::allocator<rclcpp::Node>, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:526 (rclcpp_1121_repro+0x2f86c)
    #4 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, rclcpp::Node*, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:637 (rclcpp_1121_repro+0x2dd38)
    #5 std::__shared_ptr<rclcpp::Node, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:1295 (rclcpp_1121_repro+0x2bce9)
    #6 std::shared_ptr<rclcpp::Node>::shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:344 (rclcpp_1121_repro+0x2a1da)
    #7 std::shared_ptr<rclcpp::Node> std::allocate_shared<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:691 (rclcpp_1121_repro+0x28b18)
    #8 std::shared_ptr<rclcpp::Node> std::make_shared<rclcpp::Node, char const (&) [30]>(char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:707 (rclcpp_1121_repro+0x2617e)
    #9 std::shared_ptr<rclcpp::Node> rclcpp::Node::make_shared<char const (&) [30]>(char const (&) [30]) /opt/ros/eloquent/include/rclcpp/node.hpp:75 (rclcpp_1121_repro+0x23ca9)
    #10 request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::atomic<bool>&) /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:46 (rclcpp_1121_repro+0xa4c6)
    #11 operator() /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:114 (rclcpp_1121_repro+0xa99c)
    #12 __invoke_impl<void, main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0xd419)
    #13 __invoke<main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0xbfcb)
    #14 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1b82c)
    #15 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1b5f2)
    #16 operator() /usr/include/c++/7/future:1362 (rclcpp_1121_repro+0x1aea1)
    #17 _M_invoke /usr/include/c++/7/bits/std_function.h:302 (rclcpp_1121_repro+0x19935)
    #18 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706 (rclcpp_1121_repro+0x22340)
    #19 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561 (rclcpp_1121_repro+0x20594)
    #20 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73 (rclcpp_1121_repro+0x2787a)
    #21 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x24844)
    #22 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672 (rclcpp_1121_repro+0x21e31)
    #23 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21ed6)
    #24 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21f16)
    #25 pthread_once <null> (libtsan.so.0+0x2dc39)
    #26 __gthread_once /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:699 (rclcpp_1121_repro+0x9826)
    #27 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/mutex:684 (rclcpp_1121_repro+0x2205e)
    #28 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/7/future:401 (rclcpp_1121_repro+0x2023f)
    #29 operator() /usr/include/c++/7/future:1667 (rclcpp_1121_repro+0x174e1)
    #30 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0x19c5f)
    #31 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x184c9)
    #32 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1e58e)
    #33 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1df0c)
    #34 _M_run /usr/include/c++/7/thread:186 (rclcpp_1121_repro+0x1c2a0)
    #35 <null> <null> (libstdc++.so.6+0xbd6de)

  Mutex M839216801595638304 is already destroyed.

  Thread T3 (tid=19760, running) created by main thread at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xbd994)
    #2 _Async_state_impl /usr/include/c++/7/future:1664 (rclcpp_1121_repro+0x176f1)
    #3 construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/ext/new_allocator.h:136 (rclcpp_1121_repro+0x15f7d)
    #4 construct<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/alloc_traits.h:475 (rclcpp_1121_repro+0x14ddf)
    #5 _Sp_counted_ptr_inplace<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr_base.h:526 (rclcpp_1121_repro+0x1371e)
    #6 __shared_count<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void> >, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr_base.h:637 (rclcpp_1121_repro+0x115b8)
    #7 __shared_ptr<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void> >, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr_base.h:1295 (rclcpp_1121_repro+0x104d7)
    #8 shared_ptr<std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void> >, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr.h:344 (rclcpp_1121_repro+0xfb28)
    #9 allocate_shared<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::allocator<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void> >, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr.h:691 (rclcpp_1121_repro+0xf196)
    #10 make_shared<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >, void>, std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/bits/shared_ptr.h:707 (rclcpp_1121_repro+0xe289)
    #11 _S_make_async_state<std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > > > /usr/include/c++/7/future:1705 (rclcpp_1121_repro+0xd4ef)
    #12 async<main(int, char**)::<lambda()> > /usr/include/c++/7/future:1719 (rclcpp_1121_repro+0xc0b2)
    #13 async<main(int, char**)::<lambda()> > /usr/include/c++/7/future:1749 (rclcpp_1121_repro+0xb1a4)
    #14 main /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:114 (rclcpp_1121_repro+0xab4f)

  Thread T7 (tid=19764, running) created by thread T3 at:
    #0 pthread_create <null> (libtsan.so.0+0x2bcee)
    #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (libstdc++.so.6+0xbd994)
    #2 void std::allocator_traits<std::allocator<rclcpp::Node> >::construct<rclcpp::Node, char const (&) [30]>(std::allocator<rclcpp::Node>&, rclcpp::Node*, char const (&) [30]) /usr/include/c++/7/bits/alloc_traits.h:475 (rclcpp_1121_repro+0x309e7)
    #3 std::_Sp_counted_ptr_inplace<rclcpp::Node, std::allocator<rclcpp::Node>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<char const (&) [30]>(std::allocator<rclcpp::Node>, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:526 (rclcpp_1121_repro+0x2f86c)
    #4 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, rclcpp::Node*, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:637 (rclcpp_1121_repro+0x2dd38)
    #5 std::__shared_ptr<rclcpp::Node, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr_base.h:1295 (rclcpp_1121_repro+0x2bce9)
    #6 std::shared_ptr<rclcpp::Node>::shared_ptr<std::allocator<rclcpp::Node>, char const (&) [30]>(std::_Sp_make_shared_tag, std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:344 (rclcpp_1121_repro+0x2a1da)
    #7 std::shared_ptr<rclcpp::Node> std::allocate_shared<rclcpp::Node, std::allocator<rclcpp::Node>, char const (&) [30]>(std::allocator<rclcpp::Node> const&, char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:691 (rclcpp_1121_repro+0x28b18)
    #8 std::shared_ptr<rclcpp::Node> std::make_shared<rclcpp::Node, char const (&) [30]>(char const (&) [30]) /usr/include/c++/7/bits/shared_ptr.h:707 (rclcpp_1121_repro+0x2617e)
    #9 std::shared_ptr<rclcpp::Node> rclcpp::Node::make_shared<char const (&) [30]>(char const (&) [30]) /opt/ros/eloquent/include/rclcpp/node.hpp:75 (rclcpp_1121_repro+0x23ca9)
    #10 request(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::atomic<bool>&) /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:46 (rclcpp_1121_repro+0xa4c6)
    #11 operator() /tmp/rclcpp_1121_repro/rclcpp_1121_repro.cpp:114 (rclcpp_1121_repro+0xa99c)
    #12 __invoke_impl<void, main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0xd419)
    #13 __invoke<main(int, char**)::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0xbfcb)
    #14 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1b82c)
    #15 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1b5f2)
    #16 operator() /usr/include/c++/7/future:1362 (rclcpp_1121_repro+0x1aea1)
    #17 _M_invoke /usr/include/c++/7/bits/std_function.h:302 (rclcpp_1121_repro+0x19935)
    #18 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const /usr/include/c++/7/bits/std_function.h:706 (rclcpp_1121_repro+0x22340)
    #19 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) /usr/include/c++/7/future:561 (rclcpp_1121_repro+0x20594)
    #20 void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:73 (rclcpp_1121_repro+0x2787a)
    #21 std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x24844)
    #22 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const /usr/include/c++/7/mutex:672 (rclcpp_1121_repro+0x21e31)
    #23 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21ed6)
    #24 std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() /usr/include/c++/7/mutex:677 (rclcpp_1121_repro+0x21f16)
    #25 pthread_once <null> (libtsan.so.0+0x2dc39)
    #26 __gthread_once /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:699 (rclcpp_1121_repro+0x9826)
    #27 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) /usr/include/c++/7/mutex:684 (rclcpp_1121_repro+0x2205e)
    #28 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) /usr/include/c++/7/future:401 (rclcpp_1121_repro+0x2023f)
    #29 operator() /usr/include/c++/7/future:1667 (rclcpp_1121_repro+0x174e1)
    #30 __invoke_impl<void, std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:60 (rclcpp_1121_repro+0x19c5f)
    #31 __invoke<std::__future_base::_Async_state_impl<_BoundFn, _Res>::_Async_state_impl(_BoundFn&&) [with _BoundFn = std::thread::_Invoker<std::tuple<main(int, char**)::<lambda()> > >; _Res = void]::<lambda()> > /usr/include/c++/7/bits/invoke.h:95 (rclcpp_1121_repro+0x184c9)
    #32 _M_invoke<0> /usr/include/c++/7/thread:234 (rclcpp_1121_repro+0x1e58e)
    #33 operator() /usr/include/c++/7/thread:243 (rclcpp_1121_repro+0x1df0c)
    #34 _M_run /usr/include/c++/7/thread:186 (rclcpp_1121_repro+0x1c2a0)
    #35 <null> <null> (libstdc++.so.6+0xbd6de)

SUMMARY: ThreadSanitizer: data race (/usr/lib/x86_64-linux-gnu/libtsan.so.0+0x2f437) in __interceptor_close
==================
ThreadSanitizer: reported 3 warnings

You can also export TSAN_OPTIONS="second_deadlock_stack=1" before starting the executable to get complete stack traces from both threads involved in the lock order inversion issue.

[ivanpauno]

I checked the history of changes of appropriate code in ROS2, so I'm waiting for Foxy to be released to check it - if it's not fixed there still, I'll also report another bug

You should already be able to test with Foxy, see https://discourse.ros.org/t/ros-2-foxy-fitzroy-call-for-testing-and-package-releases/13998.

[yuriy-universal-ivanov]

Thanks @ivanpauno, it's unfortunately only available in Ubuntu 20 repos, but I'll check it soon.

Anyway, the deadlock thing doesn't seem to be affected by those changes, so I believe it has to be investigated (you now have repro code to test).
Please let me know if there is anything else I could help regarding the reported issue.

[fujitatomoya]

@ivanpauno

using https://github.com/ros2/ros2/commit/4eece71bc42f9b28e1c21633d270d908bb508ca7, source build with Ubuntu20.04, I can reproduce the same warning message from TSan. reproduction procedure is following,

> cd colcon_ws
> cd src
> git clone [email protected]:fujitatomoya/ros2_test_prover.git
> cd ..
> colcon build --symlink-install --packages-select ros2_test_prover
> source install/local_setup.bash
> for i in {1..100}; do ros2 run ros2_test_prover rclcpp_1121_repro; sleep 3; done

entire log is attached, rclcpp-issue-1121.log

I did try a few hundreds times, but cannot actually make deadlock happen. it is warning that indicates potential lock inversion. (also data race can be reproducible as reported.)

[fujitatomoya]

@ivanpauno

Thread Sanitizer Analysis (potential lock inversion)

• Thread-18: (created by user request thread)

  Mutex M292025100016882824 acquired here while holding mutex M290617587694362920 in thread T18:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x5271c)
    #1 __gthread_mutex_lock /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:749 (rclcpp_1121_repro+0x35fef)
    #2 std::mutex::lock() /usr/include/c++/9/bits/std_mutex.h:100 (rclcpp_1121_repro+0x364b4)
    #3 rclcpp::graph_listener::acquire_nodes_lock_(std::mutex*, std::mutex*, rcl_guard_condition_t*) <null> (librclcpp.so+0x560149)
    #4 operator() /root/docker_ws/ros2_colcon/src/fujitatomoya/ros2_test_prover/src/rclcpp_1121_repro.cpp:60 (rclcpp_1121_repro+0x20c03)

M292025100016882824 is GraphListener::node_graph_interfaces_mutex_, M290617587694362920 is NodeGraph::graph_mutex_.

this thread is created on https://github.com/fujitatomoya/ros2_test_prover/blob/master/src/rclcpp_1121_repro.cpp#L60,

this is 1st call from this node, so call in GraphListener::add_node, https://github.com/ros2/rclcpp/blob/87bb9f9758dce239d37256ae91d58ce6325f405b/rclcpp/src/rclcpp/graph_listener.cpp#L298

https://github.com/ros2/rclcpp/blob/87bb9f9758dce239d37256ae91d58ce6325f405b/rclcpp/src/rclcpp/graph_listener.cpp#L251 but this actually will be unlocked since using std::adopt_lock on https://github.com/ros2/rclcpp/blob/87bb9f9758dce239d37256ae91d58ce6325f405b/rclcpp/src/rclcpp/graph_listener.cpp#L302-L303

• Thread-21: (created by Thread-18)

  Mutex M290617587694362920 acquired here while holding mutex M292025100016882824 in thread T21:
    #0 pthread_mutex_lock <null> (libtsan.so.0+0x5271c)
    #1 __gthread_mutex_lock /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:749 (rclcpp_1121_repro+0x35fef)
    #2 std::mutex::lock() /usr/include/c++/9/bits/std_mutex.h:100 (rclcpp_1121_repro+0x364b4)
    #3 std::lock_guard<std::mutex>::lock_guard(std::mutex&) /usr/include/c++/9/bits/std_mutex.h:159 (rclcpp_1121_repro+0x39082)
    #4 rclcpp::node_interfaces::NodeGraph::notify_graph_change() <null> (librclcpp.so+0x59ac5c)

this thread is create on https://github.com/ros2/rclcpp/blob/87bb9f9758dce239d37256ae91d58ce6325f405b/rclcpp/src/rclcpp/graph_listener.cpp#L104

M292025100016882824 is GraphListener::node_graph_interfaces_mutex_, M290617587694362920 is NodeGraph::graph_mutex_.

https://github.com/ros2/rclcpp/blob/87bb9f9758dce239d37256ae91d58ce6325f405b/rclcpp/src/rclcpp/graph_listener.cpp#L146, then https://github.com/ros2/rclcpp/blob/87bb9f9758dce239d37256ae91d58ce6325f405b/rclcpp/src/rclcpp/node_interfaces/node_graph.cpp#L326

it seems,

• Thread-18: graph_mutex_ -> node_graph_interfaces_mutex_
• Thread-21: node_graph_interfaces_mutex_ -> graph_mutex_

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6962)
Cycle in lock order graph: M290617587694362920 (0x000000000000) => M292025100016882824 (0x000000000000) => M290617587694362920
                           NodeGraph::graph_mutex_ => GraphListener::node_graph_interfaces_mutex_ => NodeGraph::graph_mutex_

[fujitatomoya]

There is another problem detected by Thread Sanitizer about WARNING: ThreadSanitizer: data race (pid=12656). Callstack is related to eprosima::fastdds::rtps, data race condition between udp socket recv and close. so I create issue https://github.com/ros2/rmw_fastrtps/issues/391.

[ivanpauno]

@fujitatomoya What I understand from your comment is:

Thread A:

• Calls wait_for_service
  
  
  
  • wait_for_service calls get_graph_event.
  
  
  • get_graph_event takes the NodeGraph::graph_mutex_
    
    
    
    
    
    • In graph_listener->add_node, GraphListener::node_graph_interfaces_mutex_ is taken and released.
    
    
    
  
  
  • NodeGraph::graph_mutex_ is released.
  
  

Thread B (graph listener thread):

• In GraphListener::run_loop, GraphListener::node_graph_interfaces_mutex_ is taken.
  
  
  
  • In NodeGraph::NotifyGraphChange, NodeGraph::graph_mutex_ is taken and then realsed.
  
  
• GraphListener::run_loop releases GraphListener::node_graph_interfaces_mutex_.

I wouldn't say this is a false positive, but actually a potential deadlock.

1135 LGTM :+1: !

[ivanpauno]

@fujitatomoya can you contribute with backports to Dashing/Eloquent?

Thanks for #1135 fix!!!

[fujitatomoya]

@ivanpauno

can you contribute with backports to Dashing/Eloquent?

working on that in today (Japan Local Time)

will get back to you with PR.

[fujitatomoya]

@ivanpauno

could you help to merge the followings?

• https://github.com/ros2/rclcpp/pull/1138
• https://github.com/ros2/rclcpp/pull/1137

[ivanpauno]

@fujitatomoya Thanks for the fix and backports!!

@yuriy-universal-ivanov A backport to Eloquent has been merged, the fix will be available on debians in next patch release.

[yuriy-universal-ivanov]

@fujitatomoya @ivanpauno
Awesome, thank you both a lot!