Rclcpp: Memory leaks: Error in destruction of rcl action client handle: the Node Handle was destructed too early.

Created on 7 Feb 2021  ·  20Comments  ·  Source: ros2/rclcpp

Memory leaks are found when running test_buffer_client of tf2_ros. with the Address Sanitizer (https://en.wikipedia.org/wiki/AddressSanitizer) on.

Below is the error message. Let me know if you need a dockerfile to reproduce it. 

root@5490e71bc767:/opt/ros_ws# ./build/tf2_ros/test_buffer_client
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from TestBufferClient
[ RUN      ] TestBufferClient.lookup_transform_available
[       OK ] TestBufferClient.lookup_transform_available (236 ms)
[ RUN      ] TestBufferClient.lookup_transform_unavailable
[       OK ] TestBufferClient.lookup_transform_unavailable (175 ms)
[ RUN      ] TestBufferClient.can_transform_available
[       OK ] TestBufferClient.can_transform_available (172 ms)
[ RUN      ] TestBufferClient.can_transform_unavailable
[ERROR] [1612723447.179594300] [rclcpp_action]: Error in destruction of rcl action client handle: the Node Handle was destructed too early. You will leak memory
[       OK ] TestBufferClient.can_transform_unavailable (170 ms)
[----------] 4 tests from TestBufferClient (753 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (823 ms total)
[  PASSED  ] 4 tests.

=================================================================
==30==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 568 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055cc2d2da in rcl_action_client_init (/opt/ros/foxy/lib/librcl_action.so+0x52da)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c635892  (/opt/ros/foxy/lib/librcl.so+0x15892)
    #2 0x7f055cd1dd0a in rclcpp::graph_listener::GraphListener::GraphListener(std::shared_ptr<rclcpp::Context>) (/opt/ros/foxy/lib/librclcpp.so+0xe2d0a)
    #3 0x7f055cd4181b in std::shared_ptr<rclcpp::graph_listener::GraphListener> rclcpp::Context::get_sub_context<rclcpp::graph_listener::GraphListener, std::shared_ptr<rclcpp::Context> >(std::shared_ptr<rclcpp::Context>&&) (/opt/ros/foxy/lib/librclcpp.so+0x10681b)

Indirect leak of 480 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c6412d8 in rcl_subscription_init (/opt/ros/foxy/lib/librcl.so+0x212d8)

Indirect leak of 456 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d4708 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1c708)

Indirect leak of 432 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c63113e in rcl_client_init (/opt/ros/foxy/lib/librcl.so+0x1113e)

Indirect leak of 384 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d3026 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1b026)

Indirect leak of 384 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597e2e60  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2ae60)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 360 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d4d20 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1cd20)

Indirect leak of 360 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d3110 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1b110)

Indirect leak of 240 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597e2d18  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2ad18)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 201 byte(s) in 6 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c5ea56f in rmw_allocate (/opt/ros/foxy/lib/librmw.so+0x256f)

Indirect leak of 192 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597d4eb0 in rmw_create_client (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x1ceb0)

Indirect leak of 164 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f05597e466e  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2c66e)

Indirect leak of 161 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f05597e43de  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2c3de)

Indirect leak of 160 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbcd3f in operator new(unsigned long, std::nothrow_t const&) (/lib/x86_64-linux-gnu/libasan.so.5+0x10fd3f)
    #1 0x7f05597e1ec5  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x29ec5)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 103 byte(s) in 3 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055cbef37c in rcutils_strndup (/opt/ros/foxy/lib/librcutils.so+0xd37c)

Indirect leak of 98 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f05597e4156  (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x2c156)

Indirect leak of 96 byte(s) in 2 object(s) allocated from:
    #0 0x7f055cfbabc8 in malloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dbc8)
    #1 0x7f055c5ea56f in rmw_allocate (/opt/ros/foxy/lib/librmw.so+0x256f)
    #2 0x7f05597e0de1 in rmw_create_subscription (/opt/ros/foxy/lib/librmw_fastrtps_cpp.so+0x28de1)

Indirect leak of 64 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f055978faf7 in rmw_fastrtps_shared_cpp::__rmw_create_guard_condition(char const*) (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x2caf7)

Indirect leak of 24 byte(s) in 1 object(s) allocated from:
    #0 0x7f055cfbc947 in operator new(unsigned long) (/lib/x86_64-linux-gnu/libasan.so.5+0x10f947)
    #1 0x7f055978fae7 in rmw_fastrtps_shared_cpp::__rmw_create_guard_condition(char const*) (/opt/ros/foxy/lib/librmw_fastrtps_shared_cpp.so+0x2cae7)

SUMMARY: AddressSanitizer: 4983 byte(s) leaked in 49 allocation(s).
bug more-information-needed
Source
picture zhoulaifu

Most helpful comment

Perhaps something is wrong with this Dockerfile since it mixes rolling and foxy?

Yes, that's a problem (we haven't backported all of the fixes from rolling to foxy). You should change that to:

RUN cd $ROS_WS \
        &&  git clone -b ros2 --depth 1 https://github.com/ros2/geometry2 $ROS_WS/src/geometry2 \
        && . /opt/ros/${ROS_DISTRO}/setup.sh \
        &&  rm -rf ${ROS_WS}/build && VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+

(notice that we checkout the ros2 branch now)

picture clalancette on 4 Mar 2021
👍2

All 20 comments

@zhoulaifu

could you share the complete procedure to reproduce this issue? i want to check if this still happens in source build.

fujitatomoya picture fujitatomoya on 8 Feb 2021

You can use this dockerfile to reproduce the environmental setting.

In the docker container, you have a rox:foxy system with geometry2
compiled with the Address Sanitizer (ASan). Then run the following
binary from the tf2_ros package to reproduce the bug.
${ROS_WS}/build/tf2_ros/test_buffer_client

Please let me know if anything unclear.

zhoulaifu picture zhoulaifu on 8 Feb 2021
👍1

memory leak confirmed with source build https://github.com/ros2/ros2/commit/9552559b4d811f4cc613d5c65e3d560b18579b2c

colcon_ws# VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+ --packages-select tf2_ros
colcon_ws# ./build/tf2_ros/test_buffer_client
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TestBufferClient
[ RUN      ] TestBufferClient.lookup_transform_available
[ERROR] [1612790271.217678208] [rclcpp_action]: Error in destruction of rcl action client handle: the Node Handle was destructed too early. You will leak memory
[       OK ] TestBufferClient.lookup_transform_available (124 ms)
[ RUN      ] TestBufferClient.lookup_transform_unavailable
[       OK ] TestBufferClient.lookup_transform_unavailable (119 ms)
[ RUN      ] TestBufferClient.can_transform_available
[       OK ] TestBufferClient.can_transform_available (118 ms)
[ RUN      ] TestBufferClient.can_transform_unavailable
[       OK ] TestBufferClient.can_transform_unavailable (120 ms)
[----------] 4 tests from TestBufferClient (481 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (484 ms total)
[  PASSED  ] 4 tests.
fujitatomoya picture fujitatomoya on 8 Feb 2021

could be similar with https://github.com/ros2/rclcpp/pull/1296

fujitatomoya picture fujitatomoya on 8 Feb 2021

just sharing debug information.

stack trace via gdb

```

0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

1 0x00007f5c9d91e859 in __GI_abort () at abort.c:79

2 0x00007f5c9e780eab in rclcpp_action::ClientBaseImpl::ClientBaseImpl(std::shared_ptr, std::shared_ptr, std::shared_ptr, std::__cxx11::basic_string, std::allocator > const&, rosidl_action_type_support_t const, rcl_action_client_options_t const&)::{lambda(rcl_action_client_t)#1}::operator()(rcl_action_client_t*) const (__closure=0x55ae103eb120, client=0x55ae103c6f00)

at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp_action/src/client.cpp:62

3 0x00007f5c9e7888c0 in std::_Sp_counted_deleter, std::shared_ptr, std::shared_ptr, std::__cxx11::basic_string, std::allocator > const&, rosidl_action_type_support_t const, rcl_action_client_options_t const&)::{lambda(rcl_action_client_t)#1}, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (this=0x55ae103eb110)

at /usr/include/c++/9/bits/shared_ptr_base.h:471

4 0x000055ae0ff16130 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55ae103eb110)

at /usr/include/c++/9/bits/shared_ptr_base.h:155

5 0x000055ae0ff12a83 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x55ae103d9b98,

__in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730

6 0x00007f5c9e780990 in std::__shared_ptr::~__shared_ptr (

this=0x55ae103d9b90, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169

7 0x00007f5c9e7809b0 in std::shared_ptr::~shared_ptr (this=0x55ae103d9b90,

__in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103

8 0x00007f5c9e7834ea in rclcpp_action::ClientBaseImpl::~ClientBaseImpl (this=0x55ae103d9b40,

__in_chrg=<optimized out>) at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp_action/src/client.cpp:34

9 0x00007f5c9e783534 in std::default_delete::operator() (this=0x55ae103c8b70,

__ptr=0x55ae103d9b40) at /usr/include/c++/9/bits/unique_ptr.h:81

10 0x00007f5c9e7824ca in std::unique_ptr >::~unique_ptr (this=0x55ae103c8b70, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292

11 0x00007f5c9e77d8fc in rclcpp_action::ClientBase::~ClientBase (this=0x55ae103c8b60, __in_chrg=)

--Type for more, q to quit, c to continue without paging--
at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp_action/src/client.cpp:137

12 0x000055ae0ff1d64c in rclcpp_action::Client::~Client (this=0x55ae103c8b60,

__in_chrg=<optimized out>) at /root/ros2_ws/colcon_ws/install/rclcpp_action/include/rclcpp_action/client.hpp:622

13 0x000055ae0ff1d688 in rclcpp_action::Client::~Client (this=0x55ae103c8b60,

__in_chrg=<optimized out>) at /root/ros2_ws/colcon_ws/install/rclcpp_action/include/rclcpp_action/client.hpp:622

14 0x000055ae0ff1bf65 in rclcpp_action::create_client(std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::__cxx11::basic_string, std::allocator > const&, std::shared_ptr, rcl_action_client_options_t const&)::{lambda(rclcpp_action::Client)#1}::operator()(rclcpp_action::Client) const (this=0x55ae103ff9f0, ptr=0x55ae103c8b60)

at /root/ros2_ws/colcon_ws/install/rclcpp_action/include/rclcpp_action/create_client.hpp:79

15 0x000055ae0ff22d42 in std::_Sp_counted_deleter, rclcpp_action::create_client(std::shared_ptr, std::shared_ptr, std::shared_ptr, std::shared_ptr, std::__cxx11::basic_string, std::allocator > const&, std::shared_ptr, rcl_action_client_options_t const&)::{lambda(rclcpp_action::Client)#1}, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (this=0x55ae103ff9e0) at /usr/include/c++/9/bits/shared_ptr_base.h:471

16 0x000055ae0ff16130 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55ae103ff9e0)

at /usr/include/c++/9/bits/shared_ptr_base.h:155

17 0x000055ae0ff12a83 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7f5c6c000c28,

__in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730

18 0x000055ae0ff10d50 in std::__shared_ptr::~__shared_ptr (

this=0x7f5c6c000c20, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169

19 0x000055ae0ff10d70 in std::shared_ptr::~shared_ptr (this=0x7f5c6c000c20,

__in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103

20 0x00007f5c9e457384 in std::_Destroy > (__pointer=0x7f5c6c000c20)

at /usr/include/c++/9/bits/stl_construct.h:98

--Type for more, q to quit, c to continue without paging--

21 0x00007f5c9e45632d in std::_Destroy_aux::__destroy*> (

__first=0x7f5c6c000c20, __last=0x7f5c6c000c30) at /usr/include/c++/9/bits/stl_construct.h:108

22 0x00007f5c9e454e00 in std::_Destroy*> (__first=0x7f5c6c000c20,

__last=0x7f5c6c000c30) at /usr/include/c++/9/bits/stl_construct.h:137

23 0x00007f5c9e4546db in std::_Destroy*, std::shared_ptr > (

__first=0x7f5c6c000c20, __last=0x7f5c6c000c30) at /usr/include/c++/9/bits/stl_construct.h:206

24 0x00007f5c9e454049 in std::vector, std::allocator > >::~vector (this=0x55ae10375f80, __in_chrg=) at /usr/include/c++/9/bits/stl_vector.h:677

25 0x00007f5c9e4934e2 in rclcpp::memory_strategies::allocator_memory_strategy::AllocatorMemoryStrategy >::~AllocatorMemoryStrategy (this=0x55ae10375f00, __in_chrg=)

at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp/include/rclcpp/strategies/allocator_memory_strategy.hpp:46

26 0x00007f5c9e4987b9 in __gnu_cxx::new_allocator > >::destroy > > (this=0x55ae10375f00, __p=0x55ae10375f00) at /usr/include/c++/9/ext/new_allocator.h:153

27 0x00007f5c9e496ecf in std::allocator_traits > > >::destroy > > (__a=..., __p=0x55ae10375f00) at /usr/include/c++/9/bits/alloc_traits.h:497

28 0x00007f5c9e4937cf in std::_Sp_counted_ptr_inplace >, std::allocator > >, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x55ae10375ef0)

at /usr/include/c++/9/bits/shared_ptr_base.h:557

29 0x000055ae0ff16130 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x55ae10375ef0)

at /usr/include/c++/9/bits/shared_ptr_base.h:155

30 0x000055ae0ff12a83 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x55ae10375cf0,

__in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:730

31 0x000055ae0ff10848 in std::__shared_ptr::~__shared_ptr (this=0x55ae10375ce8, __in_chrg=) at /usr/include/c++/9/bits/shared_ptr_base.h:1169

32 0x000055ae0ff10868 in std::shared_ptr::~shared_ptr (this=0x55ae10375ce8,

__in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103

--Type for more, q to quit, c to continue without paging--

33 0x00007f5c9e45a41a in rclcpp::Executor::~Executor (this=0x55ae10375c28, __in_chrg=)

at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp/src/rclcpp/executor.cpp:95

34 0x00007f5c9e4711b2 in rclcpp::executors::SingleThreadedExecutor::~SingleThreadedExecutor (this=0x55ae10375c28,

__in_chrg=<optimized out>)
at /root/ros2_ws/colcon_ws/src/ros2/rclcpp/rclcpp/src/rclcpp/executors/single_threaded_executor.cpp:24

35 0x000055ae0ff127a6 in TestBufferClient::~TestBufferClient (this=0x55ae10375bf0, __in_chrg=)

at /root/ros2_ws/colcon_ws/src/ros2/geometry2/tf2_ros/test/test_buffer_client.cpp:103

36 0x000055ae0ff22af8 in TestBufferClient_lookup_transform_available_Test::~TestBufferClient_lookup_transform_available_Test (this=0x55ae10375bf0, __in_chrg=)

at /root/ros2_ws/colcon_ws/src/ros2/geometry2/tf2_ros/test/test_buffer_client.cpp:156

37 0x000055ae0ff22b18 in TestBufferClient_lookup_transform_available_Test::~TestBufferClient_lookup_transform_available_Test (this=0x55ae10375bf0, __in_chrg=)

at /root/ros2_ws/colcon_ws/src/ros2/geometry2/tf2_ros/test/test_buffer_client.cpp:156

38 0x000055ae0ff53cd2 in testing::Test::DeleteSelf_ (this=0x55ae10375bf0)

at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:492

39 0x000055ae0ff63e5a in testing::internal::HandleSehExceptionsInMethodIfSupported (

object=0x55ae10375bf0, 
method=(void (testing::Test::*)(class testing::Test * const)) 0x55ae0ff53ca4 <testing::Test::DeleteSelf_()>, 
location=0x55ae0ff7c267 "the test fixture's destructor")
at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2433

40 0x000055ae0ff5c8e5 in testing::internal::HandleExceptionsInMethodIfSupported (

object=0x55ae10375bf0, 
method=(void (testing::Test::*)(class testing::Test * const)) 0x55ae0ff53ca4 <testing::Test::DeleteSelf_()>, 
location=0x55ae0ff7c267 "the test fixture's destructor")
at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2469

41 0x000055ae0ff37f62 in testing::TestInfo::Run (this=0x55ae103704b0)

at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2690

42 0x000055ae0ff38656 in testing::TestSuite::Run (this=0x55ae103709c0)

at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2816

--Type for more, q to quit, c to continue without paging--

43 0x000055ae0ff4473c in testing::internal::UnitTestImpl::RunAllTests (this=0x55ae10370680)

at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:5338

44 0x000055ae0ff65388 in testing::internal::HandleSehExceptionsInMethodIfSupported (object=0x55ae10370680,

method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x55ae0ff44334 <testing::internal::UnitTestImpl::RunAllTests()>, 
location=0x55ae0ff7cb18 "auxiliary test code (environments or event listeners)")
at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2433

45 0x000055ae0ff5db23 in testing::internal::HandleExceptionsInMethodIfSupported (object=0x55ae10370680,

method=(bool (testing::internal::UnitTestImpl::*)(class testing::internal::UnitTestImpl * const)) 0x55ae0ff44334 <testing::internal::UnitTestImpl::RunAllTests()>, 
location=0x55ae0ff7cb18 "auxiliary test code (environments or event listeners)")
at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:2469

46 0x000055ae0ff42f58 in testing::UnitTest::Run (this=0x55ae0ffb64c0 )

at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/./src/gtest.cc:4925

47 0x000055ae0ff0ff04 in RUN_ALL_TESTS ()

at /root/ros2_ws/colcon_ws/install/gtest_vendor/src/gtest_vendor/include/gtest/gtest.h:2473

48 0x000055ae0ff0f28e in main (argc=1, argv=0x7ffd561349f8)

at /root/ros2_ws/colcon_ws/src/ros2/geometry2/tf2_ros/test/test_buffer_client.cpp:197

```　

fujitatomoya picture fujitatomoya on 9 Feb 2021

Just find out that the same memory leaks can be exposed by running three other binaries (besides tf2_ros/test_buffer_client) from geometry2, with Address Sanitizers on:

  • test_tf2/test_message_filter
  • tf2_ros/buffer_server
  • tf2_ros/tf2_ros_test_time_reset

The leaks can be reproduced on the ros:foxy docker with the same dockerfile above. I have not tested with the recent rclcpp though (due to difficulties with source compilation).

zhoulaifu picture zhoulaifu on 15 Feb 2021
👍1

Besides, the same memory leak can also be triggered without sanitizers. Just compile with plain colcon build and run tf2_ros/test_buffer_client a couple of times triggers the bug on my machine.

for i in {1..20}; do ${ROS_WS}/build/tf2_ros/test_buffer_client ;done

zhoulaifu picture zhoulaifu on 20 Feb 2021

Sorry. Just realized that the memory leaks have been addressed recently.

zhoulaifu picture zhoulaifu on 21 Feb 2021

we've fixed some memory leaks in a couple of month, but i can still see this problem.

fujitatomoya picture fujitatomoya on 22 Feb 2021

@zhoulaifu

could you try https://github.com/ros2/rclcpp/pull/1562? i tried more than hundreds times, no memory leak observed.

fujitatomoya picture fujitatomoya on 25 Feb 2021

Just saw this. @fujitatomoya you mean try the newly merged one (related to another issue) ? By the way, I just realize that no Continuous Integration is used for rclcpp: https://index.ros.org/r/rclcpp/, or I may misunderstand?

zhoulaifu picture zhoulaifu on 2 Mar 2021

By the way, I just realize that no Continuous Integration is used for rclcpp: https://index.ros.org/r/rclcpp/, or I may misunderstand?

Unfortunately, the index doesn't understand all of the ways we do Continuous Integration. We actually have several forms of CI for rclcpp (and all of the core ROS 2 packages, for that matter).

Every time a PR is opened a "PR job" is run (you can see one of the runs for rclcpp here). That job is helpful to give early feedback. The downside is that it only runs on Linux, and cannot test API changes across repositories.

To handle the latter two things, we always run jobs on https://ci.ros2.org on all platforms, with all core packages built from source. You can see what that looks like here: https://github.com/ros2/rclcpp/pull/1562#issuecomment-785698760

clalancette picture clalancette on 2 Mar 2021

@zhoulaifu

you mean try the newly merged one

yes, you can use either rolling or foxy from the source. (I already did confirm the problem is gone, but just in case) or if it is okay for you to close issue, we can do that too.

fujitatomoya picture fujitatomoya on 3 Mar 2021

@fujitatomoya Could you share a Dockerfile or something that specifies your environment where this bug disappears? Probably i misuse something as I still got the error with the following Dockerfile. 

FROM ros:rolling

ENV ROS_WS /opt/ros_ws
WORKDIR $ROS_WS

RUN cd $ROS_WS \
        &&  git clone -b foxy --depth 1 https://github.com/ros2/geometry2 $ROS_WS/src/geometry2 \
        && . /opt/ros/${ROS_DISTRO}/setup.sh \
        &&  rm -rf ${ROS_WS}/build && VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+

Perhaps something is wrong with this Dockerfile since it mixes rolling and foxy?

zhoulaifu picture zhoulaifu on 4 Mar 2021

Perhaps something is wrong with this Dockerfile since it mixes rolling and foxy?

Yes, that's a problem (we haven't backported all of the fixes from rolling to foxy). You should change that to:

RUN cd $ROS_WS \
        &&  git clone -b ros2 --depth 1 https://github.com/ros2/geometry2 $ROS_WS/src/geometry2 \
        && . /opt/ros/${ROS_DISTRO}/setup.sh \
        &&  rm -rf ${ROS_WS}/build && VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+

(notice that we checkout the ros2 branch now)

clalancette picture clalancette on 4 Mar 2021
👍2

@fujitatomoya FYI, I tried with a corrected dockerfile (thanks to @clalancette),

FROM ros:rolling

ENV ROS_WS /opt/ros_ws
WORKDIR $ROS_WS

RUN cd $ROS_WS \
        &&  git clone -b ros2 --depth 1 https://github.com/ros2/geometry2 $ROS_WS/src/geometry2 \
        && . /opt/ros/${ROS_DISTRO}/setup.sh \
        &&  rm -rf ${ROS_WS}/build && VERBOSE=1 CXX="g++" CXXFLAGS="--coverage -g -fsanitize=address" colcon build --event-handlers console_direct+

I got no errors with a single run, but still got memory leaks after many runs

  • This is okay: ./build/tf2_ros/test_buffer_client
  • This triggers memory leaks again: for i in {1..10}; do ./build/tf2_ros/test_buffer_client; done
zhoulaifu picture zhoulaifu on 5 Mar 2021

Can you give us the stack trace of the leak?

clalancette picture clalancette on 5 Mar 2021
👍1

@zhoulaifu

i tried with source build https://github.com/ros2/ros2/blob/master/ros2.repos, cannot make that happen.

fujitatomoya picture fujitatomoya on 5 Mar 2021

Okay @fujitatomoya Then please ignore my message above. I thought you meant that testing on rolling would be, in this case, the same.

Also, I do not know if this is easily doable: is it possible to have a Dockerfile published on the ROS website which specifies the build instructions of ROS2 from the latest source. In this way, if one finds a bug in a release version such as foxy, he/she can readily check it again on your latest source. For now, building ROS from source following instructions from the ROS website remains a tedious task for amateurs unfortunately, which could slow down the process of bug finding/reporting/confirming.

zhoulaifu picture zhoulaifu on 5 Mar 2021

@zhoulaifu thanks for the consideration, please feel free to re-open or open new issue anytime 😃

fujitatomoya picture fujitatomoya on 5 Mar 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

charlotteheggem picture charlotteheggem  ·  6Comments
orduno picture orduno  ·  5Comments
ivanpauno picture ivanpauno  ·  13Comments
Karsten1987 picture Karsten1987  ·  4Comments
bpwilcox picture bpwilcox  ·  8Comments