Cartographer_ros: Stack smashing detected

[rohbotics]

@damonkohler This is the same issue that I sent over email, opening a ticket with more detail.

Platform: Ubuntu 16.04, ROS Kinetic, x86_64

When I try and run the turtlebot launch (off of the latest version of this repo) I get a
stack smashing error, and the program SIGABRTs.
I tried to debug this by running it in GCC's ASAN (https://github.com/google/sanitizers/wiki/AddressSanitizer), but then everything works.
(maybe it is a race condidtion being covered up because of ASANs overhead?)

I ran it in gdb to get this backtrace:

#0  0x00007ffff4b17418 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff4b1901a in __GI_abort () at abort.c:89
#2  0x00007ffff4b5972a in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7ffff4c70c7f "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff4bfa89c in __GI___fortify_fail (msg=<optimized out>, msg@entry=0x7ffff4c70c61 "stack smashing detected") at fortify_fail.c:37
#4  0x00007ffff4bfa840 in __stack_chk_fail () at stack_chk_fail.c:28
#5  0x00007ffff7984596 in ceres::internal::CovarianceImpl::ComputeCovarianceValuesUsingSuiteSparseQR() () from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#6  0x00007ffff7989ea5 in ceres::internal::CovarianceImpl::ComputeCovarianceValues() () from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#7  0x00007ffff7989f59 in ceres::internal::CovarianceImpl::Compute(std::vector<std::pair<double const*, double const*>, std::allocator<std::pair<double const*, double const*> > > const&, ceres::internal::ProblemImpl*) ()
   from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#8  0x000000000061832a in cartographer::mapping_2d::scan_matching::CeresScanMatcher::Match(cartographer::transform::Rigid2<double> const&, cartographer::transform::Rigid2<double> const&, std::vector<Eigen::Matrix<float, 2, 1, 0, 2, 1>, std::allocator<Eigen::Matrix<float, 2, 1, 0, 2, 1> > > const&, cartographer::mapping_2d::ProbabilityGrid const&, cartographer::transform::Rigid2<double>*, Eigen::Matrix<double, 3, 3, 0, 3, 3>*, ceres::Solver::Summary*) const ()
#9  0x0000000000601df8 in cartographer::mapping_2d::LocalTrajectoryBuilder::ScanMatch(std::chrono::time_point<cartographer::common::UniversalTimeScaleClock, std::chrono::duration<long, std::ratio<1l, 10000000l> > >, cartographer::transform::Rigid3<double> const&, cartographer::transform::Rigid3<double> const&, cartographer::sensor::LaserFan const&, cartographer::transform::Rigid3<double>*, Eigen::Matrix<double, 6, 6, 0, 6, 6>*) ()
#10 0x0000000000602454 in cartographer::mapping_2d::LocalTrajectoryBuilder::AddHorizontalLaserFan(std::chrono::time_point<cartographer::common::UniversalTimeScaleClock, std::chrono::duration<long, std::ratio<1l, 10000000l> > >, cartographer::sensor::LaserFan3D const&) ()
#11 0x0000000000605197 in cartographer::mapping_2d::GlobalTrajectoryBuilder::AddHorizontalLaserFan(std::chrono::time_point<cartographer::common::UniversalTimeScaleClock, std::chrono::duration<long, std::ratio<1l, 10000000l> > >, cartographer::sensor::LaserFan3D const&) ()
#12 0x000000000053bf9f in cartographer_ros::(anonymous namespace)::Node::AddHorizontalLaserFan(long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cartographer::sensor::proto::LaserScan const&) ()
#13 0x000000000053f4b1 in cartographer_ros::(anonymous namespace)::Node::HandleSensorData(long, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >) ()
#14 0x000000000053c800 in cartographer_ros::(anonymous namespace)::Node::Initialize()::{lambda(long, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >)#2}::operator()(long, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >) const ()
#15 0x000000000054263b in std::_Function_handler<void (long, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >), cartographer_ros::(anonymous namespace)::Node::Initialize()::{lambda(long, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >)#2}>::_M_invoke(std::_Any_data const&, long&&, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >&&) ()
#16 0x00000000005427cb in std::function<void (long, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >)>::operator()(long, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >) const ()
#17 0x0000000000541646 in cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::AddTrajectory(int, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::function<void (long, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >)>)::{lambda(std::unique_ptr<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value, std::default_delete<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value> >)#1}::operator()(std::unique_ptr<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value, std::default_delete<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value> >) const ()
#18 0x0000000000543d52 in std::_Function_handler<void (std::unique_ptr<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value, std::default_delete<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value> >), cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::AddTrajectory(int, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::function<void (long, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >)>)::{lambda(std::unique_ptr<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value, std::default_delete<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value> >)#1}>::_M_invoke(std::_Any_data const&, std::unique_ptr<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value, std::default_delete<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value> >&&) ()
#19 0x0000000000544a2d in std::function<void (std::unique_ptr<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value, std::default_delete<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value> >)>::operator()(std::unique_ptr<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value, std::default_delete<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value> >) const ()
#20 0x0000000000543719 in cartographer::common::OrderedMultiQueue<cartographer::mapping::SensorCollatorQueueKey, long, cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value>::Dispatch() ()
#21 0x00000000005421aa in cartographer::common::OrderedMultiQueue<cartographer::mapping::SensorCollatorQueueKey, long, cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value>::Add(cartographer::mapping::SensorCollatorQueueKey const&, long const&, std::unique_ptr<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value, std::default_delete<cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::Value> >) ()
#22 0x0000000000540652 in cartographer::mapping::SensorCollator<cartographer_ros::(anonymous namespace)::SensorData>::AddSensorData(int, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_ptr<cartographer_ros::(anonymous namespace)::SensorData, std::default_delete<cartographer_ros::(anonymous namespace)::SensorData> >) ()
#23 0x000000000053bdae in cartographer_ros::(anonymous namespace)::Node::LaserScanMessageCallback(boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&) ()
#24 0x000000000054801c in boost::_mfi::mf1<void, cartographer_ros::(anonymous namespace)::Node, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&>::operator()(cartographer_ros::(anonymous namespace)::Node*, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&) const ()
#25 0x000000000054777d in void boost::_bi::list2<boost::_bi::value<cartographer_ros::(anonymous namespace)::Node*>, boost::arg<1> >::operator()<boost::_mfi::mf1<void, cartographer_ros::(anonymous namespace)::Node, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&>, boost::_bi::list1<boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&> >(boost::_bi::type<void>, boost::_mfi::mf1<void, cartographer_ros::(anonymous namespace)::Node, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&>&, boost::_bi::list1<boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&>&, int) ()
#26 0x000000000054694d in void boost::_bi::bind_t<void, boost::_mfi::mf1<void, cartographer_ros::(anonymous namespace)::Node, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&>, boost::_bi::list2<boost::_bi::value<cartographer_ros::(anonymous namespace)::Node*>, boost::arg<1> > >::operator()<boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&>(boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&) ()
#27 0x00000000005458e9 in boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, cartographer_ros::(anonymous namespace)::Node, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&>, boost::_bi::list2<boost::_bi::value<cartographer_ros::(anonymous namespace)::Node*>, boost::arg<1> > >, void, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&>::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&) ()
#28 0x00000000005738fd in boost::function1<void, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&>::operator()(boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&) const ()
#29 0x000000000056fbd1 in boost::detail::function::void_function_obj_invoker1<boost::function<void (boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&)>, void, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> >::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const>) ()
#30 0x000000000057fe46 in boost::function1<void, boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> >::operator()(boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const>) const ()
#31 0x000000000057dae9 in ros::SubscriptionCallbackHelperT<boost::shared_ptr<sensor_msgs::LaserScan_<std::allocator<void> > const> const&, void>::call(ros::SubscriptionCallbackHelperCallParams&) ()
#32 0x00007ffff63cf25d in ros::SubscriptionQueue::call() () from /opt/ros/kinetic/lib/libroscpp.so
#33 0x00007ffff6379880 in ros::CallbackQueue::callOneCB(ros::CallbackQueue::TLS*) () from /opt/ros/kinetic/lib/libroscpp.so
#34 0x00007ffff637ac83 in ros::CallbackQueue::callAvailable(ros::WallDuration) () from /opt/ros/kinetic/lib/libroscpp.so
#35 0x00007ffff63d29b9 in ros::SingleThreadedSpinner::spin(ros::CallbackQueue*) () from /opt/ros/kinetic/lib/libroscpp.so
#36 0x00007ffff63b83cb in ros::spin() () from /opt/ros/kinetic/lib/libroscpp.so
#37 0x000000000053f553 in cartographer_ros::(anonymous namespace)::Node::SpinForever() ()
#38 0x000000000053f6eb in cartographer_ros::(anonymous namespace)::Run() ()
#39 0x000000000054011a in main ()

[rohbotics]

If I disable the stack protector by adding -o overflow -fno-stack-protector to the compiler flags of ceres and cartographer_ros, it runs for 30sec - 1min before segfaulting.

(gdb) backtrace
#0  0x00007ffff7a2efcf in ceres::internal::ReorderProgramForSparseNormalCholesky(ceres::SparseLinearAlgebraLibraryType, ceres::OrderedGroups<double*> const&, ceres::internal::Program*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#1  0x00007ffff7a4e2dd in ceres::internal::TrustRegionPreprocessor::Preprocess(ceres::Solver::Options const&, ceres::internal::ProblemImpl*, ceres::internal::PreprocessedProblem*) ()
   from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#2  0x00007ffff7a463f4 in ceres::Solver::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) () from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#3  0x00007ffff7a46edf in ceres::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) () from /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1
#4  0x0000000000601505 in cartographer::mapping_2d::sparse_pose_graph::OptimizationProblem::Solve(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&, std::vector<cartographer::mapping::Submaps const*, std::allocator<cartographer::mapping::Submaps const*> > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*) ()
#5  0x00000000005f69a9 in cartographer::mapping_2d::SparsePoseGraph::RunOptimization() ()
#6  0x00000000005f752d in std::_Function_handler<void (std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&), cartographer::mapping_2d::SparsePoseGraph::HandleScanQueue()::{lambda(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&) ()
#7  0x000000000060639a in cartographer::mapping_2d::sparse_pose_graph::ConstraintBuilder::FinishComputation(int) ()
#8  0x000000000066351d in cartographer::common::ThreadPool::DoWork() ()
#9  0x00007ffff5186c80 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007ffff6d6a6fa in start_thread (arg=0x7fffdf7fe700) at pthread_create.c:333
#11 0x00007ffff4bf5b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[rohbotics]

I should add that the segfault also disappears while running under ASan, but not TSan or GDB.

[rohbotics]

I got the segfault to happen in valgrind

Here is the output:

==15634== Memcheck, a memory error detector
==15634== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==15634== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==15634== Command: /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node __name:=cartographer __log:=/home/rohan/.ros/log/d8b26ef8-5b38-11e6-863d-801f0286f30c/cartographer-1.log
==15634== 
[ INFO] [1470423499.181136001]: I0805 11:58:19.000000 15634 submaps.cc:175] Added submap 1
[ INFO] [1470423500.068260510]: I0805 11:58:20.000000 15634 motion_filter.cc:42] Motion filter reduced the number of scans to -nan%.
[ INFO] [1470423561.852290134]: I0805 11:59:21.000000 15634 motion_filter.cc:42] Motion filter reduced the number of scans to 5.4%.
==15634== Thread 8:
==15634== Invalid read of size 8
==15634==    at 0x4F7BFCF: ceres::internal::ReorderProgramForSparseNormalCholesky(ceres::SparseLinearAlgebraLibraryType, ceres::OrderedGroups<double*> const&, ceres::internal::Program*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F9B2DC: ceres::internal::TrustRegionPreprocessor::Preprocess(ceres::Solver::Options const&, ceres::internal::ProblemImpl*, ceres::internal::PreprocessedProblem*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F933F3: ceres::Solver::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F93EDE: ceres::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x601504: cartographer::mapping_2d::sparse_pose_graph::OptimizationProblem::Solve(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&, std::vector<cartographer::mapping::Submaps const*, std::allocator<cartographer::mapping::Submaps const*> > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x5F69A8: cartographer::mapping_2d::SparsePoseGraph::RunOptimization() (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x5F752C: std::_Function_handler<void (std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&), cartographer::mapping_2d::SparsePoseGraph::HandleScanQueue()::{lambda(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x606399: cartographer::mapping_2d::sparse_pose_graph::ConstraintBuilder::FinishComputation(int) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x66351C: cartographer::common::ThreadPool::DoWork() (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x7879C7F: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==15634==    by 0x5C986F9: start_thread (pthread_create.c:333)
==15634==  Address 0x8 is not stack'd, malloc'd or (recently) free'd
==15634== 
==15634== 
==15634== Process terminating with default action of signal 11 (SIGSEGV)
==15634==  Access not within mapped region at address 0x8
==15634==    at 0x4F7BFCF: ceres::internal::ReorderProgramForSparseNormalCholesky(ceres::SparseLinearAlgebraLibraryType, ceres::OrderedGroups<double*> const&, ceres::internal::Program*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F9B2DC: ceres::internal::TrustRegionPreprocessor::Preprocess(ceres::Solver::Options const&, ceres::internal::ProblemImpl*, ceres::internal::PreprocessedProblem*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F933F3: ceres::Solver::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x4F93EDE: ceres::Solve(ceres::Solver::Options const&, ceres::Problem*, ceres::Solver::Summary*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/ceres_catkin/lib/libceres.so.1.11.0)
==15634==    by 0x601504: cartographer::mapping_2d::sparse_pose_graph::OptimizationProblem::Solve(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&, std::vector<cartographer::mapping::Submaps const*, std::allocator<cartographer::mapping::Submaps const*> > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > > const&, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*, std::vector<cartographer::transform::Rigid2<double>, std::allocator<cartographer::transform::Rigid2<double> > >*) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x5F69A8: cartographer::mapping_2d::SparsePoseGraph::RunOptimization() (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x5F752C: std::_Function_handler<void (std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&), cartographer::mapping_2d::SparsePoseGraph::HandleScanQueue()::{lambda(std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<cartographer::mapping::SparsePoseGraph::Constraint2D, std::allocator<cartographer::mapping::SparsePoseGraph::Constraint2D> > const&) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x606399: cartographer::mapping_2d::sparse_pose_graph::ConstraintBuilder::FinishComputation(int) (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x66351C: cartographer::common::ThreadPool::DoWork() (in /home/rohan/cartographer_catkin_ws/devel_isolated/cartographer_ros/lib/cartographer_ros/cartographer_node)
==15634==    by 0x7879C7F: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21)
==15634==    by 0x5C986F9: start_thread (pthread_create.c:333)
==15634==    by 0x7E5FB5C: clone (clone.S:109)
==15634==  If you believe this happened as a result of a stack
==15634==  overflow in your program's main thread (unlikely but
==15634==  possible), you can try to increase the size of the
==15634==  main thread stack using the --main-stacksize= flag.
==15634==  The main thread stack size used in this run was 8388608.
==15634== 
==15634== HEAP SUMMARY:
==15634==     in use at exit: 131,395,694 bytes in 23,975 blocks
==15634==   total heap usage: 7,911,062 allocs, 7,887,087 frees, 1,637,650,075 bytes allocated
==15634== 
==15634== LEAK SUMMARY:
==15634==    definitely lost: 184 bytes in 1 blocks
==15634==    indirectly lost: 0 bytes in 0 blocks
==15634==      possibly lost: 4,800 bytes in 15 blocks
==15634==    still reachable: 131,390,710 bytes in 23,959 blocks
==15634==         suppressed: 0 bytes in 0 blocks
==15634== Rerun with --leak-check=full to see details of leaked memory
==15634== 
==15634== For counts of detected and suppressed errors, rerun with: -v
==15634== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

[rohbotics]

Here is a bag file that reproduces the error:

https://drive.google.com/file/d/0B2W9KeFIetG2NW9mNm5relQ0WEU/view?usp=sharing

[wohe]

Hi, I looked into it a bit and managed to reproduce the issue. I used the "Standalone Workspace" instructions for building on xenial/kinetic.

On my trusty/indigo system these instructions work without the issue. Likewise, installing as described in https://github.com/googlecartographer/cartographer, and using catkin only for cartographer and cartographer_ros (and not ceres_catkin) also works on xenial/kinetic. So this could at least be a short term workaround.

All crashes so far seem to be Ceres related, and can even be reproduced by some of the unit tests. A pretty small test which crashes on my system is https://github.com/googlecartographer/cartographer/blob/master/cartographer/mapping_2d/scan_matching/ceres_scan_matcher_test.cc.

[sandwichmaker]

@wohe could you share a smaller reproduction with me if possible? and i will work on fixing it?

[rohbotics]

This might be a bug in ceres_catkin, here it is checking out a commit (6a13e39) from the ceres repo, and saying that it is v1.11, https://github.com/ethz-asl/ceres_catkin/blob/master/CMakeLists.txt#L12.

In the github repo, v1.11 has is tagged as commit 4145556.
https://github.com/ceres-solver/ceres-solver/releases/tag/1.11.0

[wohe]

Already checked that. The hash is referring to upstream at https://ceres-solver.googlesource.com/ceres-solver/ and is indeed 1.11. Newer version and removing patches also did not help.

[wohe]

Shortest reproduction I have so far is on a Ubuntu 16.04 system with ROS kinetic installed.

sudo apt-get install \
  g++ \
  google-mock \
  libboost-all-dev \
  libgflags-dev \
  libgoogle-glog-dev \
  liblua5.2-dev \
  libprotobuf-dev \
  libsuitesparse-dev \
  libwebp-dev \
  ninja-build \
  protobuf-compiler \
  python-sphinx

mkdir -p ~/catkin_ws/src
cd ~/catkin_ws/src
source /opt/ros/kinetic/setup.bash
catkin_init_workspace

git clone https://github.com/googlecartographer/cartographer.git
git clone https://github.com/ethz-asl/ceres_catkin.git
git clone https://github.com/ethz-asl/suitesparse.git
git clone https://github.com/ethz-asl/glog_catkin.git
git clone https://github.com/ethz-asl/gflags_catkin.git
git clone https://github.com/ethz-asl/catkin_simple.git

cd ~/catkin_ws
catkin_make_isolated

Then several tests crash:

build_isolated/cartographer/devel/cartographer/mapping_2d/scan_matching/mapping_2d_scan_matching_ceres_scan_matcher_test
build_isolated/cartographer/devel/cartographer/mapping_2d/mapping_2d_sparse_pose_graph_test
build_isolated/cartographer/devel/cartographer/mapping_3d/sparse_pose_graph/mapping_3d_sparse_pose_graph_optimization_problem_test

[sandwichmaker]

is it possible for compile in debug mode and get a more detailed stack trace?

[wohe]

I investigated a bit more, and found the following:
The two functions from Ceres that lead to crashes (ComputeCovarianceValuesUsingSuiteSparseQR and OrderingForSparseNormalCholeskyUsingSuiteSparse) both have UsingSuiteSparse in their name, and it turns out that the version of catkin suitesparse and trusty's SuiteSparse are both 4.2.1, but xenial has 4.4.6. I uninstalled the xenial libsuitesparse-dev (which should be entirely unused in this build) and the stack smashing bug went away.

My conclusion: The catkin builds of Ceres and SuiteSparse are not ignoring the system's SuiteSparse as they should and compile/link together parts belonging to different versions of SuiteSparse. I don't know enough of catkin to fix this easily. Any ideas?

It seemed like it could be an ideal solution on xenial to remove the catkin build of Ceres, and just use xenial's libceres-dev. Unfortunately there is https://bugs.launchpad.net/ubuntu/+source/ceres-solver/+bug/1596296.

[rohbotics]

We can add FindCeres.cmake as part of ROS's cmake_modules (https://github.com/ros/cmake_modules) and add that as a dependency.

[wohe]

I think I found a solution that allows us to fix it on our side for now. @rohbotics, can you verify that #18 solves the issue for you? I tried it here with trusty/indigo and xenial/kinetic. Both seem fine.

[rohbotics]

18 fixes this issue

[bochen87]

Hi, I'm having the stack smashing as well.

roslaunch cartographer_ros demo_backpack_2d.launch bag_filename:=${HOME}/Downloads/cartographer_paper_deutsches_museum.bag
... logging to /home/bo/.ros/log/aa0e2930-07d5-11e7-a596-e4a7a035a11f/roslaunch-bo-ThinkPad-T460p-10557.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://bo-ThinkPad-T460p:34053/

SUMMARY
========

PARAMETERS
 * /robot_description: <...>
 * /rosdistro: kinetic
 * /rosversion: 1.12.7
 * /use_sim_time: True

NODES
  /
    cartographer_node (cartographer_ros/cartographer_node)
    playbag (rosbag/play)
    robot_state_publisher (robot_state_publisher/robot_state_publisher)
    rviz (rviz/rviz)

auto-starting new master
process[master]: started with pid [10570]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to aa0e2930-07d5-11e7-a596-e4a7a035a11f
process[rosout-1]: started with pid [10583]
started core service [/rosout]
process[robot_state_publisher-2]: started with pid [10590]
process[cartographer_node-3]: started with pid [10602]
process[rviz-4]: started with pid [10610]
process[playbag-5]: started with pid [10618]
[ INFO] [1489399992.671926019]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer_ros/configuration_files/backpack_2d.lua' for 'backpack_2d.lua'.
[ INFO] [1489399992.672306517]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/map_builder.lua' for 'map_builder.lua'.
[ INFO] [1489399992.672354654]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/map_builder.lua' for 'map_builder.lua'.
[ INFO] [1489399992.672440033]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/trajectory_builder_2d.lua' for 'trajectory_builder_2d.lua'.
[ INFO] [1489399992.672481481]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/trajectory_builder_2d.lua' for 'trajectory_builder_2d.lua'.
[ INFO] [1489399992.672639777]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/trajectory_builder_3d.lua' for 'trajectory_builder_3d.lua'.
[ INFO] [1489399992.672681100]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/trajectory_builder_3d.lua' for 'trajectory_builder_3d.lua'.
[ INFO] [1489399992.672858258]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/sparse_pose_graph.lua' for 'sparse_pose_graph.lua'.
[ INFO] [1489399992.672896972]: I0313 11:13:12.000000 10602 configuration_file_resolver.cc:40] Found '/home/bo/catkin_ws/install_isolated/share/cartographer/configuration_files/sparse_pose_graph.lua' for 'sparse_pose_graph.lua'.
[ INFO] [1489399992.792620165]: I0313 11:13:12.000000 10602 submaps.cc:176] Added submap 1
[ INFO] [1489399992.792723199]: I0313 11:13:12.000000 10602 map_builder_bridge.cc:37] Added trajectory with ID '0'.
[ INFO] [1489399993.392465301, 1432647016.509920142]: I0313 11:13:13.000000 10602 ordered_multi_queue.cc:172] All sensor data for trajectory 0 is available starting at '635682438164885689'.
*** stack smashing detected ***: /home/bo/catkin_ws/install_isolated/lib/cartographer_ros/cartographer_node terminated
[cartographer_node-3] process has died [pid 10602, exit code -6, cmd /home/bo/catkin_ws/install_isolated/lib/cartographer_ros/cartographer_node -configuration_directory /home/bo/catkin_ws/install_isolated/share/cartographer_ros/configuration_files -configuration_basename backpack_2d.lua echoes:=horizontal_laser_2d __name:=cartographer_node __log:=/home/bo/.ros/log/aa0e2930-07d5-11e7-a596-e4a7a035a11f/cartographer_node-3.log].
log file: /home/bo/.ros/log/aa0e2930-07d5-11e7-a596-e4a7a035a11f/cartographer_node-3*.log
0x1b98050 void QWindowPrivate::setTopLevelScreen(QScreen*, bool) ( QScreen(0xf95250) ): Attempt to set a screen on a child window.
0x1b98470 void QWindowPrivate::setTopLevelScreen(QScreen*, bool) ( QScreen(0xf95250) ): Attempt to set a screen on a child window.
0x1b8c830 void QWindowPrivate::setTopLevelScreen(QScreen*, bool) ( QScreen(0xf95250) ): Attempt to set a screen on a child window.
0x1b8b540 void QWindowPrivate::setTopLevelScreen(QScreen*, bool) ( QScreen(0xf95250) ): Attempt to set a screen on a child window.
^C[playbag-5] killing on exit
[rviz-4] killing on exit
[robot_state_publisher-2] killing on exit
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

I saw that you changed to a different ceres package now. I have the OpenBLAS and Suitesparse libraries installed on my PC and the suitsparse package in my catkin workspace. I removed the suitesparse package, but it sill gives the same error. I also uninstalled the libsuitesparse-dev package and it still gives the same error.

[bochen87]

I found the error, the suitsparse library was still in the catkin workspace devel folder. I had to remove it from there as well and it works now.

[BrannonKing]

You can modify the CMakeLists.txt to not use SuitSparse (and use EigenSparse instead). It appears that there is a bug in the SuiteSparse (v4.4.6) that comes on Ubuntu 16.04. It wasn't obvious how to compile the newer SuiteSparse on Ubuntu and get it installed into the right folders.

[skohlbr]

Also ran into the stack smashing issue, similar to what is described in the OP. In my case it was caused by another repo containing a suitesparse package (https://github.com/ethz-asl/kalibr/tree/master/suitesparse). After removing that and doing a clean build, the problem is gone. Adding this here in the interest of people googling for stack smashing cartographer who probably end up reading this issue :)