Mavsdk: Seg fault at system()

[jtroeth1]

Hello,

I am seeing a segfault which has to be in this line:

System& uav = mavlink.system();

I currently have no other lines in my code. It is seemingly random when it happens but usually within 3 minutes of running the software.
I am using mavsdk version on master at: a9b24162
Linux version: 5.7.10-201.fc32.x86_64

Any help appreciated.

Thanks,

[JonasVautherin]

mavsdk.system() is deprecated, but that does not mean it is ok for it to segfault :sweat_smile:.

This said, would you mind trying with the new interface? Something like:

auto system = mavsdk.systems.at(0);

Then system is a std::shared_ptr<System>. That's the new way.

[jtroeth1]

Also exists in f98ca541.

Sorry, I'm not seeing how I can actually implement it?
There isn't an interface to use it like that, what am i missing?

If it is deprecated, can we get the example code updated?

Thanks

[JonasVautherin]

Oh sorry, I understood you were on latest develop, but you are on the latest release actually.

Could you share the code and instructions on how to reproduce this?

[jtroeth1]

so I can't really give the code to reproduce it becuase it is our own in house software using the mavsdk library.
In normal operation just using the library examples the issue does not occur.

This is one of the backtraces, i'm not sure that is of much help though as the program crashes seemingly after the real problem has occured.

0x0000000000444429 in stkdispatch::DispatchHandler::queuePulse(int, bool) ()

0 0x0000000000444429 in stkdispatch::DispatchHandler::queuePulse(int, bool) ()

1 0x00000000004216e9 in pulseSignalHandler (signo=34, info=0x7ffff6cf28f0, context=0x7ffff6cf27c0) at /home/jtroeth/oresense/ostk/src/apps/UAVIsfResourceManager/DispatchManager.cpp:37

2

3 0x00007ffff6f84801 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib64/libc.so.6

4 0x00007ffff6f8a157 in nanosleep () from /usr/lib64/libc.so.6

5 0x00007ffff7ed346b in mavsdk::MavsdkImpl::work_thread() () from /usr/local/lib64/libmavsdk.so.0

6 0x00007ffff72c1ac4 in execute_native_thread_routine () from /usr/lib64/libstdc++.so.6

7 0x00007ffff7faf432 in start_thread () from /usr/lib64/libpthread.so.0

8 0x00007ffff6fbd913 in clone () from /usr/lib64/libc.so.6

Thanks

[jtroeth1]

I haven't been able to replicate the issue in valgrind, so it could be a timing, threading thing.

[julianoes]

@jtroeth1 the backtrace is very confusing. How do we get from a sleep inside the MavsdkImpl::work_thread() to your stkdispatch?

I suggest to check thread apply all bt to get the backtrace for all threads.

Another way to debug this is to use thread, address, and undefined behavior sanitizers.

For MAVDKS, you can do it with these flags:
https://github.com/mavlink/MAVSDK/blob/88e932449984f96d1956568c0182a41f3bac66c8/src/cmake/compiler_flags.cmake#L81-L94

[JonasVautherin]

Wild guess: are you sure that your Mavsdk object does not get out of scope? If that object gets destructed, then your systems will probably segfault. That won't be the case anymore with the new interface, but it hasn't been released yet.

[jtroeth1]

(gdb) thread apply all where

Thread 9 (Thread 0x7fffe6ffd700 (LWP 6167)):

0 0x00007ffff6f84801 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib64/libc.so.6

1 0x00007ffff6f8a157 in nanosleep () from /usr/lib64/libc.so.6

2 0x00007ffff7ec9a39 in mavsdk::SystemImpl::system_thread() () from /usr/local/lib64/libmavsdk.so.0

3 0x00007ffff72c1ac4 in execute_native_thread_routine () from /usr/lib64/libstdc++.so.6

4 0x00007ffff7faf432 in start_thread () from /usr/lib64/libpthread.so.0

5 0x00007ffff6fbd913 in clone () from /usr/lib64/libc.so.6

Thread 8 (Thread 0x7fffe77fe700 (LWP 6166)):

0 0x00007ffff7fb5e92 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

1 0x00007ffff74a77bb in ACE_Condition::wait() () from /opt/dds/lib/libACE.so.6.2a_p16

2 0x00007ffff7cbf33c in OpenDDS::DCPS::QueueTaskBase >::svc() () from /opt/dds/lib/libOpenDDS_Dcps.so.3.13.3

3 0x00007ffff75028d5 in ACE_Task_Base::svc_run(void*) () from /opt/dds/lib/libACE.so.6.2a_p16

4 0x00007ffff75036ed in ACE_Thread_Adapter::invoke() () from /opt/dds/lib/libACE.so.6.2a_p16

5 0x00007ffff7faf432 in start_thread () from /usr/lib64/libpthread.so.0

6 0x00007ffff6fbd913 in clone () from /usr/lib64/libc.so.6

Thread 7 (Thread 0x7fffe7fff700 (LWP 6165)):

0 0x00007ffff7fb5e92 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

1 0x00007ffff74a77bb in ACE_Condition::wait() () from /opt/dds/lib/libACE.so.6.2a_p16

2 0x00007ffff7e75a64 in OpenDDS::DCPS::QueueTaskBase >::svc() () from /opt/dds/lib/libOpenDDS_Tcp.so.3.13.3

3 0x00007ffff75028d5 in ACE_Task_Base::svc_run(void*) () from /opt/dds/lib/libACE.so.6.2a_p16

4 0x00007ffff75036ed in ACE_Thread_Adapter::invoke() () from /opt/dds/lib/libACE.so.6.2a_p16

5 0x00007ffff7faf432 in start_thread () from /usr/lib64/libpthread.so.0

6 0x00007ffff6fbd913 in clone () from /usr/lib64/libc.so.6

Thread 6 (Thread 0x7ffff4c6b700 (LWP 6164)):

0 0x00007ffff6fb4fcb in select () from /usr/lib64/libc.so.6

1 0x00007ffff7c331c3 in ACE_Select_Reactor_T >::wait_for_multiple_events(ACE_Select_Reactor_Handle_Set&, ACE_Time_Value*) () from /opt/dds/lib/libOpenDDS_Dcps.so.3.13.3

2 0x00007ffff7c396c2 in ACE_Select_Reactor_T >::handle_events(ACE_Time_Value*) () from /opt/dds/lib/libOpenDDS_Dcps.so.3.13.3

3 0x00007ffff74f433c in ACE_Reactor::run_reactor_event_loop(int ()(ACE_Reactor)) () from /opt/dds/lib/libACE.so.6.2a_p16

4 0x00007ffff7c2faf5 in OpenDDS::DCPS::ReactorTask::svc() () from /opt/dds/lib/libOpenDDS_Dcps.so.3.13.3

5 0x00007ffff75028d5 in ACE_Task_Base::svc_run(void*) () from /opt/dds/lib/libACE.so.6.2a_p16

6 0x00007ffff75036ed in ACE_Thread_Adapter::invoke() () from /opt/dds/lib/libACE.so.6.2a_p16

7 0x00007ffff7faf432 in start_thread () from /usr/lib64/libpthread.so.0

8 0x00007ffff6fbd913 in clone () from /usr/lib64/libc.so.6

Thread 5 (Thread 0x7ffff546c700 (LWP 6163)):

0 0x00007ffff6fb4fcb in select () from /usr/lib64/libc.so.6

1 0x00007ffff750ba8c in ACE_TP_Reactor::get_event_for_dispatching(ACE_Time_Value*) () from /opt/dds/lib/libACE.so.6.2a_p16

2 0x00007ffff750c8d5 in ACE_TP_Reactor::dispatch_i(ACE_Time_Value*, ACE_TP_Token_Guard&) () from /opt/dds/lib/libACE.so.6.2a_p16

3 0x00007ffff750cc1f in ACE_TP_Reactor::handle_events(ACE_Time_Value*) () from /opt/dds/lib/libACE.so.6.2a_p16

4 0x00007ffff76505dd in TAO_ORB_Core::run(ACE_Time_Value*, int) () from /opt/dds/lib/libTAO.so.2.2a_p16

5 0x00007ffff54b9add in OpenDDS::DCPS::InfoRepoDiscovery::OrbRunner::svc() () from /opt/dds/lib/libOpenDDS_InfoRepoDiscovery.so

6 0x00007ffff75028d5 in ACE_Task_Base::svc_run(void*) () from /opt/dds/lib/libACE.so.6.2a_p16

7 0x00007ffff75036ed in ACE_Thread_Adapter::invoke() () from /opt/dds/lib/libACE.so.6.2a_p16

8 0x00007ffff7faf432 in start_thread () from /usr/lib64/libpthread.so.0

9 0x00007ffff6fbd913 in clone () from /usr/lib64/libc.so.6

Thread 4 (Thread 0x7ffff5cdd700 (LWP 6162)):

0 0x00007ffff6fb4fcb in select () from /usr/lib64/libc.so.6

1 0x00007ffff7c331c3 in ACE_Select_Reactor_T >::wait_for_multiple_events(ACE_Select_Reactor_Handle_Set&, ACE_Time_Value*) () from /opt/dds/lib/libOpenDDS_Dcps.so.3.13.3

2 0x00007ffff7c396c2 in ACE_Select_Reactor_T >::handle_events(ACE_Time_Value*) () from /opt/dds/lib/libOpenDDS_Dcps.so.3.13.3

3 0x00007ffff74f433c in ACE_Reactor::run_reactor_event_loop(int ()(ACE_Reactor)) () from /opt/dds/lib/libACE.so.6.2a_p16

4 0x00007ffff7c2faf5 in OpenDDS::DCPS::ReactorTask::svc() () from /opt/dds/lib/libOpenDDS_Dcps.so.3.13.3

5 0x00007ffff75028d5 in ACE_Task_Base::svc_run(void*) () from /opt/dds/lib/libACE.so.6.2a_p16

6 0x00007ffff75036ed in ACE_Thread_Adapter::invoke() () from /opt/dds/lib/libACE.so.6.2a_p16

7 0x00007ffff7faf432 in start_thread () from /usr/lib64/libpthread.so.0

8 0x00007ffff6fbd913 in clone () from /usr/lib64/libc.so.6

Thread 3 (Thread 0x7ffff64f2700 (LWP 6161)):

0 0x00007ffff7fb5e92 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

1 0x00007ffff72bb870 in std::condition_variable::wait(std::unique_lock&) () from /usr/lib64/libstdc++.so.6

2 0x00007ffff7ed35b7 in mavsdk::MavsdkImpl::process_user_callbacks_thread() () from /usr/local/lib64/libmavsdk.so.0

3 0x00007ffff72c1ac4 in execute_native_thread_routine () from /usr/lib64/libstdc++.so.6

4 0x00007ffff7faf432 in start_thread () from /usr/lib64/libpthread.so.0

5 0x00007ffff6fbd913 in clone () from /usr/lib64/libc.so.6

Thread 2 (Thread 0x7ffff6cf3700 (LWP 6160)):

0 0x0000000000444439 in stkdispatch::DispatchHandler::queuePulse(int, bool) ()

1 0x00000000004216e9 in pulseSignalHandler (signo=34, info=0x7ffff6cf28f0, context=0x7ffff6cf27c0) at /home/jtroeth/oresense/ostk/src/apps/UAVIsfResourceManager/DispatchManager.cpp:37

2

3 0x00007ffff6f84801 in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib64/libc.so.6

--Type for more, q to quit, c to continue without paging--

4 0x00007ffff6f8a157 in nanosleep () from /usr/lib64/libc.so.6

5 0x00007ffff7ed346b in mavsdk::MavsdkImpl::work_thread() () from /usr/local/lib64/libmavsdk.so.0

6 0x00007ffff72c1ac4 in execute_native_thread_routine () from /usr/lib64/libstdc++.so.6

7 0x00007ffff7faf432 in start_thread () from /usr/lib64/libpthread.so.0

8 0x00007ffff6fbd913 in clone () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7ffff6cf4740 (LWP 6156)):

0 0x0000000000444439 in stkdispatch::DispatchHandler::queuePulse(int, bool) ()

1 0x00000000004216e9 in pulseSignalHandler (signo=34, info=0x7fffffffcab0, context=0x7fffffffc980) at /home/jtroeth/oresense/ostk/src/apps/UAVIsfResourceManager/DispatchManager.cpp:37

2

3 0x00007ffff6ef8e6a in sigsuspend () from /usr/lib64/libc.so.6

4 0x00000000004446b4 in stkdispatch::DispatchHandler::dispatch() ()

5 0x0000000000421a11 in UAVIsfResourceManager::WaitForEvent (this=0x691260) at /home/jtroeth/oresense/ostk/src/apps/UAVIsfResourceManager/DispatchManager.cpp:111

6 0x000000000042c759 in UAVIsfResourceManager::execute (this=0x691260) at /home/jtroeth/oresense/ostk/build/src/apps/UAVIsfResourceManager/state.i:354

7 0x00000000004285b7 in main (argc=4, argv=0x7fffffffd558) at /home/jtroeth/oresense/ostk/src/apps/UAVIsfResourceManager/UAVIsfResourceManager.cpp:123

[jtroeth1]

It doesn't seem to be going out of scope...
Thanks Julian, I'll try those flags.

[jtroeth1]

Our software uses SIG34 for the timers, is it possible these could be conflicting with something?

Thanks,
James

[jtroeth1]

Issue was in our software. We needed a lock around the queue.
Unsure why the issue only showed itself in conjunction with the mavsdk library. We have over 20 apps and have never seen this.
Thanks,
James