Mavsdk: Intermittent mavsdk_server unit test failure

[julianoes]

After running many many times, at some point the test CameraServiceImplTest.sendsMultipleModes seems to fail in an odd way.

[ RUN      ] CameraServiceImplTest.sendsMultipleModes
modes: 
0: Video
1: Video
2: Video
3: Photo
4: Video
5: Unknown
received_modes: 
0: Video
1: Video
2: Video
3: Photo
4: Video
5: Unknown
6: Video
/home/julianoes/src/MAVSDK/src/backend/test/camera_service_impl_test.cpp:434: Failure
Expected equality of these values:
  modes.size()
    Which is: 6
  received_modes.size()
    Which is: 7

To reproduce:

cmake -DBUILD_BACKEND=ON -DENABLE_MAVLINK_PASSTHROUGH=ON -DCMAKE_BUILD_TYPE=Release -Bbuild/release -S.
cmake --build build/release -j8
while build/release/src/backend/test/unit_tests_backend; do :; done

[julianoes]

The backtrace for this is:

Thread 5 (Thread 0x7f964587d700 (LWP 3466848)):
#0  0x00007f964653af8d in syscall () from /usr/lib/libc.so.6
#1  0x00007f9646834862 in std::__atomic_futex_unsigned_base::_M_futex_wait_until (this=<optimized out>, __addr=0x7f9630001670, __val=2147483648, __has_timeout=<optimized out>, __s=..., __ns=...) at /build/gcc/src/gcc/libstdc++-v3/src/c++11/futex.cc:55
#2  0x00005607d55145d4 in std::__basic_future<void>::wait() const ()
#3  0x00005607d5527018 in mavsdk::backend::CameraServiceImpl<testing::NiceMock<mavsdk::testing::MockCamera> >::SubscribeMode(grpc_impl::ServerContext*, mavsdk::rpc::camera::SubscribeModeRequest const*, grpc_impl::ServerWriter<mavsdk::rpc::camera::ModeResponse>*) ()
#4  0x00007f9647437d23 in std::_Function_handler<grpc::Status (mavsdk::rpc::camera::CameraService::Service*, grpc_impl::ServerContext*, mavsdk::rpc::camera::SubscribeModeRequest const*, grpc_impl::ServerWriter<mavsdk::rpc::camera::ModeResponse>*), std::_Mem_fn<grpc::Status (mavsdk::rpc::camera::CameraService::Service::*)(grpc_impl::ServerContext*, mavsdk::rpc::camera::SubscribeModeRequest const*, grpc_impl::ServerWriter<mavsdk::rpc::camera::ModeResponse>*)> >::_M_invoke(std::_Any_data const&, mavsdk::rpc::camera::CameraService::Service*&&, grpc_impl::ServerContext*&&, mavsdk::rpc::camera::SubscribeModeRequest const*&&, grpc_impl::ServerWriter<mavsdk::rpc::camera::ModeResponse>*&&) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#5  0x00007f964745c40a in grpc_impl::internal::ServerStreamingHandler<mavsdk::rpc::camera::CameraService::Service, mavsdk::rpc::camera::SubscribeModeRequest, mavsdk::rpc::camera::ModeResponse>::RunHandler(grpc::internal::MethodHandler::HandlerParameter const&) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#6  0x00007f9647350e42 in grpc_impl::Server::SyncRequest::CallData::ContinueRunAfterInterception() () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#7  0x00007f9647352349 in grpc_impl::Server::SyncRequestThreadManager::DoWork(void*, bool, bool) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#8  0x00007f964735b854 in grpc::ThreadManager::MainWorkLoop() () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#9  0x00007f964735b96c in grpc::ThreadManager::WorkerThread::Run() () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#10 0x00007f9647a99748 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#11 0x00007f96469e946f in start_thread () from /usr/lib/libpthread.so.0
#12 0x00007f96465403d3 in clone () from /usr/lib/libc.so.6

Thread 4 (Thread 0x7f964487b700 (LWP 3466847)):
#0  0x00007f96469f001a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x00007f9647a975c0 in gpr_cv_wait () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#2  0x00007f964785d173 in timer_thread(void*) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#3  0x00007f9647a99748 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#4  0x00007f96469e946f in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f96465403d3 in clone () from /usr/lib/libc.so.6

Thread 3 (Thread 0x7f963ffff700 (LWP 3466846)):
#0  0x00007f96469efcf5 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x00007f9647a97553 in gpr_cv_wait () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#2  0x00007f964784f4b8 in grpc_core::Executor::ThreadMain(void*) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#3  0x00007f9647a99748 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#4  0x00007f96469e946f in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f96465403d3 in clone () from /usr/lib/libc.so.6

Thread 2 (Thread 0x7f963f7fe700 (LWP 3466845)):
#0  0x00007f96469efcf5 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x00007f9647a97553 in gpr_cv_wait () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#2  0x00007f964784f4b8 in grpc_core::Executor::ThreadMain(void*) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#3  0x00007f9647a99748 in grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::{lambda(void*)#1}::_FUN(void*) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#4  0x00007f96469e946f in start_thread () from /usr/lib/libpthread.so.0
#5  0x00007f96465403d3 in clone () from /usr/lib/libc.so.6

Thread 1 (Thread 0x7f9646081c40 (LWP 3466748)):
#0  0x00007f96469efcf5 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib/libpthread.so.0
#1  0x00007f9647a97553 in gpr_cv_wait () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#2  0x00007f964735af5b in grpc::ThreadManager::Wait() () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#3  0x00007f9647348981 in grpc_impl::Server::ShutdownInternal(gpr_timespec) () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#4  0x00007f9647348d4d in grpc_impl::Server::~Server() () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#5  0x00007f9647349079 in grpc_impl::Server::~Server() () from /home/julianoes/src/MAVSDK/build/release/src/backend/src/libmavsdk_server.so
#6  0x00005607d54e55e0 in (anonymous namespace)::CameraServiceImplTest::~CameraServiceImplTest() ()
#7  0x00005607d54e7a0b in (anonymous namespace)::CameraServiceImplTest_sendsMultipleModes_Test::~CameraServiceImplTest_sendsMultipleModes_Test() ()
#8  0x00007f96469a471d in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) () from /home/julianoes/src/MAVSDK/build/release/lib/libgtest.so
#9  0x00007f9646998a9d in testing::TestInfo::Run() () from /home/julianoes/src/MAVSDK/build/release/lib/libgtest.so
#10 0x00007f9646998c1d in testing::TestSuite::Run() () from /home/julianoes/src/MAVSDK/build/release/lib/libgtest.so
#11 0x00007f96469991b3 in testing::internal::UnitTestImpl::RunAllTests() () from /home/julianoes/src/MAVSDK/build/release/lib/libgtest.so--Type <RET> for more, q to quit, c to continue without paging--c

#12 0x00007f9646999418 in testing::UnitTest::Run() () from /home/julianoes/src/MAVSDK/build/release/lib/libgtest.so
#13 0x00005607d54b5eaf in main ()

[julianoes]

@JonasVautherin it looks like it's caused by:

https://github.com/mavlink/MAVSDK/blob/43bd1bc35db3362b4f702061f3b9429fdd48c222/src/backend/test/camera_service_impl_test.cpp#L418

[julianoes]

I have not seen this in a while, closing.