Icinga2: debug console with API connection sometimes hangs since 2.8.2
Hi All,
I use some easy script that shows a diff of checks when doing a config change in icinga2. So that its visible what checks were added/removed when adjusting the config.
All it does is:
ICINGA2_API_PASSWORD=xxxx icinga2 console --connect 'https://root@localhost:5665/' --eval "get_objects(Service).map(s => s.__name).sort();"
Now since upgrading to 2.8.2, we've noticed this command seems to hang quite often.
Even if we run it manually, we see it hanging, no output comes from icinga2, and not errors in the logs.
I've enabled debug logging, and no errors are thrown there neither. The only issue I can see is the following:
When it works I see the following:
[2018-03-26 12:18:59 +0200] information/ApiListener: New client connection from [127.0.0.1]:47054 (no client certificate)
[2018-03-26 12:18:59 +0200] notice/ApiListener: New HTTP client
[2018-03-26 12:18:59 +0200] debug/HttpRequest: line: POST /v1/console/execute-script?command=get_objects%28Service%29.map%28s%20%3D%3E%20s.__name%29.sort%28%29%3B&sandboxed=0&session=mon001-1522059539-0 HTTP/1.1, tokens: 3
[2018-03-26 12:18:59 +0200] information/HttpServerConnection: Request: POST /v1/console/execute-script?command=get_objects%28Service%29.map%28s%20%3D%3E%20s.__name%29.sort%28%29%3B&sandboxed=0&session=mon001-1522059539-0 (from [127.0.0.1]:47054, user: root)
[2018-03-26 12:18:59 +0200] notice/WorkQueue: Spawning WorkQueue threads for 'HttpServerConnection'
[2018-03-26 12:18:59 +0200] notice/Console: Executing expression: get_objects(Service).map(s => s.__name).sort();
[2018-03-26 12:18:59 +0200] debug/HttpServerConnection: Http client disconnected
[2018-03-26 12:18:59 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'HttpServerConnection'
When it hangs:
[2018-03-26 12:19:06 +0200] information/ApiListener: New client connection from [127.0.0.1]:47072 (no client certificate)
[2018-03-26 12:19:06 +0200] notice/ApiListener: New HTTP client
[2018-03-26 12:19:06 +0200] debug/HttpRequest: line: POST /v1/console/execute-script?command=get_objects%28Service%29.map%28s%20%3D%3E%20s.__name%29.sort%28%29%3B&sandboxed=0&session=mon001-1522059546-0 HTTP/1.1, tokens: 3
[2018-03-26 12:19:06 +0200] information/HttpServerConnection: Request: POST /v1/console/execute-script?command=get_objects%28Service%29.map%28s%20%3D%3E%20s.__name%29.sort%28%29%3B&sandboxed=0&session=mon001-1522059546-0 (from [127.0.0.1]:47072, user: root)
[2018-03-26 12:19:06 +0200] notice/WorkQueue: Spawning WorkQueue threads for 'HttpServerConnection'
[2018-03-26 12:19:16 +0200] information/WorkQueue: #28 (HttpServerConnection) items: 0, rate: 0.0166667/s (1/min 1/5min 1/15min);
[2018-03-26 12:19:16 +0200] information/HttpServerConnection: No messages for Http connection have been received in the last 10 seconds.
[2018-03-26 12:19:16 +0200] debug/HttpServerConnection: Http client disconnected
[2018-03-26 12:19:16 +0200] information/HttpServerConnection: Unable to disconnect Http client, I/O thread busy
[2018-03-26 12:19:16 +0200] notice/WorkQueue: Stopped WorkQueue threads for 'HttpServerConnection'
Lets hope this can be fixed!
dupondje
All 26 comments
Same problem here. Since 2.8.2 every command on icinga2 console hangs.
lgraziosi
on 26 Mar 2018
I see, when we implemented out fix we did not have the remote console on our mind. Ie. somebody having a good reason to keep the connection open for >10 seconds without doing anything. :G
Crunsher
on 26 Mar 2018
@Crunsher: Thats not the problem I guess. Its not that I'm waiting 10 seconds to execute it ..
I run the following from shell:
ICINGA2_API_PASSWORD=xxxx icinga2 console --connect 'https://root@localhost:5665/' --eval "get_objects(Service).map(s => s.__name).sort();"
And that hangs like 1 out of 3 times.
dupondje
on 26 Mar 2018
How many (service) objects? get_objects(Service) is pretty expensive.
dnsmichi
on 26 Mar 2018
2929 :)
Thing is, most of the time its quite fast (within the second). And other moments it just hangs. And according to the logs I pasted before, it doesn't even seem to get executed.
dupondje
on 26 Mar 2018
If you catch such a case, can you please extract a full gdb backtrace via pid connect? Details in https://www.icinga.com/docs/icinga2/latest/doc/21-development/#gdb-backtrace-from-running-process
dnsmichi
on 26 Mar 2018
Attached a gdb output when the command was hanging.
dupondje
on 26 Mar 2018
There is no timeout and the client waits for the server indefinitely. I am still unsure why those requests sometimes take forever, it's only happening sporadically.
The request shouldn't take that long but there are cases where this could very well happen and we don't want to kill a command just because it takes a while. But since the server will kill the connection after 10 seconds anyways, this sounds like a good timeout for the client.
This would solve the problem with it hanging.
Edit: Currently observed and semi-reliably reproducable behavior, icinga 2 is busy (ton of notifications in my case) -> console connects -> script execution is put into the working queue -> console waits -> icinga2 takes over 10 seconds to run the command -> icinga2 disconnects the console because of timeout -> client console waits forever an an answer. Even if the script is ever run (haven't found out if it is yet) there won't be an answer since the connection was closed.
Crunsher
on 4 Apr 2018
check_nscp_api has the same problem.
dnsmichi
on 9 Apr 2018
(lldb)
frame #3: 0x0000000100008ef1 check_nscp_api`QueryEndpoint(host=0x00007ffeefbfe9f8, port=0x00007ffeefbfe9b0, username=0x00007ffeefbfea48, password=0x00007ffeefbfe968, endpoint=0x00007ffeefbfec08) at check_nscp_api.cpp:111
108 // We need to spinlock here because our 'HttpRequest' works asynchronous
109 boost::mutex::scoped_lock lock(mtx);
110 while (!ready) {
-> 111 cv.wait(lock);
112 }
113
114 return result;
(lldb) bt all
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
frame #0: 0x00007fff5a6eacee libsystem_kernel.dylib`__psynch_cvwait + 10
* frame #1: 0x00007fff5a827662 libsystem_pthread.dylib`_pthread_cond_wait + 732
frame #2: 0x000000010000cf83 check_nscp_api`boost::condition_variable::wait(this=0x00007ffeefbfe5a8, m=0x00007ffeefbfe038) at condition_variable.hpp:81
frame #3: 0x0000000100008ef1 check_nscp_api`QueryEndpoint(host=0x00007ffeefbfe9f8, port=0x00007ffeefbfe9b0, username=0x00007ffeefbfea48, password=0x00007ffeefbfe968, endpoint=0x00007ffeefbfec08) at check_nscp_api.cpp:111
frame #4: 0x00000001000077b4 check_nscp_api`main(argc=10, argv=0x00007ffeefbff318) at check_nscp_api.cpp:317
frame #5: 0x00007fff5a59b115 libdyld.dylib`start + 1
frame #6: 0x00007fff5a59b115 libdyld.dylib`start + 1
thread #2
frame #0: 0x00007fff5a6eacee libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff5a827662 libsystem_pthread.dylib`_pthread_cond_wait + 732
frame #2: 0x000000010000cf83 check_nscp_api`boost::condition_variable::wait(this=0x00000001006fe3b0, m=0x000070000654ca08) at condition_variable.hpp:81
frame #3: 0x00000001002dab6b check_nscp_api`icinga::ThreadPool::WorkerThread::ThreadProc(this=0x00000001006fe4d8, queue=0x00000001006fe370) at threadpool.cpp:110
frame #4: 0x00000001002e4c34 check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] decltype(__f=0x000000010210fbb0, __a0=0x00000001006fe4d8, __args=0x00000001006fe370).*fp(std::__1::forward<icinga::ThreadPool::Queue&>(fp1))) std::__1::__invoke<void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&, void>(void (icinga::ThreadPool::WorkerThread::*&&&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&&&, icinga::ThreadPool::Queue&&&) at type_traits:4236
frame #5: 0x00000001002e4bac check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] std::__1::__bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x000000010210fbb0, __bound_args=0x000000010210fbc0, __args=0x000070000654cdd8)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, 0ul, 1ul, std::__1::tuple<> >(void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #6: 0x00000001002e4b13 check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] std::__1::__bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x000000010210fbb0)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >::operator()<>() at functional:2257
frame #7: 0x00000001002e4ade check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run(this=0x000000010210f9b0) at thread.hpp:116
frame #8: 0x000000010123e2ac libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 156
frame #9: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #10: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #11: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #3
frame #0: 0x00007fff5a6eacee libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff5a827662 libsystem_pthread.dylib`_pthread_cond_wait + 732
frame #2: 0x000000010000cf83 check_nscp_api`boost::condition_variable::wait(this=0x00000001006fe718, m=0x00007000065cfa08) at condition_variable.hpp:81
frame #3: 0x00000001002dab6b check_nscp_api`icinga::ThreadPool::WorkerThread::ThreadProc(this=0x00000001006fe840, queue=0x00000001006fe6d8) at threadpool.cpp:110
frame #4: 0x00000001002e4c34 check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] decltype(__f=0x000000010210fdd0, __a0=0x00000001006fe840, __args=0x00000001006fe6d8).*fp(std::__1::forward<icinga::ThreadPool::Queue&>(fp1))) std::__1::__invoke<void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&, void>(void (icinga::ThreadPool::WorkerThread::*&&&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&&&, icinga::ThreadPool::Queue&&&) at type_traits:4236
frame #5: 0x00000001002e4bac check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] std::__1::__bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x000000010210fdd0, __bound_args=0x000000010210fde0, __args=0x00007000065cfdd8)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, 0ul, 1ul, std::__1::tuple<> >(void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #6: 0x00000001002e4b13 check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] std::__1::__bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x000000010210fdd0)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >::operator()<>() at functional:2257
frame #7: 0x00000001002e4ade check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run(this=0x000000010210fbd0) at thread.hpp:116
frame #8: 0x000000010123e2ac libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 156
frame #9: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #10: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #11: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #4
frame #0: 0x00007fff5a6eacee libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff5a827662 libsystem_pthread.dylib`_pthread_cond_wait + 732
frame #2: 0x000000010000cf83 check_nscp_api`boost::condition_variable::wait(this=0x00000001006fea80, m=0x0000700006652a08) at condition_variable.hpp:81
frame #3: 0x00000001002dab6b check_nscp_api`icinga::ThreadPool::WorkerThread::ThreadProc(this=0x00000001006feba8, queue=0x00000001006fea40) at threadpool.cpp:110
frame #4: 0x00000001002e4c34 check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] decltype(__f=0x000000010210fff0, __a0=0x00000001006feba8, __args=0x00000001006fea40).*fp(std::__1::forward<icinga::ThreadPool::Queue&>(fp1))) std::__1::__invoke<void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&, void>(void (icinga::ThreadPool::WorkerThread::*&&&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&&&, icinga::ThreadPool::Queue&&&) at type_traits:4236
frame #5: 0x00000001002e4bac check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] std::__1::__bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x000000010210fff0, __bound_args=0x0000000102110000, __args=0x0000700006652dd8)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, 0ul, 1ul, std::__1::tuple<> >(void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #6: 0x00000001002e4b13 check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] std::__1::__bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x000000010210fff0)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >::operator()<>() at functional:2257
frame #7: 0x00000001002e4ade check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run(this=0x000000010210fdf0) at thread.hpp:116
frame #8: 0x000000010123e2ac libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 156
frame #9: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #10: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #11: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #5
frame #0: 0x00007fff5a6eacee libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff5a827662 libsystem_pthread.dylib`_pthread_cond_wait + 732
frame #2: 0x000000010000cf83 check_nscp_api`boost::condition_variable::wait(this=0x00000001006fede8, m=0x00007000066d5a08) at condition_variable.hpp:81
frame #3: 0x00000001002dab6b check_nscp_api`icinga::ThreadPool::WorkerThread::ThreadProc(this=0x00000001006fef10, queue=0x00000001006feda8) at threadpool.cpp:110
frame #4: 0x00000001002e4c34 check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] decltype(__f=0x0000000102110260, __a0=0x00000001006fef10, __args=0x00000001006feda8).*fp(std::__1::forward<icinga::ThreadPool::Queue&>(fp1))) std::__1::__invoke<void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&, icinga::ThreadPool::Queue&, void>(void (icinga::ThreadPool::WorkerThread::*&&&)(icinga::ThreadPool::Queue&), icinga::ThreadPool::WorkerThread&&&, icinga::ThreadPool::Queue&&&) at type_traits:4236
frame #5: 0x00000001002e4bac check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] std::__1::__bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x0000000102110260, __bound_args=0x0000000102110270, __args=0x00007000066d5dd8)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, 0ul, 1ul, std::__1::tuple<> >(void (icinga::ThreadPool::WorkerThread::*&)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #6: 0x00000001002e4b13 check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run() [inlined] std::__1::__bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::tuple<std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x0000000102110260)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> >::operator()<>() at functional:2257
frame #7: 0x00000001002e4ade check_nscp_api`boost::detail::thread_data<std::__1::__bind<void (icinga::ThreadPool::WorkerThread::*)(icinga::ThreadPool::Queue&), std::__1::reference_wrapper<icinga::ThreadPool::WorkerThread>, std::__1::reference_wrapper<icinga::ThreadPool::Queue> > >::run(this=0x0000000102110060) at thread.hpp:116
frame #8: 0x000000010123e2ac libboost_thread-mt.dylib`boost::(anonymous namespace)::thread_proxy(void*) + 156
frame #9: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #10: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #11: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #6
frame #0: 0x00007fff5a6eacee libsystem_kernel.dylib`__psynch_cvwait + 10
frame #1: 0x00007fff5a827662 libsystem_pthread.dylib`_pthread_cond_wait + 732
frame #2: 0x00000001002705d3 check_nscp_api`boost::condition_variable::do_wait_until(this=0x00000001006fe2f8, m=0x0000700006758920, timeout=0x0000700006758428) at condition_variable.hpp:115
frame #3: 0x000000010026c57c check_nscp_api`boost::condition_variable::timed_wait(this=0x00000001006fe2f8, m=0x0000700006758920, abs_time=0x0000700006758478) at condition_variable_fwd.hpp:140
frame #4: 0x00000001002dbcc6 check_nscp_api`bool boost::condition_variable::timed_wait<boost::date_time::subsecond_duration<boost::posix_time::time_duration, 1000ll> >(this=0x00000001006fe2f8, m=0x0000700006758920, wait_duration=0x0000700006758918) at condition_variable_fwd.hpp:164
frame #5: 0x00000001002d95dd check_nscp_api`icinga::ThreadPool::ManagerThreadProc(this=0x00000001006fe0f0) at threadpool.cpp:239
frame #6: 0x00000001002df204 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::ThreadPool::*)(), icinga::ThreadPool*> > >(void*) [inlined] decltype(__f=0x0000000102110048, __a0=0x0000000102110058)).*fp(std::__1::forward<>(fp1))) std::__1::__invoke<void (icinga::ThreadPool::*&)(), icinga::ThreadPool*&, void>(void (icinga::ThreadPool::*&&&)(), icinga::ThreadPool*&&&) at type_traits:4264
frame #7: 0x00000001002df185 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::ThreadPool::*)(), icinga::ThreadPool*> > >(void*) [inlined] std::__1::__bind_return<void (icinga::ThreadPool::*)(), std::__1::tuple<icinga::ThreadPool*>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::*)(), std::__1::tuple<icinga::ThreadPool*>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x0000000102110048, __bound_args=0x0000000102110058, __args=0x0000700006758e78)(), std::__1::tuple<icinga::ThreadPool*>, 0ul, std::__1::tuple<> >(void (icinga::ThreadPool::*&)(), std::__1::tuple<icinga::ThreadPool*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) at functional:2224
frame #8: 0x00000001002df155 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::ThreadPool::*)(), icinga::ThreadPool*> > >(void*) [inlined] std::__1::__bind_return<void (icinga::ThreadPool::*)(), std::__1::tuple<icinga::ThreadPool*>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::ThreadPool::*)(), std::__1::tuple<icinga::ThreadPool*>, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x0000000102110048)(), icinga::ThreadPool*>::operator()<>() at functional:2257
frame #9: 0x00000001002df132 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::ThreadPool::*)(), icinga::ThreadPool*> > >(void*) [inlined] decltype(__f=0x0000000102110048)(), icinga::ThreadPool*> >(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<std::__1::__bind<void (icinga::ThreadPool::*)(), icinga::ThreadPool*> >(std::__1::__bind<void (icinga::ThreadPool::*)(), icinga::ThreadPool*>&&) at type_traits:4323
frame #10: 0x00000001002df11b check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::ThreadPool::*)(), icinga::ThreadPool*> > >(void*) [inlined] void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::ThreadPool::*)(), icinga::ThreadPool*> >(__t=0x0000000102110040)(), icinga::ThreadPool*> >&, std::__1::__tuple_indices<>) at thread:342
frame #11: 0x00000001002df0e5 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::ThreadPool::*)(), icinga::ThreadPool*> > >(__vp=0x0000000102110040) at thread:352
frame #12: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #13: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #14: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #7
frame #0: 0x00007fff5a6ec372 libsystem_kernel.dylib`poll + 10
frame #1: 0x0000000100273568 check_nscp_api`icinga::SocketEventEnginePoll::ThreadProc(this=0x0000000101816200, tid=0) at socketevents-poll.cpp:88
frame #2: 0x0000000100272344 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x00000001017184a8, __a0=0x00000001017184b8, __args=0x00000001017184c0)).*fp(std::__1::forward<int&>(fp1))) std::__1::__invoke<void (icinga::SocketEventEngine::*&)(int), icinga::SocketEventEngine*&, int&, void>(void (icinga::SocketEventEngine::*&&&)(int), icinga::SocketEventEngine*&&&, int&&&) at type_traits:4264
frame #3: 0x00000001002722b7 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x00000001017184a8, __bound_args=0x00000001017184b8, __args=0x00007000067dbe48)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, 0ul, 1ul, std::__1::tuple<> >(void (icinga::SocketEventEngine::*&)(int), std::__1::tuple<icinga::SocketEventEngine*, int>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #4: 0x0000000100272260 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x00000001017184a8)(int), icinga::SocketEventEngine*, int&>::operator()<>() at functional:2257
frame #5: 0x000000010027222e check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x00000001017184a8)(int), icinga::SocketEventEngine*, int&> >(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&>&&) at type_traits:4323
frame #6: 0x000000010027220b check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(__t=0x00000001017184a0)(int), icinga::SocketEventEngine*, int&> >&, std::__1::__tuple_indices<>) at thread:342
frame #7: 0x00000001002721d5 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(__vp=0x00000001017184a0) at thread:352
frame #8: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #9: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #10: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #8
frame #0: 0x00007fff5a6ec372 libsystem_kernel.dylib`poll + 10
frame #1: 0x0000000100273568 check_nscp_api`icinga::SocketEventEnginePoll::ThreadProc(this=0x0000000101816200, tid=1) at socketevents-poll.cpp:88
frame #2: 0x0000000100272344 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101718f98, __a0=0x0000000101718fa8, __args=0x0000000101718fb0)).*fp(std::__1::forward<int&>(fp1))) std::__1::__invoke<void (icinga::SocketEventEngine::*&)(int), icinga::SocketEventEngine*&, int&, void>(void (icinga::SocketEventEngine::*&&&)(int), icinga::SocketEventEngine*&&&, int&&&) at type_traits:4264
frame #3: 0x00000001002722b7 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x0000000101718f98, __bound_args=0x0000000101718fa8, __args=0x000070000685ee48)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, 0ul, 1ul, std::__1::tuple<> >(void (icinga::SocketEventEngine::*&)(int), std::__1::tuple<icinga::SocketEventEngine*, int>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #4: 0x0000000100272260 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x0000000101718f98)(int), icinga::SocketEventEngine*, int&>::operator()<>() at functional:2257
frame #5: 0x000000010027222e check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101718f98)(int), icinga::SocketEventEngine*, int&> >(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&>&&) at type_traits:4323
frame #6: 0x000000010027220b check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(__t=0x0000000101718f90)(int), icinga::SocketEventEngine*, int&> >&, std::__1::__tuple_indices<>) at thread:342
frame #7: 0x00000001002721d5 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(__vp=0x0000000101718f90) at thread:352
frame #8: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #9: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #10: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #9
frame #0: 0x00007fff5a6ec372 libsystem_kernel.dylib`poll + 10
frame #1: 0x0000000100273568 check_nscp_api`icinga::SocketEventEnginePoll::ThreadProc(this=0x0000000101816200, tid=2) at socketevents-poll.cpp:88
frame #2: 0x0000000100272344 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101719038, __a0=0x0000000101719048, __args=0x0000000101719050)).*fp(std::__1::forward<int&>(fp1))) std::__1::__invoke<void (icinga::SocketEventEngine::*&)(int), icinga::SocketEventEngine*&, int&, void>(void (icinga::SocketEventEngine::*&&&)(int), icinga::SocketEventEngine*&&&, int&&&) at type_traits:4264
frame #3: 0x00000001002722b7 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x0000000101719038, __bound_args=0x0000000101719048, __args=0x00007000068e1e48)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, 0ul, 1ul, std::__1::tuple<> >(void (icinga::SocketEventEngine::*&)(int), std::__1::tuple<icinga::SocketEventEngine*, int>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #4: 0x0000000100272260 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x0000000101719038)(int), icinga::SocketEventEngine*, int&>::operator()<>() at functional:2257
frame #5: 0x000000010027222e check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101719038)(int), icinga::SocketEventEngine*, int&> >(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&>&&) at type_traits:4323
frame #6: 0x000000010027220b check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(__t=0x0000000101719030)(int), icinga::SocketEventEngine*, int&> >&, std::__1::__tuple_indices<>) at thread:342
frame #7: 0x00000001002721d5 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(__vp=0x0000000101719030) at thread:352
frame #8: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #9: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #10: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #10
frame #0: 0x00007fff5a6ec372 libsystem_kernel.dylib`poll + 10
frame #1: 0x0000000100273568 check_nscp_api`icinga::SocketEventEnginePoll::ThreadProc(this=0x0000000101816200, tid=3) at socketevents-poll.cpp:88
frame #2: 0x0000000100272344 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101719738, __a0=0x0000000101719748, __args=0x0000000101719750)).*fp(std::__1::forward<int&>(fp1))) std::__1::__invoke<void (icinga::SocketEventEngine::*&)(int), icinga::SocketEventEngine*&, int&, void>(void (icinga::SocketEventEngine::*&&&)(int), icinga::SocketEventEngine*&&&, int&&&) at type_traits:4264
frame #3: 0x00000001002722b7 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x0000000101719738, __bound_args=0x0000000101719748, __args=0x0000700006964e48)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, 0ul, 1ul, std::__1::tuple<> >(void (icinga::SocketEventEngine::*&)(int), std::__1::tuple<icinga::SocketEventEngine*, int>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #4: 0x0000000100272260 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x0000000101719738)(int), icinga::SocketEventEngine*, int&>::operator()<>() at functional:2257
frame #5: 0x000000010027222e check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101719738)(int), icinga::SocketEventEngine*, int&> >(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&>&&) at type_traits:4323
frame #6: 0x000000010027220b check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(__t=0x0000000101719730)(int), icinga::SocketEventEngine*, int&> >&, std::__1::__tuple_indices<>) at thread:342
frame #7: 0x00000001002721d5 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(__vp=0x0000000101719730) at thread:352
frame #8: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #9: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #10: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #11
frame #0: 0x00007fff5a6ec372 libsystem_kernel.dylib`poll + 10
frame #1: 0x0000000100273568 check_nscp_api`icinga::SocketEventEnginePoll::ThreadProc(this=0x0000000101816200, tid=4) at socketevents-poll.cpp:88
frame #2: 0x0000000100272344 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x00000001017197d8, __a0=0x00000001017197e8, __args=0x00000001017197f0)).*fp(std::__1::forward<int&>(fp1))) std::__1::__invoke<void (icinga::SocketEventEngine::*&)(int), icinga::SocketEventEngine*&, int&, void>(void (icinga::SocketEventEngine::*&&&)(int), icinga::SocketEventEngine*&&&, int&&&) at type_traits:4264
frame #3: 0x00000001002722b7 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x00000001017197d8, __bound_args=0x00000001017197e8, __args=0x00007000069e7e48)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, 0ul, 1ul, std::__1::tuple<> >(void (icinga::SocketEventEngine::*&)(int), std::__1::tuple<icinga::SocketEventEngine*, int>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #4: 0x0000000100272260 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x00000001017197d8)(int), icinga::SocketEventEngine*, int&>::operator()<>() at functional:2257
frame #5: 0x000000010027222e check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x00000001017197d8)(int), icinga::SocketEventEngine*, int&> >(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&>&&) at type_traits:4323
frame #6: 0x000000010027220b check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(__t=0x00000001017197d0)(int), icinga::SocketEventEngine*, int&> >&, std::__1::__tuple_indices<>) at thread:342
frame #7: 0x00000001002721d5 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(__vp=0x00000001017197d0) at thread:352
frame #8: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #9: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #10: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #12
frame #0: 0x00007fff5a6ec372 libsystem_kernel.dylib`poll + 10
frame #1: 0x0000000100273568 check_nscp_api`icinga::SocketEventEnginePoll::ThreadProc(this=0x0000000101816200, tid=5) at socketevents-poll.cpp:88
frame #2: 0x0000000100272344 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101719888, __a0=0x0000000101719898, __args=0x00000001017198a0)).*fp(std::__1::forward<int&>(fp1))) std::__1::__invoke<void (icinga::SocketEventEngine::*&)(int), icinga::SocketEventEngine*&, int&, void>(void (icinga::SocketEventEngine::*&&&)(int), icinga::SocketEventEngine*&&&, int&&&) at type_traits:4264
frame #3: 0x00000001002722b7 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x0000000101719888, __bound_args=0x0000000101719898, __args=0x0000700006a6ae48)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, 0ul, 1ul, std::__1::tuple<> >(void (icinga::SocketEventEngine::*&)(int), std::__1::tuple<icinga::SocketEventEngine*, int>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #4: 0x0000000100272260 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x0000000101719888)(int), icinga::SocketEventEngine*, int&>::operator()<>() at functional:2257
frame #5: 0x000000010027222e check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101719888)(int), icinga::SocketEventEngine*, int&> >(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&>&&) at type_traits:4323
frame #6: 0x000000010027220b check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(__t=0x0000000101719880)(int), icinga::SocketEventEngine*, int&> >&, std::__1::__tuple_indices<>) at thread:342
frame #7: 0x00000001002721d5 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(__vp=0x0000000101719880) at thread:352
frame #8: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #9: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #10: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #13
frame #0: 0x00007fff5a6ec372 libsystem_kernel.dylib`poll + 10
frame #1: 0x0000000100273568 check_nscp_api`icinga::SocketEventEnginePoll::ThreadProc(this=0x0000000101816200, tid=6) at socketevents-poll.cpp:88
frame #2: 0x0000000100272344 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101719938, __a0=0x0000000101719948, __args=0x0000000101719950)).*fp(std::__1::forward<int&>(fp1))) std::__1::__invoke<void (icinga::SocketEventEngine::*&)(int), icinga::SocketEventEngine*&, int&, void>(void (icinga::SocketEventEngine::*&&&)(int), icinga::SocketEventEngine*&&&, int&&&) at type_traits:4264
frame #3: 0x00000001002722b7 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x0000000101719938, __bound_args=0x0000000101719948, __args=0x0000700006aede48)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, 0ul, 1ul, std::__1::tuple<> >(void (icinga::SocketEventEngine::*&)(int), std::__1::tuple<icinga::SocketEventEngine*, int>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #4: 0x0000000100272260 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x0000000101719938)(int), icinga::SocketEventEngine*, int&>::operator()<>() at functional:2257
frame #5: 0x000000010027222e check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x0000000101719938)(int), icinga::SocketEventEngine*, int&> >(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&>&&) at type_traits:4323
frame #6: 0x000000010027220b check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(__t=0x0000000101719930)(int), icinga::SocketEventEngine*, int&> >&, std::__1::__tuple_indices<>) at thread:342
frame #7: 0x00000001002721d5 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(__vp=0x0000000101719930) at thread:352
frame #8: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #9: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #10: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
thread #14
frame #0: 0x00007fff5a6ec372 libsystem_kernel.dylib`poll + 10
frame #1: 0x0000000100273568 check_nscp_api`icinga::SocketEventEnginePoll::ThreadProc(this=0x0000000101816200, tid=7) at socketevents-poll.cpp:88
frame #2: 0x0000000100272344 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x00000001017186b8, __a0=0x00000001017186c8, __args=0x00000001017186d0)).*fp(std::__1::forward<int&>(fp1))) std::__1::__invoke<void (icinga::SocketEventEngine::*&)(int), icinga::SocketEventEngine*&, int&, void>(void (icinga::SocketEventEngine::*&&&)(int), icinga::SocketEventEngine*&&&, int&&&) at type_traits:4264
frame #3: 0x00000001002722b7 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (__f=0x00000001017186b8, __bound_args=0x00000001017186c8, __args=0x0000700006b70e48)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, 0ul, 1ul, std::__1::tuple<> >(void (icinga::SocketEventEngine::*&)(int), std::__1::tuple<icinga::SocketEventEngine*, int>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at functional:2224
frame #4: 0x0000000100272260 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] std::__1::__bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<>, __is_valid_bind_return<void (icinga::SocketEventEngine::*)(int), std::__1::tuple<icinga::SocketEventEngine*, int>, std::__1::tuple<> >::value>::type std::__1::__bind<void (this=0x00000001017186b8)(int), icinga::SocketEventEngine*, int&>::operator()<>() at functional:2257
frame #5: 0x000000010027222e check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] decltype(__f=0x00000001017186b8)(int), icinga::SocketEventEngine*, int&> >(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&>&&) at type_traits:4323
frame #6: 0x000000010027220b check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(void*) [inlined] void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> >(__t=0x00000001017186b0)(int), icinga::SocketEventEngine*, int&> >&, std::__1::__tuple_indices<>) at thread:342
frame #7: 0x00000001002721d5 check_nscp_api`void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::__bind<void (icinga::SocketEventEngine::*)(int), icinga::SocketEventEngine*, int&> > >(__vp=0x00000001017186b0) at thread:352
frame #8: 0x00007fff5a8266c1 libsystem_pthread.dylib`_pthread_body + 340
frame #9: 0x00007fff5a82656d libsystem_pthread.dylib`_pthread_start + 377
frame #10: 0x00007fff5a825c5d libsystem_pthread.dylib`thread_start + 13
(lldb)
I'm tempted to call wont-fix for the nscp issue since we want to move away from the icinga http client in that case. Is that a common issue? I didn't run into that when testing #6207
Crunsher
on 9 Apr 2018
On the long run, yes, maybe. For the short run, we cannot throw away check_nscp_api in 2.9 I believe. That being said, I want to release 2.9 quite soon after resolving remaining issues and implementing customer sponsored features.
I can reproduce the issue in 2 out of 5 queries. It might be the case that NSClient++ itself behaves wonky, but looking at the lldb stacktrace, the handler is waiting forever. I'll debug further, since the code is the exact same as the console command.
dnsmichi
on 9 Apr 2018
It's a problem with the underlying low level logic in the http client library. We can solve both problems with one patch, it's just someone has to write that patch > <
Crunsher
on 9 Apr 2018
Something broke on server side between 2.8.1 and 2.8.2. We never noticed a single hang in 2.8.1, but since upgrading to 2.8.2, like 2 out of 5 api queries hang indeed.
Correct that the client should have some timeout, but the server-side introduced some bug between 2.8.1 and 2.8.2 I think!
dupondje
on 9 Apr 2018
It definitely is a client problem, as check_nscp_api talks to NSClient++ and not Icinga 2.
dnsmichi
on 9 Apr 2018
We changed quite a bit for the security issues in 2.8.2 with the http server, it's possible this just made the issue visible.
Crunsher
on 9 Apr 2018
It is somewhere in HttpClientConnection::DataAvailableHandler() and SocketEvents where poll just hangs indefinitely. The callback for completion for the request is never called.
dnsmichi
on 9 Apr 2018
Happens inside m_Stream->Close() in DataAvailableHandler(). I've seen this before with holding a reference on its own.
(lldb) r
Process 4863 launched: '/usr/local/sbin/check_nscp_api' (x86_64)
Sending request to 'https://10.10.98.87:8443/query/check_nscp'
[2018-04-09 15:44:17 +0200] critical/HttpClientConnection: DataAvailableHandler: Before m_Stream->IsEof().
[2018-04-09 15:44:17 +0200] critical/HttpClientConnection: DataAvailableHandler: After m_Stream->IsEof().
[2018-04-09 15:44:17 +0200] critical/HttpClientConnection: DataAvailableHandler: Closing socket.
Haven't found a good solution or indication here, this all is _very_ asynchronous and a race condition. DataAvailableHandler() is only called if there's data and the stream is not EOF. Inside the function call, the stream is EOF and nothing is processed anymore (no call to ProcessMessage()) and as such the call just hangs indefinitely. I've tested this, it also happens with v2.8.1 but highly likely not so often. It may be the case that "corked" influences this thing.
I'm inclined to entirely drop the httpclientconnection thing, and just implement a light-weight variant with synchronous request from the InfluxDBWriter code. Though this doesn't help the console where longer script expressions may run into timeouts, it will only help check_nscp_api running standalone against NSClient++.
Still I'd love to find out why m_Stream->IsEof() is true aka why another IO thread is interfering here.
dnsmichi
on 10 Apr 2018
Hi,
do I understand this issue correctly that it is a client implementation problem in icinga2 console and not an API bug?
So it should be safe to update to 2.8.2 (or 2.8.4) when we use the API via HTTP?
Thank you in advance!
ekeih
on 26 Apr 2018
@ekeih Yes, this is only about the icinga2 console command. The http API is not affected.
Crunsher
on 26 Apr 2018
👍1
I'm pausing this for now. I rewrote the whole ApiClient twice but the issue is with the server too. After HttpRequest::ParseBody(), HttpServerConnection::ProcessMessage() is not called again (in some cases). I failed to understand as to why though and why it would work with curl (the new ApiClient is completely synchronous).
Crunsher
on 29 May 2018
As discussed w/ @Crunsher (offline):
I'll try Boost ASIO. If it succeeds, it will be an option for problems like #6361 in the future.
Al2Klimov
on 14 Jun 2018
Please don't use the last commit in production. Thanks. (However it seems to fix the problem.)
Al2Klimov
on 15 Jun 2018
Summary
There's no way we can use boost::asio or any other client library. That's an open end, and won't help here.
Problem Analysis
We've added debug logging to each stage within
- HttpServerConnection::DataAvailableHandler()
- HttpServerConnection::ProcessMessage()
- HttpRequest::ParseBody()
Chunked Encoding
Our HTTP clients use chunked encoding by default (this is hardcoded if HTTP/1.1 is selected).
curl does not use this by default which is why the error is not visible with that test method.
Flow
Data Available -> Process Message Loop -> Body not complete -> Parse Body (Chunked)
if (Headers->Get("transfer-encoding") == "chunked") {
calls HttpChunkedEncoding::ReadChunkFromStream which reads the chunked body.
Imagine sending just an empty JSON body
{}with the size of 2 characters.
1. Iteration
ParseBody() returns true, since size is not 0. This returns {} as data. CompleteBody is set to true, there's no more data expected afterwards.
2. Iteration
ParseBody() returns false, since size == 0, no more chunked data to read.
The Problem
When ParseBody() returns false in the last call, this is returned to ProcessMessage().
while (ProcessMessage())
This immediately breaks the loop. We never will jump into ProcessMessage() again.
if (!m_CurrentRequest.CompleteBody) {
gets skipped, to call the code parts below.
m_RequestQueue.Enqueue(std::bind(&HttpServerConnection::ProcessMessageAsync,
HttpServerConnection::Ptr(this), m_CurrentRequest, response, m_AuthenticatedUser));
The Solution
Once CompleteBody is completed, ParseBody() needs to return true to indicate the caller that one more ProcessMessage() call is needed. #6383 fixes a regression introduced in 2.8.2 in ee5954726d.
dnsmichi
on 18 Jun 2018
🎉3
Was this page helpful?
0 / 5 - 0 ratings
Related issues
shellscriptcode
路
6Comments
icinga-migration
路
6Comments
adamparker
路
5Comments
peteeckel
路
4Comments
xeiss
路
6Comments
Most helpful comment
Summary
There's no way we can use boost::asio or any other client library. That's an open end, and won't help here.
Problem Analysis
We've added debug logging to each stage within
Chunked Encoding
Our HTTP clients use chunked encoding by default (this is hardcoded if HTTP/1.1 is selected).
Flow
Data Available -> Process Message Loop -> Body not complete -> Parse Body (Chunked)
calls
HttpChunkedEncoding::ReadChunkFromStreamwhich reads the chunked body.1. Iteration
ParseBody() returns true, since size is not 0. This returns
{}as data.CompleteBodyis set to true, there's no more data expected afterwards.2. Iteration
ParseBody() returns false, since size == 0, no more chunked data to read.
The Problem
When ParseBody() returns false in the last call, this is returned to ProcessMessage().
This immediately breaks the loop. We never will jump into ProcessMessage() again.
gets skipped, to call the code parts below.
The Solution
Once
CompleteBodyis completed, ParseBody() needs to return true to indicate the caller that one more ProcessMessage() call is needed. #6383 fixes a regression introduced in 2.8.2 in ee5954726d.