Icinga2: Logrotate issues since 2.10.1-1

Created on 29 Oct 2018  ·  59Comments  ·  Source: Icinga/icinga2

Hey!

Since upgrading to 2.10.1-1 we face some issues with the logrotating part. The daemon sometime crashes after the logrotate run, this did not happen before. This is the crash report:

cat /var/log/icinga2/crash/report.1540790715.016844

Application version: r2.10.1-1

System information:
Platform: Debian GNU/Linux
Platform version: 9 (stretch)
Kernel: Linux
Kernel version: 4.9.0-6-amd64
Architecture: x86_64

Build information:
Compiler: GNU 6.3.0
Build host: 1880c6977367

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Stacktrace:

(0) libc.so.6: gsignal (+0xcf) [0x7fa1362c5fff]
(1) libc.so.6: abort (+0x16a) [0x7fa1362c742a]
(2) libc.so.6: <unknown function> (+0x70c00) [0x7fa136303c00]
(3) libc.so.6: <unknown function> (+0x76fc6) [0x7fa136309fc6]
(4) libc.so.6: <unknown function> (+0x7780e) [0x7fa13630a80e]
(5) icinga2: std::_Function_base::_Base_manager<std::_Bind<std::_Mem_fn<void (icinga::Timer::*)()> (boost::intrusive_ptr<icinga::Timer>)> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (+0x38) [0x563e339d0358]
(6) icinga2: icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (+0x4ca) [0x563e339c116a]
(7) libboost_thread.so.1.62.0: <unknown function> (+0x12116) [0x7fa138c1f116]
(8) libpthread.so.0: <unknown function> (+0x7494) [0x7fa137c41494]
(9) libc.so.6: clone (+0x3f) [0x7fa13637bacf]
  • This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
  • please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
  • information that might be useful in order to reproduce this problem.

Failed to launch GDB: No such file or directory

corcrash reIP
Source
picture sschindel
👍2

Most helpful comment

Thanks for sharing your findings. I'm leaving this as feedback for others to share more insights. This is a nasty one, on my TODO list and I am working to figure out the underlaying problem.

picture dnsmichi on 14 Feb 2019
❤1 👍1

All 59 comments

A description of the circumstances to reproduce this is required. You say it happens during log rotate, what makes you think that?

The log around this time would also be very welcome.

Crunsher picture Crunsher on 29 Oct 2018

We've got a stock Debian cron and logrotate setup, around 200 Hosts and randomly one or two of those hosts has this issue. Logrotate in in cron.daily, which gets executed at 06:25:

25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )

/var/log/icinga2# ls -l
total 1624
drwxr-s--x 3 nagios adm 4096 Mar 7 2017 compat
drwxr-s--x 2 nagios adm 4096 Oct 29 06:25 crash
-rw------- 1 nagios adm 0 Mar 7 2017 icinga2.err
-rw-r--r-- 1 nagios nagios 57755 Oct 29 10:35 icinga2.log
-rw-r--r-- 1 nagios nagios 1301697 Oct 29 06:25 icinga2.log.1
-rw-r--r-- 1 nagios nagios 48305 Oct 28 06:25 icinga2.log.2.gz
-rw-r--r-- 1 nagios nagios 42199 Oct 27 06:25 icinga2.log.3.gz
-rw-r--r-- 1 nagios nagios 42196 Oct 26 06:25 icinga2.log.4.gz
-rw-r--r-- 1 nagios nagios 42458 Oct 25 06:25 icinga2.log.5.gz
-rw-r--r-- 1 nagios nagios 41915 Oct 24 06:25 icinga2.log.6.gz
-rw-r--r-- 1 nagios nagios 43024 Oct 23 06:25 icinga2.log.7.gz
-rw-r--r-- 1 root adm 6361 Mar 7 2017 startup.log

The crash report is from that time:

stat /var/log/icinga2/crash/report.1540790715.016844

File: /var/log/icinga2/crash/report.1540790715.016844
Size: 2148 Blocks: 8 IO Block: 4096 regular file
Device: 802h/2050d Inode: 1711371 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 82/ nagios) Gid: ( 4/ adm)
Access: 2018-10-29 10:03:38.490438878 +0100
Modify: 2018-10-29 06:25:15.105257350 +0100
Change: 2018-10-29 06:25:15.105257350 +0100
Birth: -

And the last entry of the log files are:

[2018-10-29 06:23:17 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2018-10-29 06:23:25 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (30/min 150/5min 450/15min);
[2018-10-29 06:23:26 +0100] information/WorkQueue: #15 (JsonRpcConnection, #6) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2018-10-29 06:23:35 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (18/min 90/5min 270/15min);
[2018-10-29 06:23:36 +0100] information/WorkQueue: #9 (JsonRpcConnection, #0) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2018-10-29 06:23:37 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2018-10-29 06:23:45 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (12/min 60/5min 180/15min);
[2018-10-29 06:23:45 +0100] information/WorkQueue: #13 (JsonRpcConnection, #4) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2018-10-29 06:23:55 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:23:57 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2018-10-29 06:24:05 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (30/min 150/5min 450/15min);
[2018-10-29 06:24:06 +0100] information/WorkQueue: #11 (JsonRpcConnection, #2) items: 0, rate: 1.33333/s (80/min 400/5min 1200/15min);
[2018-10-29 06:24:07 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2018-10-29 06:24:15 +0100] information/WorkQueue: #16 (JsonRpcConnection, #7) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2018-10-29 06:24:15 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (36/min 180/5min 540/15min);
[2018-10-29 06:24:25 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (30/min 150/5min 450/15min);
[2018-10-29 06:24:35 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (18/min 90/5min 270/15min);
[2018-10-29 06:24:45 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (12/min 60/5min 180/15min);
[2018-10-29 06:24:55 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2018-10-29 06:25:05 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (30/min 150/5min 450/15min);

Do you need any further information?

sschindel picture sschindel on 29 Oct 2018
👍2

I think i got the same problem on a machine i host.
````
root@log:/home/user# cat /var/log/icinga2/crash/report.1540790722.611491
Application version: r2.10.1-1

System information:
Platform: Debian GNU/Linux
Platform version: 9 (stretch)
Kernel: Linux
Kernel version: 4.9.0-7-amd64
Architecture: x86_64

Build information:
Compiler: GNU 6.3.0
Build host: 1880c6977367

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Stacktrace:

(0) libc.so.6: gsignal (+0xcf) [0x7f3a80164fff]
(1) libc.so.6: abort (+0x16a) [0x7f3a8016642a]
(2) libc.so.6: <unknown function> (+0x70c00) [0x7f3a801a2c00]
(3) libc.so.6: <unknown function> (+0x76fc6) [0x7f3a801a8fc6]
(4) libc.so.6: <unknown function> (+0x7780e) [0x7f3a801a980e]
(5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa6) [0x55f96871f5c6]
(6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x55f9686d5f0a]
(7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x55f9686b33a2]
(8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x55f9688e7974]
(9) icinga2: icinga::Application::Run() (+0xc1) [0x55f9686b1351]
(10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x55f968860259]
(11) icinga2: <unknown function> (+0x266ef6) [0x55f9686b6ef6]
(12) icinga2: main (+0x49) [0x55f96865b299]
(13) libc.so.6: __libc_start_main (+0xf1) [0x7f3a801522e1]
(14) icinga2: _start (+0x2a) [0x55f96867881a]
  • This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
  • please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
  • information that might be useful in order to reproduce this problem.

Failed to launch GDB: No such file or directory
root@log:/home/user# stat /var/log/icinga2/crash/report.1540790722.611491
File: /var/log/icinga2/crash/report.1540790722.611491
Size: 2517 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 12323212 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 112/ nagios) Gid: ( 4/ adm)
Access: 2018-10-30 07:51:47.899211999 +0100
Modify: 2018-10-29 06:25:23.771421824 +0100
Change: 2018-10-29 06:25:23.771421824 +0100
Birth: -
````

Last log lines:
[2018-10-29 06:20:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:21:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:21:22 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:21:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:22:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:22:22 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:22:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:23:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:23:22 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:23:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:24:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:24:22 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:24:22 +0100] information/WorkQueue: #10 (JsonRpcConnection, #1) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:24:23 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:24:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:24:52 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:24:52 +0100] information/WorkQueue: #9 (JsonRpcConnection, #0) items: 0, rate: 0.3/s (18/min 90/5min 270/15min); [2018-10-29 06:24:52 +0100] information/WorkQueue: #13 (JsonRpcConnection, #4) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:24:53 +0100] information/WorkQueue: #15 (JsonRpcConnection, #6) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:25:02 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-10-29 06:25:02 +0100] information/WorkQueue: #11 (JsonRpcConnection, #2) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:25:12 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-10-29 06:25:19 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state' [2018-10-29 06:25:23 +0100] information/WorkQueue: #16 (JsonRpcConnection, #7) items: 0, rate: 0/s (0/min 0/5min 0/15min);
I happens randomly on other machine, just like @sschindel, but the stacktrace is not exactly the same... Can provide more info if it can help.

Oh, and about configuration, nothing has been manually changed. Only launched a # icinga2 node wizard to connect to central Icinga2 monitoring system.

Thank you very much.

Edit: High probability this is a double of #6736

fladna9 picture fladna9 on 30 Oct 2018

Are you using SysVinit or Systemd here? I'm thinking about a relationship to #6356.

dnsmichi picture dnsmichi on 30 Oct 2018

If it is just Debian 9, #6571 could come into play as well.

dnsmichi picture dnsmichi on 30 Oct 2018

@dnsmichi : Standard Debian Stretch install, so systemd :-)
I also have 3 or 4 Ubuntu VMs using systemd as well (compared to about 30+ Debian Stretch machines), but no crash so far.

Hope it helps.

fladna9 picture fladna9 on 30 Oct 2018

Hi,

we are also using Debian Stretch and had no issues with 2.9.2-1. This issue only appeared since upgrading to 2.10.1-1. We are using the packages from packges.icinga.com:

deb http://packages.icinga.com/debian icinga-stretch main

sschindel picture sschindel on 30 Oct 2018

I'm still using stable deb source.list, so still using the sometimes-crashing-on-my-vm version.

It did not happen for a week, but then it happened today again when logrotate was called again.
My Icinga2 (slave) node is out again. Strange thing: it is always the same machine.

We host about 40 VMs and hosts (35 Debian Stretch (Systemd), 1 Debian Jessie (Systemd), rest is Ubuntu (Systemd)), and only 1 is posing problems.

Exactly the same time as previously (6:25 AM CET, that is to say 5:25 AM UTC, some machines aren't on the same timezone, but i've taken care to take the correct log lines).

Here is my syslog:
image

Here is crash report:
````
root# cat /var/log/icinga2/crash/report.1541481925.521074
Application version: r2.10.1-1

System information:
Platform: Debian GNU/Linux
Platform version: 9 (stretch)
Kernel: Linux
Kernel version: 4.9.0-7-amd64
Architecture: x86_64

Build information:
Compiler: GNU 6.3.0
Build host: 1880c6977367

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Stacktrace:

(0) libc.so.6: gsignal (+0xcf) [0x7fc551ebdfff]
(1) libc.so.6: abort (+0x16a) [0x7fc551ebf42a]
(2) libc.so.6: <unknown function> (+0x70c00) [0x7fc551efbc00]
(3) libc.so.6: <unknown function> (+0x76fc6) [0x7fc551f01fc6]
(4) libc.so.6: <unknown function> (+0x7780e) [0x7fc551f0280e]
(5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa6) [0x562c3d7c25c6]
(6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x562c3d778f0a]
(7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x562c3d7563a2]
(8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x562c3d98a974]
(9) icinga2: icinga::Application::Run() (+0xc1) [0x562c3d754351]
(10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x562c3d903259]
(11) icinga2: <unknown function> (+0x266ef6) [0x562c3d759ef6]
(12) icinga2: main (+0x49) [0x562c3d6fe299]
(13) libc.so.6: __libc_start_main (+0xf1) [0x7fc551eab2e1]
(14) icinga2: _start (+0x2a) [0x562c3d71b81a]
  • This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
  • please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
  • information that might be useful in order to reproduce this problem.

Failed to launch GDB: No such file or directory
````

Here are the last /var/log/icinga2/icinga2.log.1file corresponding to before the crash.
[2018-11-06 06:20:18 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-06 06:20:36 +0100] information/WorkQueue: #16 (JsonRpcConnection, #7) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-06 06:20:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:20:56 +0100] information/WorkQueue: #11 (JsonRpcConnection, #2) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-06 06:20:56 +0100] information/WorkQueue: #15 (JsonRpcConnection, #6) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-06 06:20:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:21:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:21:36 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-06 06:21:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:21:56 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-06 06:21:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:22:06 +0100] information/WorkQueue: #10 (JsonRpcConnection, #1) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-06 06:22:16 +0100] information/WorkQueue: #9 (JsonRpcConnection, #0) items: 0, rate: 0.3/s (18/min 90/5min 270/15min); [2018-11-06 06:22:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:22:22 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state' [2018-11-06 06:22:28 +0100] information/WorkQueue: #13 (JsonRpcConnection, #4) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-06 06:22:38 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-06 06:22:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:22:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:23:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:23:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:23:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:24:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:24:38 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:24:58 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-06 06:25:18 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);

Here is the systemctl status icinga2 result
````
● icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/icinga2.service.d
└─limits.conf
Active: failed (Result: signal) since Tue 2018-11-06 06:25:25 CET; 2h 0min ago
Process: 9187 ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} (code=killed, signal=ABRT)
Process: 9177 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
Main PID: 9187 (code=killed, signal=ABRT)

Nov 06 06:25:25 log systemd[1]: icinga2.service: Main process exited, code=killed, status=6/ABRT
Nov 06 06:25:25 log systemd[1]: icinga2.service: Unit entered failed state.
Nov 06 06:25:25 log systemd[1]: icinga2.service: Failed with result 'signal'.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
````

Here is the /etc/logrotate.d/icinga2 file.
````
root# cat /etc/logrotate.d/icinga2
/var/log/icinga2/icinga2.log /var/log/icinga2/debug.log {
daily
rotate 7
su nagios nagios
compress
delaycompress
missingok
notifempty
create 644 nagios nagios
postrotate
/bin/kill -USR1 $(cat /run/icinga2/icinga2.pid 2> /dev/null) 2> /dev/null || true
endscript
}

/var/log/icinga2/error.log {
daily
rotate 90
su nagios nagios
compress
delaycompress
missingok
notifempty
create 644 nagios nagios
# TODO: figure out how to get Icinga to re-open this log file
}
````
Hope it helps.

To dev staff: Here to help/test things if needed ;-)

Edit: added systemctl status icinga2 output

fladna9 picture fladna9 on 6 Nov 2018

This looks like a wrong file descriptor handle, i.e. logrotate moves something away where Icinga still wants to write to. I suspect the close-stdio changes and flushing the streams to cause trouble here.

If anyone has time to lookup the code changes, I‘m out for OSMC this week.

dnsmichi picture dnsmichi on 6 Nov 2018

Hi,

I also have this issue, please see my crash log as well. This also includes the gdb output:

 Application version: r2.10.1-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-8-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 1880c6977367

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7fbf8b938fff]
    (1) libc.so.6: abort (+0x16a) [0x7fbf8b93a42a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7fbf8b976c00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7fbf8b97cfc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7fbf8b97d80e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa6) [0x560aed2475c6]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x560aed1fdf0a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x560aed1db3a2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x560aed40f974]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x560aed1d9351]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x560aed388259]
    (11) icinga2: <unknown function> (+0x266ef6) [0x560aed1deef6]
    (12) icinga2: main (+0x49) [0x560aed183299]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x7fbf8b9262e1]
    (14) icinga2: _start (+0x2a) [0x560aed1a081a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

[New LWP 31300]
[New LWP 31301]
[New LWP 31302]
[New LWP 31303]
[New LWP 31304]
[New LWP 31316]
[New LWP 31326]
[New LWP 31327]
[New LWP 31353]
[New LWP 31354]
[New LWP 31355]
[New LWP 31356]
[New LWP 31357]
[New LWP 31358]
[New LWP 31359]
[New LWP 31360]
[New LWP 31361]
[New LWP 31362]
[New LWP 31363]
[New LWP 31364]
[New LWP 31365]
[New LWP 31366]
[New LWP 31367]
[New LWP 31368]
[New LWP 31420]
[New LWP 31485]
[New LWP 31486]
[New LWP 31487]
[New LWP 31488]
[New LWP 6596]
[New LWP 20511]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fbf8d2bdb3a in __waitpid (pid=15301, stat_loc=0x7ffca6d8b7bc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29  ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.

Thread 32 (Thread 0x7fbf73910700 (LWP 20511)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73910700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73910700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460254365440, 4463560644109586558, 0, 140460254893855, 140460254365440, 140460061493968, -4427276064668987266, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 31 (Thread 0x7fbf7388e700 (LWP 6596)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7388e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7388e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460253832960, 4463560644109586558, 0, 140460254893855, 140460253832960, 140460061493184, -4427276132314722178, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 30 (Thread 0x7fbf73339700 (LWP 31488)):
#0  0x00007fbf8b9e567d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed2111ae in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73339700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73339700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460248241920, 4463560644109586558, 0, 140460253818223, 0, 140460707086400, -4427277410604363650, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 29 (Thread 0x7fbf7337a700 (LWP 31487)):
#0  0x00007fbf8b9e567d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed2111ae in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7337a700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7337a700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460248508160, 4463560644109586558, 0, 140460253818223, 0, 140460707086400, -4427277375707754370, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 28 (Thread 0x7fbf733bb700 (LWP 31486)):
#0  0x00007fbf8b9e567d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed2111ae in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf733bb700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf733bb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460248774400, 4463560644109586558, 0, 140460253818223, 0, 140460707086400, -4427277342958628738, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 27 (Thread 0x7fbf733fc700 (LWP 31485)):
#0  0x00007fbf8b9e567d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed2111ae in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf733fc700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf733fc700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460249040640, 4463560644109586558, 0, 140460253818223, 0, 140460707086400, -4427277308062019458, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 26 (Thread 0x7fbf7343d700 (LWP 31420)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed3179c8 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()
No symbol table info available.
#5  0x0000560aed31877b in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()
No symbol table info available.
#6  0x0000560aed26213e in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#7  0x0000560aed26ac85 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#8  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7343d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7343d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460249306880, 4463560644109586558, 0, 140460253820031, 0, 140460707086400, -4427277825068707714, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 25 (Thread 0x7fbf7347e700 (LWP 31368)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed26aebb in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7347e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7347e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460249573120, 4463560644109586558, 0, 140460253820927, 0, 140460707086400, -4427277790172098434, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 24 (Thread 0x7fbf734bf700 (LWP 31367)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed26aebb in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf734bf700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf734bf700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460249839360, 4463560644109586558, 0, 140460253820927, 0, 140460707086400, -4427277757422972802, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 23 (Thread 0x7fbf73500700 (LWP 31366)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed26aebb in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73500700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73500700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460250105600, 4463560644109586558, 0, 140460253820927, 0, 140460707086400, -4427277722526363522, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 22 (Thread 0x7fbf73541700 (LWP 31365)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed26aebb in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73541700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73541700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460250371840, 4463560644109586558, 0, 140460253820927, 0, 140460707086400, -4427277672597368706, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 21 (Thread 0x7fbf73582700 (LWP 31364)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed3179c8 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()
No symbol table info available.
#5  0x0000560aed31877b in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()
No symbol table info available.
#6  0x0000560aed26213e in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#7  0x0000560aed26ac85 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#8  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73582700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73582700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460250638080, 4463560644109586558, 0, 140460253820031, 0, 140460707086400, -4427277637700759426, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 20 (Thread 0x7fbf735c3700 (LWP 31363)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf735c3700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf735c3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460250904320, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427277604951633794, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 19 (Thread 0x7fbf73604700 (LWP 31362)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73604700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73604700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460251170560, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427278119810838402, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 18 (Thread 0x7fbf73645700 (LWP 31361)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73645700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73645700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460251436800, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427278087061712770, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 17 (Thread 0x7fbf73686700 (LWP 31360)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73686700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73686700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460251703040, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427278052165103490, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 16 (Thread 0x7fbf736c7700 (LWP 31359)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf736c7700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf736c7700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460251969280, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427278019415977858, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 15 (Thread 0x7fbf73708700 (LWP 31358)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73708700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73708700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460252235520, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427277984519368578, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 14 (Thread 0x7fbf73749700 (LWP 31357)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73749700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73749700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460252501760, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427277951770242946, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 13 (Thread 0x7fbf7378a700 (LWP 31356)):
#0  0x00007fbf8b9ef0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23f577 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7378a700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7378a700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460252768000, 4463560644109586558, 0, 140460253819983, 0, 140460707086400, -4427277916873633666, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 12 (Thread 0x7fbf737cb700 (LWP 31355)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf737cb700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf737cb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460253034240, 4463560644109586558, 0, 140460254893855, 0, 140460707086400, -4427277884124508034, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 11 (Thread 0x7fbf7380c700 (LWP 31354)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed241506 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7380c700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7380c700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460253300480, 4463560644109586558, 0, 140460254894383, 0, 140460707086400, -4427276199960457090, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 10 (Thread 0x7fbf7384d700 (LWP 31353)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf7384d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf7384d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460253566720, 4463560644109586558, 0, 140460254893743, 0, 140460707086400, -4427276167211331458, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 9 (Thread 0x7fbf73951700 (LWP 31327)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed498f7b in icinga::CheckerComponent::CheckThreadProc() ()
No symbol table info available.
#3  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73951700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73951700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460254631680, 4463560644109586558, 0, 140723107711823, 140460254631680, 94604956656800, -4427276014739992450, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 8 (Thread 0x7fbf73992700 (LWP 31326)):
#0  0x00007fbf8d2bd3ed in accept () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x0000560aed23af57 in icinga::Socket::Accept() ()
No symbol table info available.
#2  0x0000560aed2b9a97 in icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr<icinga::Socket> const&) ()
No symbol table info available.
#3  0x0000560aed370d06 in ?? ()
No symbol table info available.
#4  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#5  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf73992700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf73992700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460254897920, 4463560644109586558, 0, 140723107709919, 140460254897920, 94604956605632, -4427275979843383170, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 7 (Thread 0x7fbf739d3700 (LWP 31316)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
No locals.
#1  0x0000560aed242833 in icinga::Timer::TimerThreadProc() ()
No symbol table info available.
#2  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf739d3700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf739d3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460255164160, 4463560644109586558, 0, 140723107712367, 0, 140460707086400, -4427275947094257538, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 6 (Thread 0x7fbf8e71e700 (LWP 31304)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed241506 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007fbf8c279e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e71e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e71e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460705310464, 4463560644109586558, 0, 140723107732495, 0, 140460707086400, -4427359343011114882, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 5 (Thread 0x7fbf8e75f700 (LWP 31303)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed2ad41d in icinga::ApiListener::ApiTimerHandler() ()
No symbol table info available.
#5  0x0000560aed25e931 in ?? ()
No symbol table info available.
#6  0x0000560aed220a14 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x0000560aed2440d8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e75f700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e75f700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460705576704, 4463560644109586558, 0, 140723107731855, 0, 140460707086400, -4427359310261989250, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 4 (Thread 0x7fbf8e822700 (LWP 31302)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007fbf8d2b6c06 in __GI___pthread_mutex_lock (mutex=0x560aeed05d10) at ../nptl/pthread_mutex_lock.c:115
        id = -1926508708
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x0000560aed1ee5f5 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x0000560aed1fb388 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x0000560aed49773e in icinga::CheckerComponent::ResultTimerHandler() ()
No symbol table info available.
#5  0x0000560aed25e931 in ?? ()
No symbol table info available.
#6  0x0000560aed220a14 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x0000560aed2440d8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e822700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e822700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460706375424, 4463560644109586558, 0, 140723107731855, 140460706375424, 94604950740848, -4427357541272334210, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 3 (Thread 0x7fbf8e7e1700 (LWP 31301)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e7e1700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e7e1700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460706109184, 4463560644109586558, 0, 140723107731855, 140460706109184, 94604950813008, -4427359225436385154, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 2 (Thread 0x7fbf8e7a0700 (LWP 31300)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x0000560aed1c866b in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x0000560aed243f97 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007fbf8e292116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007fbf8d2b4494 in start_thread (arg=0x7fbf8e7a0700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fbf8e7a0700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140460705842944, 4463560644109586558, 0, 140723107731855, 140460705842944, 94604950813920, -4427359275365379970, -4427361040356159362}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007fbf8b9eeacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 1 (Thread 0x7fbf8e82e480 (LWP 31268)):
#0  0x00007fbf8d2bdb3a in __waitpid (pid=15301, stat_loc=0x7ffca6d8b7bc, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
        resultvar = 18446744073709551104
        sc_cancel_oldtype = 0
#1  0x0000560aed1b528e in icinga::Application::AttachDebugger(icinga::String const&, bool) ()
No symbol table info available.
#2  0x0000560aed1d99a7 in icinga::Application::SigAbrtHandler(int) ()
No symbol table info available.
#3  <signal handler called>
No locals.
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {__val = {0, 2314885530818453536, 8660248813382869024, 7596496373740942904, 3419760881481315694, 7310595060593224044, 3328211180835464301, 7090415180238960433, 3472328515408705638, 7293692169334238512, 8223625903103632184, 3472328295963438455, 4192904167887482928, 7365366848950382640, 3472665881546876514, 4064043904900345904}}
        pid = <optimized out>
        tid = <optimized out>
#5  0x00007fbf8b93a42a in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x756e672d78756e69, sa_sigaction = 0x756e672d78756e69}, sa_mask = {__val = {3761119431852583983, 7377571526170604334, 3472328301395600696, 4063716251160426285, 8606977229197423204, 3472328296226648109, 3475143045726351408, 7378645556122361904, 3472331625700155747, 3918522442241748272, 8223625903103818084, 3472328295963438455, 4192904167887482928, 2314885531086893104, 2314885530818453536, 140723107711424}}, sa_flags = 123, sa_restorer = 0x7ffca6d8c9c0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#6  0x00007fbf8b976c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fbf8ba6bd98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
        ap = {{gp_offset = 40, fp_offset = 32703, overflow_arg_area = 0x7ffca6d8c9d0, reg_save_area = 0x7ffca6d8c960}}
        fd = 2
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#7  0x00007fbf8b97cfc6 in malloc_printerr (action=3, str=0x7fbf8ba6bef0 "double free or corruption (fasttop)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
        buf = "0000560aeed709c0"
        cp = <optimized out>
        ar_ptr = <optimized out>
        ptr = <optimized out>
        str = 0x7fbf8ba6bef0 "double free or corruption (fasttop)"
        action = 3
#8  0x00007fbf8b97d80e in _int_free (av=0x7fbf8bc9fb00 <main_arena>, p=0x560aeed709b0, have_lock=0) at malloc.c:3905
        size = <optimized out>
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = <optimized out>
        locked = <optimized out>
        __func__ = "_int_free"
#9  0x0000560aed2475c6 in icinga::StreamLogger::BindStream(std::ostream*, bool) ()
No symbol table info available.
#10 0x0000560aed1fdf0a in icinga::FileLogger::ReopenLogFile() ()
No symbol table info available.
#11 0x0000560aed1db3a2 in icinga::Application::RunEventLoop() ()
No symbol table info available.
#12 0x0000560aed40f974 in icinga::IcingaApplication::Main() ()
No symbol table info available.
#13 0x0000560aed1d9351 in icinga::Application::Run() ()
No symbol table info available.
#14 0x0000560aed388259 in icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const ()
No symbol table info available.
#15 0x0000560aed1deef6 in ?? ()
No symbol table info available.
#16 0x0000560aed183299 in main ()
No symbol table info available.

After the crash a new icinga2.log is already created:

root@icingasql02:~# ls -lah /var/log/icinga2/
total 1.1M
drwxr-s--x  4 nagios adm    4.0K Nov  7 06:25 .
drwxr-xr-x 14 root   root   4.0K Nov  7 06:25 ..
drwxr-s--x  3 nagios adm    4.0K Oct 17  2017 compat
drwxr-s--x  2 nagios adm    4.0K Nov  7 06:25 crash
-rw-r--r--  1 nagios nagios    0 Nov  7 06:25 icinga2.log
-rw-r--r--  1 nagios nagios 838K Nov  7 06:25 icinga2.log.1
-rw-r--r--  1 nagios nagios  29K Nov  6 06:25 icinga2.log.2.gz
-rw-r--r--  1 nagios nagios  26K Nov  5 06:25 icinga2.log.3.gz
-rw-r--r--  1 nagios nagios  26K Nov  4 06:25 icinga2.log.4.gz
-rw-r--r--  1 nagios nagios  26K Nov  3 06:24 icinga2.log.5.gz
-rw-r--r--  1 nagios nagios  30K Nov  2 06:24 icinga2.log.6.gz
-rw-r--r--  1 nagios nagios  30K Nov  1 06:24 icinga2.log.7.gz

But as you can see nothing is written to it yet. When I start up the Icinga2 daemon again after the crash it starts writing to icinga2.log

NielsH picture NielsH on 7 Nov 2018

Can you try the snapshot packages? In #6736 a bug was fixed which seems similar to this.

mcktr picture mcktr on 7 Nov 2018

Hello @mcktr,
Is snapshot deb repository stable enough to go on a production server?
Because my crashing machine is an internal-but-important-on-production server...

fladna9 picture fladna9 on 7 Nov 2018

@mcktr the changes you are referring to are solely cluster related, nothing which points to the stream logger class. I've fixed a possible double free as see in the gdb stacktrace yesterday, please test them in the snapshot packages once being built. On a related note, right now the master solely contains bugfixes and no radical changes, so it is safe to assume it works.

dnsmichi picture dnsmichi on 9 Nov 2018

Thus far, I am not able to reproduce this with 2.10.1 on Debian Stretch.

wget -O - https://packages.icinga.com/icinga.key | apt-key add -

DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release); \
 echo "deb http://packages.icinga.com/debian icinga-${DIST} main" > \
 /etc/apt/sources.list.d/${DIST}-icinga.list
 echo "deb-src http://packages.icinga.com/debian icinga-${DIST} main" >> \
 /etc/apt/sources.list.d/${DIST}-icinga.list

apt-get update

apt-get install -y icinga2 vim


vim /etc/logrotate.d/icinga2

/var/log/icinga2/icinga2.log /var/log/icinga2/debug.log {
    size 1K
    rotate 7
    su nagios nagios
    compress
    delaycompress
    missingok
    notifempty
    create 644 nagios nagios
    postrotate
        /bin/kill -USR1 $(cat /run/icinga2/icinga2.pid 2> /dev/null) 2> /dev/null || true
    endscript
}

/var/log/icinga2/error.log {
    size 1K
    rotate 90
    su nagios nagios
    compress
    delaycompress
    missingok
    notifempty
    create 644 nagios nagios
    # TODO: figure out how to get Icinga to re-open this log file
}

logrotate -d /etc/logrotate.d/icinga2
dnsmichi picture dnsmichi on 12 Nov 2018

Has this already been released? There was an update some days ago, but this one still does not fix that issue.

sschindel picture sschindel on 26 Nov 2018

@sschindel Same, crashed this morning as well.

fladna9 picture fladna9 on 26 Nov 2018

Please add more logs from such crashes and everything else which helps reproduce the issue. I wasn't able to reproduce this yet.

dnsmichi picture dnsmichi on 26 Nov 2018

Today's logs

File /var/log/daemon.log.1
Nov 26 06:25:46 log systemd[1]: Reloading The Apache HTTP Server. Nov 26 06:25:48 log apachectl[11749]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message Nov 26 06:25:48 log systemd[1]: Reloaded The Apache HTTP Server. Nov 26 06:25:59 log fail2ban.server[30559]: INFO flush performed on SYSLOG Nov 26 06:26:03 log systemd[1]: icinga2.service: Main process exited, code=killed, status=11/SEGV Nov 26 06:26:03 log systemd[1]: icinga2.service: Unit entered failed state. Nov 26 06:26:03 log systemd[1]: icinga2.service: Failed with result 'signal'.

File /var/log/icinga2/icinga2.log.1
[2018-11-26 06:24:37 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-26 06:24:43 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-26 06:24:44 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-26 06:25:13 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-26 06:25:14 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-26 06:25:23 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-26 06:25:43 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-11-26 06:25:43 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-11-26 06:25:43 +0100] information/WorkQueue: #15 (JsonRpcConnection, #6) items: 0, rate: 0/s (0/min 0/5min 0/15min);

File /var/log/icinga2/crash/report.1541741165.911804
````
Application version: r2.10.1-1

System information:
Platform: Debian GNU/Linux
Platform version: 9 (stretch)
Kernel: Linux
Kernel version: 4.9.0-7-amd64
Architecture: x86_64

Build information:
Compiler: GNU 6.3.0
Build host: 1880c6977367

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Stacktrace:

(0) libc.so.6: gsignal (+0xcf) [0x7f20779d7fff]
(1) libc.so.6: abort (+0x16a) [0x7f20779d942a]
(2) libc.so.6: <unknown function> (+0x70c00) [0x7f2077a15c00]
(3) libc.so.6: <unknown function> (+0x76fc6) [0x7f2077a1bfc6]
(4) libc.so.6: <unknown function> (+0x7780e) [0x7f2077a1c80e]
(5) icinga2: std::_Function_base::_Base_manager<std::_Bind<std::_Mem_fn<void (icinga::Timer::*)()> (boost::intrusive_ptr<icinga::Timer>)> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) (+0x38) [0x55a98c8d6358]
(6) icinga2: icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (+0x4ca) [0x55a98c8c716a]
(7) libboost_thread.so.1.62.0: <unknown function> (+0x12116) [0x7f207a331116]
(8) libpthread.so.0: <unknown function> (+0x7494) [0x7f2079353494]
(9) libc.so.6: clone (+0x3f) [0x7f2077a8dacf]
  • This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
  • please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
  • information that might be useful in order to reproduce this problem.

Failed to launch GDB: No such file or directory
````

File /var/log/messages.1
Nov 26 06:26:01 log kernel: [5674078.627855] traps: icinga2[4528] general protection ip:7f5712a4025b sp:7ffcaa2ff590 error:0 Nov 26 06:26:01 log kernel: [5674078.627860] in libc-2.24.so[7f57129c9000+195000] Nov 26 06:26:05 log liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="11297" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

File /var/log/syslog.1
Nov 26 06:26:01 log kernel: [5674078.627855] traps: icinga2[4528] general protection ip:7f5712a4025b sp:7ffcaa2ff590 error:0 Nov 26 06:26:01 log kernel: [5674078.627860] in libc-2.24.so[7f57129c9000+195000] Nov 26 06:26:03 log systemd[1]: icinga2.service: Main process exited, code=killed, status=11/SEGV Nov 26 06:26:03 log systemd[1]: icinga2.service: Unit entered failed state. Nov 26 06:26:03 log systemd[1]: icinga2.service: Failed with result 'signal'. Nov 26 06:26:05 log liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="11297" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

File /var/log/apt/history.log
Start-Date: 2018-11-20 11:14:37 Commandline: apt full-upgrade Requested-By: user (1000) Upgrade: python3-josepy:amd64 (1.0.1-1~bpo9+1, 1.1.0-2~bpo9+1), icinga2-bin:amd64 (2.10.1-1.stretch, 2.10.2-1.stretch), certbot:amd64 (0.25.0-1~bpo9+1, 0.28.0-1~bpo9+1), icinga2-doc:amd64 (2.10.1-1.stretch, 2.10.2-1.stretch), python3-acme:amd64 (0.25.1-1~bpo9+1, 0.28.0-1~bpo9+1), python3-certbot-apache:amd64 (0.25.0-2~bpo9+1, 0.28.0-1~bpo9+1), icinga2-common:amd64 (2.10.1-1.stretch, 2.10.2-1.stretch), libmariadbclient18:amd64 (10.1.26-0+deb9u1, 10.1.37-0+deb9u1), python3-certbot:amd64 (0.25.0-1~bpo9+1, 0.28.0-1~bpo9+1), python-certbot-apache:amd64 (0.25.0-2~bpo9+1, 0.28.0-1~bpo9+1), icinga2:amd64 (2.10.1-1.stretch, 2.10.2-1.stretch) End-Date: 2018-11-20 11:16:08

fladna9 picture fladna9 on 26 Nov 2018

Does 2.10.2 change something here?

dnsmichi picture dnsmichi on 26 Nov 2018

@dnsmichi : Doesn't seem so, because it crashed this morning.

How to reproduce: good question. Installed Debian Stretch, followed the GrayLog setup tutorial.
Nothing really special. As told many times before, i manage +30 Debian/Ubuntu machines, and this one is the only one having this problem.
All have latest Stable version (as many are production servers).

fladna9 picture fladna9 on 26 Nov 2018

Application version: r2.10.1-1 is older than 2.10.2.

dnsmichi picture dnsmichi on 26 Nov 2018
👍1

Oh, it seems i forgot to restart icinga2 service after update.

Since it crashed this morning, i'll have to wait for it to maybe crash next week.

fladna9 picture fladna9 on 26 Nov 2018

Ahh, indeed it crashed with an older version. Seems like the package upgrade did not come with a daemon restart hook. I restarted all icinga instances on all my boxes... I will you give some feedback if this issue still occurs or not.

sschindel picture sschindel on 26 Nov 2018

We haven't had the issues for a little while (2-3 weeks) but just before the weekend this started again on the same server. In addition, we also had the first server running Debian 8 crashing on this. This was a new server that was installed with 2.10.2-1.jessie
Logs show:

cat /var/log/icinga2/error.log 
icinga2: /usr/include/boost/signals2/detail/lwm_pthreads.hpp:60: void boost::signals2::mutex::lock(): Assertion `pthread_mutex_lock(&m_) == 0' failed.
Caught SIGABRT.
Current time: 2018-11-26 06:25:17 +0100

Also crash report:

cat /var/log/icinga2/crash/report.1543209917.836953 
  Application version: r2.10.2-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 8 (jessie)
  Kernel: Linux
  Kernel version: 3.16.0-7-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.9.2
  Build host: bc2295102b6c

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0x37) [0x2af59a8e9067]
    (1) libc.so.6: abort (+0x148) [0x2af59a8ea448]
    (2) libc.so.6: <unknown function> (+0x2e266) [0x2af59a8e2266]
    (3) libc.so.6: <unknown function> (+0x2e312) [0x2af59a8e2312]
    (4) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x581c9a]
    (5) icinga2: icinga::Timer::~Timer() (+0x112) [0x5f3bb2]
    (6) icinga2: icinga::Timer::~Timer() (+0x9) [0x5f3bd9]
    (7) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa9) [0x5fd399]
    (8) icinga2: icinga::FileLogger::ReopenLogFile() (+0x134) [0x66db24]
    (9) icinga2: icinga::Application::RunEventLoop() (+0x482) [0x5cc632]
    (10) icinga2: icinga::IcingaApplication::Main() (+0x261) [0x7fdfe1]
    (11) icinga2: icinga::Application::Run() (+0xb4) [0x5c7af4]
    (12) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::string, std::allocator<std::string> > const&) const (+0xebc) [0x7d91fc]
    (13) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x5d4d66]
    (14) icinga2: main (+0x47) [0x5830a7]
    (15) libc.so.6: __libc_start_main (+0xf5) [0x2af59a8d5b45]
    (16) /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2() [0x5904f7]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory

I have now installed gdb on the server so if it crashes again I can sent a gdb report.

Also, regarding the Debian 9 server that I also reported earlier - I do think something changed with the new version, but I'm a bit reluctant reporting it since I am not 100% sure and don't wanna give false signals.
What I noticed here is that when it crashed, it was disconnected from the Icinga monitoring system (endpoints not connected), but the daemon was still running. I thought it crashed again so I restarted the daemon but only later saw that the daemon was still running. It refused to restart and after a systemd timeout it sent a SIGKILL to the Icinga Daemon and restarted it properly.

I was a bit too soon restarting here on Debian 9, otherwise I'd have investigated more. I thought it was not running when I gave the restart command, but it was... so it might've been possible to get more debug information from it.
If it happens again I'll try to get more information. it might also have been a one-time-thing since previously it has always completely crashed, and not keep running but simply disconnected.

NielsH picture NielsH on 26 Nov 2018

We have still the same issue:

# cat /var/log/icinga2/crash/report.1543469122.888419
  Application version: r2.10.2-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-6-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 486e413fb159

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7fec4a6a0fff]
    (1) libc.so.6: abort (+0x16a) [0x7fec4a6a242a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7fec4a6dec00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7fec4a6e4fc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7fec4a6e580e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa4) [0x55e06f43c1b4]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x55e06f3f145a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x55e06f3ce8f2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x55e06f5d7b64]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x55e06f3cc8a1]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x55e06f5a7c19]
    (11) icinga2: <unknown function> (+0x266446) [0x55e06f3d2446]
    (12) icinga2: main (+0x49) [0x55e06f376889]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x7fec4a68e2e1]
    (14) icinga2: _start (+0x2a) [0x55e06f393d6a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory
sschindel picture sschindel on 29 Nov 2018

Ok, thanks. So this problem only affects Debian 9 Stretch thus far. I would expect that everyone reporting these problems is using Systemd, right?

dnsmichi picture dnsmichi on 29 Nov 2018

@dnsmichi So-far I've also seen it on a single Debian 8 machine; see: https://github.com/Icinga/icinga2/issues/6737#issuecomment-441616058

NielsH picture NielsH on 29 Nov 2018

Ok. Again, with Systemd or not?

dnsmichi picture dnsmichi on 29 Nov 2018

Yes, systemd. That server was a new install with no modifications to the init system.

NielsH picture NielsH on 29 Nov 2018

Systemd as well, as it is included as default in Debian.

fladna9 picture fladna9 on 29 Nov 2018

Systemd here as well.

sschindel picture sschindel on 30 Nov 2018

Just had another crash on Debian 9, previously I mentioned I thought the daemon kept running on Debian 9 but no longer connected to the endpoint, however this is not the case now.

Please see the crash report:

cat crash/report.1543814713.893191
  Application version: r2.10.2-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-8-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 486e413fb159

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x7f9cbf3b5fff]
    (1) libc.so.6: abort (+0x16a) [0x7f9cbf3b742a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x7f9cbf3f3c00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x7f9cbf3f9fc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x7f9cbf3fa80e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa4) [0x5642d47aa1b4]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x5642d475f45a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x5642d473c8f2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x5642d4945b64]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x5642d473a8a1]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x5642d4915c19]
    (11) icinga2: <unknown function> (+0x266446) [0x5642d4740446]
    (12) icinga2: main (+0x49) [0x5642d46e4889]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x7f9cbf3a32e1]
    (14) icinga2: _start (+0x2a) [0x5642d4701d6a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

[New LWP 28461]
[New LWP 28462]
[New LWP 28463]
[New LWP 28464]
[New LWP 28465]
[New LWP 28473]
[New LWP 28478]
[New LWP 28479]
[New LWP 28514]
[New LWP 28516]
[New LWP 28517]
[New LWP 28518]
[New LWP 28519]
[New LWP 28520]
[New LWP 28521]
[New LWP 28522]
[New LWP 28523]
[New LWP 28524]
[New LWP 28525]
[New LWP 28526]
[New LWP 28527]
[New LWP 28528]
[New LWP 28529]
[New LWP 28530]
[New LWP 28531]
[New LWP 28542]
[New LWP 28599]
[New LWP 28600]
[New LWP 28601]
[New LWP 28602]
[New LWP 29089]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f9cc0d3ab3a in __waitpid (pid=29734, stat_loc=0x7fff89dd787c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29  ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.

Thread 32 (Thread 0x7f9cbb3aa700 (LWP 29089)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb3aa700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb3aa700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311132808960, 9004287024282259819, 0, 140311133337375, 140311132808960, 140310676981584, -8949548412424892053, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 31 (Thread 0x7f9cbadd3700 (LWP 28602)):
#0  0x00007f9cbf46267d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47726fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbadd3700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbadd3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311126685440, 9004287024282259819, 0, 140311132794223, 0, 140311249866816, -8949546440498032277, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 30 (Thread 0x7f9cbae14700 (LWP 28601)):
#0  0x00007f9cbf46267d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47726fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbae14700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbae14700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311126951680, 9004287024282259819, 0, 140311132794223, 0, 140311249866816, -8949546817918283413, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 29 (Thread 0x7f9cbae55700 (LWP 28600)):
#0  0x00007f9cbf46267d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47726fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbae55700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbae55700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311127217920, 9004287024282259819, 0, 140311132794223, 0, 140311249866816, -8949546787316641429, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 28 (Thread 0x7f9cbae96700 (LWP 28599)):
#0  0x00007f9cbf46267d in poll () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47726fe in icinga::Process::IOThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbae96700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbae96700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311127484160, 9004287024282259819, 0, 140311132794223, 0, 140311249866816, -8949546889858985621, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 27 (Thread 0x7f9cbaed7700 (LWP 28542)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d47fb718 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()
No symbol table info available.
#5  0x00005642d47fc4cb in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()
No symbol table info available.
#6  0x00005642d47c5fce in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#7  0x00005642d47cd565 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbaed7700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbaed7700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311127750400, 9004287024282259819, 0, 140311131464895, 0, 140311249866816, -8949546854962376341, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 26 (Thread 0x7f9cbaf18700 (LWP 28531)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47cd79b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbaf18700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbaf18700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311128016640, 9004287024282259819, 0, 140311132796927, 0, 140311249866816, -8949546682626813589, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 25 (Thread 0x7f9cbaf59700 (LWP 28530)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47cd79b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbaf59700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbaf59700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311128282880, 9004287024282259819, 0, 140311132796927, 0, 140311249866816, -8949546652025171605, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 24 (Thread 0x7f9cbaf9a700 (LWP 28529)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47cd79b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbaf9a700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbaf9a700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311128549120, 9004287024282259819, 0, 140311132796927, 0, 140311249866816, -8949546754567515797, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 23 (Thread 0x7f9cbafdb700 (LWP 28528)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47cd79b in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbafdb700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbafdb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311128815360, 9004287024282259819, 0, 140311132796927, 0, 140311249866816, -8949546719670906517, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 22 (Thread 0x7f9cbb01c700 (LWP 28527)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d47fb718 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()
No symbol table info available.
#5  0x00005642d47fc4cb in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()
No symbol table info available.
#6  0x00005642d47c5fce in icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) ()
No symbol table info available.
#7  0x00005642d47cd565 in icinga::WorkQueue::WorkerThreadProc() ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb01c700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb01c700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311129081600, 9004287024282259819, 0, 140311132796095, 0, 140311249866816, -8949548196602785429, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 21 (Thread 0x7f9cbb05d700 (LWP 28526)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb05d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb05d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311129347840, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548166001143445, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 20 (Thread 0x7f9cbb09e700 (LWP 28525)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb09e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb09e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311129614080, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548268543487637, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 19 (Thread 0x7f9cbb0df700 (LWP 28524)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb0df700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb0df700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311129880320, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548233646878357, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 18 (Thread 0x7f9cbb120700 (LWP 28523)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb120700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb120700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311130146560, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548061311315605, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 17 (Thread 0x7f9cbb161700 (LWP 28522)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb161700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb161700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311130412800, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548030709673621, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 16 (Thread 0x7f9cbb1a2700 (LWP 28521)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb1a2700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb1a2700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311130679040, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548133252017813, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 15 (Thread 0x7f9cbb1e3700 (LWP 28520)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb1e3700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb1e3700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311130945280, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548098355408533, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 14 (Thread 0x7f9cbb224700 (LWP 28519)):
#0  0x00007f9cbf46c0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d47a2197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb224700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb224700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311131211520, 9004287024282259819, 0, 140311132795983, 0, 140311249866816, -8949548475775659669, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 13 (Thread 0x7f9cbb265700 (LWP 28518)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb265700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb265700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311131477760, 9004287024282259819, 0, 140311133337375, 0, 140311249866816, -8949548445174017685, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 12 (Thread 0x7f9cbb2a6700 (LWP 28517)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d47a4126 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb2a6700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb2a6700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311131744000, 9004287024282259819, 0, 140311133337903, 0, 140311249866816, -8949548547716361877, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 11 (Thread 0x7f9cbb2e7700 (LWP 28516)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb2e7700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb2e7700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311132010240, 9004287024282259819, 0, 140311133337263, 0, 140311249866816, -8949548512819752597, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 10 (Thread 0x7f9cbb369700 (LWP 28514)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb369700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb369700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311132542720, 9004287024282259819, 0, 140311133337263, 0, 140311249866816, -8949548309882547861, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 9 (Thread 0x7f9cbb3eb700 (LWP 28479)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d4a4ea4b in icinga::CheckerComponent::CheckThreadProc() ()
No symbol table info available.
#3  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb3eb700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb3eb700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311133075200, 9004287024282259819, 0, 140735506385967, 140311133075200, 94845075171552, -8949548377528282773, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 8 (Thread 0x7f9cbb42c700 (LWP 28478)):
#0  0x00007f9cc0d3a3ed in accept () at ../sysdeps/unix/syscall-template.S:84
No locals.
#1  0x00005642d479db77 in icinga::Socket::Accept() ()
No symbol table info available.
#2  0x00005642d489c0c7 in icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr<icinga::Socket> const&) ()
No symbol table info available.
#3  0x00005642d48fdb96 in ?? ()
No symbol table info available.
#4  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#5  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb42c700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb42c700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311133341440, 9004287024282259819, 0, 140735506384063, 140311133341440, 94845075098336, -8949547655436906133, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#6  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 7 (Thread 0x7f9cbb46d700 (LWP 28473)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
No locals.
#1  0x00005642d47a5453 in icinga::Timer::TimerThreadProc() ()
No symbol table info available.
#2  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f9cc0d31494 in start_thread (arg=0x7f9cbb46d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cbb46d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311133607680, 9004287024282259819, 0, 140735506386511, 0, 140311249866816, -8949547624835264149, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#4  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 6 (Thread 0x7f9cc219b700 (LWP 28465)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d47a4126 in icinga::ThreadPool::ManagerThreadProc() ()
No symbol table info available.
#5  0x00007f9cbfcf6e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc219b700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc219b700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311248090880, 9004287024282259819, 0, 140735506406639, 0, 140311249866816, -8949739443443412629, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#7  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 5 (Thread 0x7f9cc21dc700 (LWP 28464)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00005642d4729bbb in boost::condition_variable::wait(boost::unique_lock<boost::mutex>&) ()
No symbol table info available.
#2  0x00005642d47a6bb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#3  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#4  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc21dc700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc21dc700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311248357120, 9004287024282259819, 0, 140735506405999, 0, 140311249866816, -8949739408546803349, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 4 (Thread 0x7f9cc229f700 (LWP 28463)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d4a4d71e in icinga::CheckerComponent::ResultTimerHandler() ()
No symbol table info available.
#5  0x00005642d47c29c1 in ?? ()
No symbol table info available.
#6  0x00005642d4781e74 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x00005642d47a6cf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc229f700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc229f700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311249155840, 9004287024282259819, 0, 140735506405999, 140311249155840, 94845069307328, -8949739857907756693, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 3 (Thread 0x7f9cc225e700 (LWP 28462)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d475c8d8 in icinga::Log::~Log() ()
No symbol table info available.
#4  0x00005642d48895d2 in icinga::ApiListener::ApiReconnectTimerHandler() ()
No symbol table info available.
#5  0x00005642d47c29c1 in ?? ()
No symbol table info available.
#6  0x00005642d4781e74 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x00005642d47a6cf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc225e700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc225e700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311248889600, 9004287024282259819, 0, 140735506405999, 140311248889600, 94845069379488, -8949739755365412501, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 2 (Thread 0x7f9cc221d700 (LWP 28461)):
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
No locals.
#1  0x00007f9cc0d33c06 in __GI___pthread_mutex_lock (mutex=0x5642d7084270) at ../nptl/pthread_mutex_lock.c:115
        id = -1059872932
        __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
        type = 1
        id = <optimized out>
#2  0x00005642d474fb45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()
No symbol table info available.
#3  0x00005642d471f5a9 in icinga::ConfigObject::SetAuthority(bool) ()
No symbol table info available.
#4  0x00005642d48964dc in icinga::ApiListener::UpdateObjectAuthority() ()
No symbol table info available.
#5  0x00005642d47c29c1 in ?? ()
No symbol table info available.
#6  0x00005642d4781e74 in icinga::Timer::Call() ()
No symbol table info available.
#7  0x00005642d47a6cf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()
No symbol table info available.
#8  0x00007f9cc1d0f116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
No symbol table info available.
#9  0x00007f9cc0d31494 in start_thread (arg=0x7f9cc221d700) at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7f9cc221d700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140311248623360, 9004287024282259819, 0, 140735506405999, 140311248623360, 94845069380400, -8949739790262021781, -8949735540098674325}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#10 0x00007f9cbf46bacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

Thread 1 (Thread 0x7f9cc22ab480 (LWP 28417)):
#0  0x00007f9cc0d3ab3a in __waitpid (pid=29734, stat_loc=0x7fff89dd787c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
        resultvar = 18446744073709551104
        sc_cancel_oldtype = 0
#1  0x00005642d47167de in icinga::Application::AttachDebugger(icinga::String const&, bool) ()
No symbol table info available.
#2  0x00005642d473aef7 in icinga::Application::SigAbrtHandler(int) ()
No symbol table info available.
#3  <signal handler called>
No locals.
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = {__val = {0, 2314885530818453536, 8660248813382869024, 7596496373740942904, 3419760881481315694, 7310595060593224044, 3328211180835464301, 4136053824683915057, 3472385483767702371, 3630854822769929520, 8223625903107027763, 3472328295963438455, 4192904167887482928, 7365366848950382640, 3487249781450040121, 7161630943624376368}}
        pid = <optimized out>
        tid = <optimized out>
#5  0x00007f9cbf3b742a in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x756e672d78756e69, sa_sigaction = 0x756e672d78756e69}, sa_mask = {__val = {3761119431852583983, 7149858270011683630, 3472328523928253027, 3689120322585769773, 8606977229197436468, 3472328296226648109, 3475143045726351408, 7378645556122361904, 3472387885476952166, 4123158048845278512, 8223625903103567204, 3472328295963438455, 4192904167887482928, 2314885531086893104, 2314885530818453536, 140735506385568}}, sa_flags = 123, sa_restorer = 0x7fff89dd8aa0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#6  0x00007f9cbf3f3c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f9cbf4e8d98 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
        ap = {{gp_offset = 40, fp_offset = 32668, overflow_arg_area = 0x7fff89dd8ab0, reg_save_area = 0x7fff89dd8a40}}
        fd = 2
        on_2 = <optimized out>
        list = <optimized out>
        nlist = <optimized out>
        cp = <optimized out>
        written = <optimized out>
#7  0x00007f9cbf3f9fc6 in malloc_printerr (action=3, str=0x7f9cbf4e8ef0 "double free or corruption (fasttop)", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5049
        buf = "00005642d7013a70"
        cp = <optimized out>
        ar_ptr = <optimized out>
        ptr = <optimized out>
        str = 0x7f9cbf4e8ef0 "double free or corruption (fasttop)"
        action = 3
#8  0x00007f9cbf3fa80e in _int_free (av=0x7f9cbf71cb00 <main_arena>, p=0x5642d7013a60, have_lock=0) at malloc.c:3905
        size = <optimized out>
        fb = <optimized out>
        nextchunk = <optimized out>
        nextsize = <optimized out>
        nextinuse = <optimized out>
        prevsize = <optimized out>
        bck = <optimized out>
        fwd = <optimized out>
        errstr = <optimized out>
        locked = <optimized out>
        __func__ = "_int_free"
#9  0x00005642d47aa1b4 in icinga::StreamLogger::BindStream(std::ostream*, bool) ()
No symbol table info available.
#10 0x00005642d475f45a in icinga::FileLogger::ReopenLogFile() ()
No symbol table info available.
#11 0x00005642d473c8f2 in icinga::Application::RunEventLoop() ()
No symbol table info available.
#12 0x00005642d4945b64 in icinga::IcingaApplication::Main() ()
No symbol table info available.
#13 0x00005642d473a8a1 in icinga::Application::Run() ()
No symbol table info available.
#14 0x00005642d4915c19 in icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const ()
No symbol table info available.
#15 0x00005642d4740446 in ?? ()
No symbol table info available.
#16 0x00005642d46e4889 in main ()
No symbol table info available.
NielsH picture NielsH on 3 Dec 2018

Update: happened again on the same machine as usual.
I installed gdb for more crash output to help out. Hope it'll help you guys :)

Reminder: Debian Stretch up-to-date, using systemd, Stable Icinga2 repository, package version r2.10.2-1

/var/log/icinga2/crash/report.1543728730.982800
````
Application version: r2.10.2-1

System information:
Platform: Debian GNU/Linux
Platform version: 9 (stretch)
Kernel: Linux
Kernel version: 4.9.0-7-amd64
Architecture: x86_64

Build information:
Compiler: GNU 6.3.0
Build host: 486e413fb159

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Stacktrace:

(0) libc.so.6: gsignal (+0xcf) [0x7f3b5c9f6fff]
(1) libc.so.6: abort (+0x16a) [0x7f3b5c9f842a]
(2) libc.so.6: <unknown function> (+0x70c00) [0x7f3b5ca34c00]
(3) libc.so.6: <unknown function> (+0x76fc6) [0x7f3b5ca3afc6]
(4) libc.so.6: <unknown function> (+0x7780e) [0x7f3b5ca3b80e]
(5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa4) [0x55c0505031b4]
(6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x55c0504b845a]
(7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x55c0504958f2]
(8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x55c05069eb64]
(9) icinga2: icinga::Application::Run() (+0xc1) [0x55c0504938a1]
(10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x55c05066ec19]
(11) icinga2: <unknown function> (+0x266446) [0x55c050499446]
(12) icinga2: main (+0x49) [0x55c05043d889]
(13) libc.so.6: __libc_start_main (+0xf1) [0x7f3b5c9e42e1]
(14) icinga2: _start (+0x2a) [0x55c05045ad6a]
  • This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
  • please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
  • information that might be useful in order to reproduce this problem.

[New LWP 24901]
[New LWP 24902]
[New LWP 24904]
[New LWP 24905]
[New LWP 24913]
[New LWP 24931]
[New LWP 24932]
[New LWP 24933]
[New LWP 24935]
[New LWP 24936]
[New LWP 24937]
[New LWP 24938]
[New LWP 24939]
[New LWP 24940]
[New LWP 24941]
[New LWP 24942]
[New LWP 24943]
[New LWP 24944]
[New LWP 24945]
[New LWP 24946]
[New LWP 24947]
[New LWP 24948]
[New LWP 24949]
[New LWP 24950]
[New LWP 24951]
[New LWP 25471]
[New LWP 25472]
[New LWP 25473]
[New LWP 25474]
[New LWP 21531]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f3b5e37bb3a in __waitpid (pid=21230, stat_loc=0x7ffefe45533c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29 ../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.

Thread 31 (Thread 0x7f3ae3fbe700 (LWP 21531)):

0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

No locals.

1 0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115

    id = 1580707676
    __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
    type = 1
    id = <optimized out>

2 0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()

No symbol table info available.

3 0x000055c0504785a9 in icinga::ConfigObject::SetAuthority(bool) ()

No symbol table info available.

4 0x000055c0505ef4dc in icinga::ApiListener::UpdateObjectAuthority() ()

No symbol table info available.

5 0x000055c05051b9c1 in ?? ()

No symbol table info available.

6 0x000055c0504dae74 in icinga::Timer::Call() ()

No symbol table info available.

7 0x000055c0504ffcf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()

No symbol table info available.

8 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

9 0x00007f3b5e372494 in start_thread (arg=0x7f3ae3fbe700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3ae3fbe700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139890909767424, 1733757316368709993, 0, 139892981867343, 0, 139892983652416, -1839072971467769495, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

10 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 30 (Thread 0x7f3ae3fff700 (LWP 25474)):

0 0x00007f3b5caa367d in poll () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504cb6fe in icinga::Process::IOThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3ae3fff700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3ae3fff700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139890910033664, 1733757316368709993, 0, 139892866313583, 0, 139892983652416, -1839072937644902039, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 29 (Thread 0x7f3b58079700 (LWP 25473)):

0 0x00007f3b5caa367d in poll () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504cb6fe in icinga::Process::IOThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b58079700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58079700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892856690432, 1733757316368709993, 0, 139892866313583, 0, 139892983652416, -1839373994451256983, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 28 (Thread 0x7f3b580ba700 (LWP 25472)):

0 0x00007f3b5caa367d in poll () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504cb6fe in icinga::Process::IOThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b580ba700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b580ba700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892856956672, 1733757316368709993, 0, 139892866313583, 0, 139892983652416, -1839374098067342999, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 27 (Thread 0x7f3b580fb700 (LWP 25471)):

0 0x00007f3b5caa367d in poll () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504cb6fe in icinga::Process::IOThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b580fb700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b580fb700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892857222912, 1733757316368709993, 0, 139892866313583, 0, 139892983652416, -1839374064244475543, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 26 (Thread 0x7f3b5813c700 (LWP 24951)):

0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

No locals.

1 0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115

    id = 1580707676
    __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
    type = 1
    id = <optimized out>

2 0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()

No symbol table info available.

3 0x000055c0504b58d8 in icinga::Log::~Log() ()

No symbol table info available.

4 0x000055c050554718 in icinga::JsonRpcConnection::MessageHandler(icinga::String const&) ()

No symbol table info available.

5 0x000055c0505554cb in icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) ()

No symbol table info available.

6 0x000055c05051efce in icinga::WorkQueue::RunTaskFunction(std::function

No symbol table info available.

7 0x000055c050526565 in icinga::WorkQueue::WorkerThreadProc() ()

No symbol table info available.

8 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

9 0x00007f3b5e372494 in start_thread (arg=0x7f3b5813c700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b5813c700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892857489152, 1733757316368709993, 0, 139892860609807, 0, 139892983652416, -1839373892982654615, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

10 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 25 (Thread 0x7f3b5817d700 (LWP 24950)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c05052679b in icinga::WorkQueue::WorkerThreadProc() ()

No symbol table info available.

3 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b5817d700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b5817d700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892857755392, 1733757316368709993, 0, 139892865249151, 0, 139892983652416, -1839373859159787159, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 24 (Thread 0x7f3b581be700 (LWP 24949)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c05052679b in icinga::WorkQueue::WorkerThreadProc() ()

No symbol table info available.

3 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b581be700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b581be700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892858021632, 1733757316368709993, 0, 139892865249151, 0, 139892983652416, -1839373962775873175, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 23 (Thread 0x7f3b581ff700 (LWP 24948)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c05052679b in icinga::WorkQueue::WorkerThreadProc() ()

No symbol table info available.

3 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b581ff700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b581ff700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892858287872, 1733757316368709993, 0, 139892865249151, 0, 139892983652416, -1839373928953005719, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 22 (Thread 0x7f3b58240700 (LWP 24947)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c05052679b in icinga::WorkQueue::WorkerThreadProc() ()

No symbol table info available.

3 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b58240700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58240700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892858554112, 1733757316368709993, 0, 139892865249151, 0, 139892983652416, -1839373757691184791, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 21 (Thread 0x7f3b58281700 (LWP 24946)):

0 0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b58281700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58281700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892858820352, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373792587794071, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 20 (Thread 0x7f3b582c2700 (LWP 24945)):

0 0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b582c2700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b582c2700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892859086592, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373758764926615, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 19 (Thread 0x7f3b58303700 (LWP 24944)):

0 0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b58303700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58303700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892859352832, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373587503105687, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 18 (Thread 0x7f3b58344700 (LWP 24943)):

0 0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b58344700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58344700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892859619072, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373553680238231, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 17 (Thread 0x7f3b58385700 (LWP 24942)):

0 0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b58385700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58385700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892859885312, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373657296324247, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 16 (Thread 0x7f3b583c6700 (LWP 24941)):

0 0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b583c6700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b583c6700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892860151552, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373623473456791, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 15 (Thread 0x7f3b58407700 (LWP 24940)):

0 0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b58407700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58407700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892860417792, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373452211635863, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 14 (Thread 0x7f3b58448700 (LWP 24939)):

0 0x00007f3b5caad0c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504fb197 in icinga::SocketEventEngineEpoll::ThreadProc(int) ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b58448700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58448700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892860684032, 1733757316368709993, 0, 139892865248207, 0, 139892983652416, -1839373418388768407, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 13 (Thread 0x7f3b588a6700 (LWP 24938)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()

No symbol table info available.

3 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b588a6700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b588a6700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892865263360, 1733757316368709993, 0, 139892873056479, 139892865263360, 94284516776112, -1839375186841552535, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 12 (Thread 0x7f3b588e7700 (LWP 24937)):

0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

No locals.

1 0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115

    id = 1580707676
    __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
    type = 1
    id = <optimized out>

2 0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()

No symbol table info available.

3 0x000055c0504b58d8 in icinga::Log::~Log() ()

No symbol table info available.

4 0x000055c0504fd126 in icinga::ThreadPool::ManagerThreadProc() ()

No symbol table info available.

5 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

6 0x00007f3b5e372494 in start_thread (arg=0x7f3b588e7700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b588e7700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892865529600, 1733757316368709993, 0, 139892873057007, 139892865529600, 94284516737424, -1839375153018685079, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

7 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 11 (Thread 0x7f3b58928700 (LWP 24936)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()

No symbol table info available.

3 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b58928700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58928700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892865795840, 1733757316368709993, 0, 139892873056367, 139892865795840, 94284516775776, -1839374981756864151, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 10 (Thread 0x7f3b58969700 (LWP 24935)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()

No symbol table info available.

3 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b58969700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b58969700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892866062080, 1733757316368709993, 0, 139892873056367, 139892866062080, 94284516774960, -1839374947933996695, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 9 (Thread 0x7f3b5f8df700 (LWP 24933)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()

No symbol table info available.

3 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b5f8df700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b5f8df700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892982937344, 1733757316368709993, 0, 139892873056367, 139892982937344, 94284511321424, -1839381745793484439, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 8 (Thread 0x7f3b5f89e700 (LWP 24932)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c0507a7a4b in icinga::CheckerComponent::CheckThreadProc() ()

No symbol table info available.

3 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b5f89e700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b5f89e700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892982671104, 1733757316368709993, 0, 140733164381599, 139892982671104, 94284511393584, -1839381779616351895, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 7 (Thread 0x7f3b5f85d700 (LWP 24931)):

0 0x00007f3b5e37b3ed in accept () at ../sysdeps/unix/syscall-template.S:84

No locals.

1 0x000055c0504f6b77 in icinga::Socket::Accept() ()

No symbol table info available.

2 0x000055c0505f50c7 in icinga::ApiListener::ListenerThreadProc(boost::intrusive_ptr const&) ()

No symbol table info available.

3 0x000055c050656b96 in ?? ()

No symbol table info available.

4 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

5 0x00007f3b5e372494 in start_thread (arg=0x7f3b5f85d700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b5f85d700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892982404864, 1733757316368709993, 0, 140733164379695, 139892982404864, 94284511394496, -1839381676000265879, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

6 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 6 (Thread 0x7f3b5f81c700 (LWP 24913)):

0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225

No locals.

1 0x000055c0504fe453 in icinga::Timer::TimerThreadProc() ()

No symbol table info available.

2 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

3 0x00007f3b5e372494 in start_thread (arg=0x7f3b5f81c700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b5f81c700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892982138624, 1733757316368709993, 0, 140733164382143, 139892982138624, 94284511404816, -1839381709823133335, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

4 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 5 (Thread 0x7f3b5f7db700 (LWP 24905)):

0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

No locals.

1 0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115

    id = 1580707676
    __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
    type = 1
    id = <optimized out>

2 0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()

No symbol table info available.

3 0x000055c0504b58d8 in icinga::Log::~Log() ()

No symbol table info available.

4 0x000055c0504fd126 in icinga::ThreadPool::ManagerThreadProc() ()

No symbol table info available.

5 0x00007f3b5d337e6f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

6 0x00007f3b5e372494 in start_thread (arg=0x7f3b5f7db700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b5f7db700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892981872384, 1733757316368709993, 0, 140733164402271, 139892981872384, 94284511405728, -1839379682061698711, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

7 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 4 (Thread 0x7f3b590da700 (LWP 24904)):

0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185

No locals.

1 0x000055c050482bbb in boost::condition_variable::wait(boost::unique_lock&) ()

No symbol table info available.

2 0x000055c0504ffbb7 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()

No symbol table info available.

3 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

4 0x00007f3b5e372494 in start_thread (arg=0x7f3b590da700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b590da700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892873864960, 1733757316368709993, 0, 140733164401631, 139892873864960, 94284511406640, -1839376245550990999, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

5 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 3 (Thread 0x7f3b59058700 (LWP 24902)):

0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

No locals.

1 0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115

    id = 1580707676
    __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
    type = 1
    id = <optimized out>

2 0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()

No symbol table info available.

3 0x000055c0504b58d8 in icinga::Log::~Log() ()

No symbol table info available.

4 0x000055c0507a671e in icinga::CheckerComponent::ResultTimerHandler() ()

No symbol table info available.

5 0x000055c05051b9c1 in ?? ()

No symbol table info available.

6 0x000055c0504dae74 in icinga::Timer::Call() ()

No symbol table info available.

7 0x000055c0504ffcf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()

No symbol table info available.

8 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

9 0x00007f3b5e372494 in start_thread (arg=0x7f3b59058700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b59058700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892873332480, 1733757316368709993, 0, 140733164401631, 139892873332480, 94284511356432, -1839376175757772439, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

10 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 2 (Thread 0x7f3b59017700 (LWP 24901)):

0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

No locals.

1 0x00007f3b5e374c06 in __GI___pthread_mutex_lock (mutex=0x55c0531fb730) at ../nptl/pthread_mutex_lock.c:115

    id = 1580707676
    __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
    type = 1
    id = <optimized out>

2 0x000055c0504a8b45 in icinga::ObjectLock::LockMutex(icinga::Object const*) ()

No symbol table info available.

3 0x000055c0504b58d8 in icinga::Log::~Log() ()

No symbol table info available.

4 0x000055c0505e278e in icinga::ApiListener::ApiReconnectTimerHandler() ()

No symbol table info available.

5 0x000055c05051b9c1 in ?? ()

No symbol table info available.

6 0x000055c0504dae74 in icinga::Timer::Call() ()

No symbol table info available.

7 0x000055c0504ffcf8 in icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) ()

No symbol table info available.

8 0x00007f3b5f350116 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0

No symbol table info available.

9 0x00007f3b5e372494 in start_thread (arg=0x7f3b59017700) at pthread_create.c:333

    __res = <optimized out>
    pd = 0x7f3b59017700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139892873066240, 1733757316368709993, 0, 140733164401631, 139892873066240, 94284511398640, -1839376209580639895, -1839377977283036823}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
    __PRETTY_FUNCTION__ = "start_thread"

10 0x00007f3b5caacacf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

No locals.

Thread 1 (Thread 0x7f3b5f8eb480 (LWP 24865)):

0 0x00007f3b5e37bb3a in __waitpid (pid=21230, stat_loc=0x7ffefe45533c, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29

    resultvar = 18446744073709551104
    sc_cancel_oldtype = 0

1 0x000055c05046f7de in icinga::Application::AttachDebugger(icinga::String const&, bool) ()

No symbol table info available.

2 0x000055c050493ef7 in icinga::Application::SigAbrtHandler(int) ()

No symbol table info available.

3

No locals.

4 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51

    set = {__val = {0, 3472339291342909488, 7306639832034324512, 3472328511026652518, 3775536448315733805, 8606977229197435701, 3472328296226648109, 3475143045726351408, 2314885530819502128, 2314885530818453536, 2314885530818453536, 8391150724257423392, 7378645557149655905, 3472335116723119717, 7306638935931170096, 8223625903104091189}}
    pid = <optimized out>
    tid = <optimized out>

5 0x00007f3b5c9f842a in __GI_abort () at abort.c:89

    save_stage = 2
    act = {__sigaction_handler = {sa_handler = 0x623366370a302e37, sa_sigaction = 0x623366370a302e37}, sa_mask = {__val = {3472328335000430133, 4136052003671783213, 3274715270390768439, 3762810472192045101, 3475151841819373616, 4121131437673029681, 2314885530818453558, 2314885530818453536, 7596498486491619360, 3762299225194901346, 7434731005617466413, 8751446705837143406, 8030813569398699123, 734137553844580398, 3979324125225379383, 140733164381200}}, sa_flags = 123, sa_restorer = 0x7ffefe456410}
    sigs = {__val = {32, 0 <repeats 15 times>}}

6 0x00007f3b5ca34c00 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f3b5cb29d98 "* Error in `%s': %s: 0x%s *\n") at ../sysdeps/posix/libc_fatal.c:175

    ap = {{gp_offset = 40, fp_offset = 32571, overflow_arg_area = 0x7ffefe456420, reg_save_area = 0x7ffefe4563b0}}
    fd = 2
    on_2 = <optimized out>
    list = <optimized out>
    nlist = <optimized out>
    cp = <optimized out>
    written = <optimized out>

7 0x00007f3b5ca3afc6 in malloc_printerr (action=3, str=0x7f3b5cb29ef0 "double free or corruption (fasttop)", ptr=, ar_ptr=) at malloc.c:5049

    buf = "000055c052fe14f0"
    cp = <optimized out>
    ar_ptr = <optimized out>
    ptr = <optimized out>
    str = 0x7f3b5cb29ef0 "double free or corruption (fasttop)"
    action = 3

8 0x00007f3b5ca3b80e in _int_free (av=0x7f3b5cd5db00 , p=0x55c052fe14e0, have_lock=0) at malloc.c:3905

    size = <optimized out>
    fb = <optimized out>
    nextchunk = <optimized out>
    nextsize = <optimized out>
    nextinuse = <optimized out>
    prevsize = <optimized out>
    bck = <optimized out>
    fwd = <optimized out>
    errstr = <optimized out>
    locked = <optimized out>
    __func__ = "_int_free"

9 0x000055c0505031b4 in icinga::StreamLogger::BindStream(std::ostream*, bool) ()

No symbol table info available.

10 0x000055c0504b845a in icinga::FileLogger::ReopenLogFile() ()

No symbol table info available.

11 0x000055c0504958f2 in icinga::Application::RunEventLoop() ()

No symbol table info available.

12 0x000055c05069eb64 in icinga::IcingaApplication::Main() ()

No symbol table info available.

13 0x000055c0504938a1 in icinga::Application::Run() ()

No symbol table info available.

14 0x000055c05066ec19 in icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector, std::allocator >, std::allocator, std::allocator > > > const&) const ()

No symbol table info available.

15 0x000055c050499446 in ?? ()

No symbol table info available.

16 0x000055c05043d889 in main ()

No symbol table info available.
````

/var/log/icinga2/error.log.1
*** Error in `/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2': double free or corruption (fasttop): 0x000055c052fe14f0 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x70bfb)[0x7f3b5ca34bfb] /lib/x86_64-linux-gnu/libc.so.6(+0x76fc6)[0x7f3b5ca3afc6] /lib/x86_64-linux-gnu/libc.so.6(+0x7780e)[0x7f3b5ca3b80e] /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga12StreamLogger10BindStreamEPSob+0xa4)[0x55c0505031b4] /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga10FileLogger13ReopenLogFileEv+0xaa)[0x55c0504b845a] /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga11Application12RunEventLoopEv+0x7c2)[0x55c0504958f2] /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga17IcingaApplication4MainEv+0x1f4)[0x55c05069eb64] /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZN6icinga11Application3RunEv+0xc1)[0x55c0504938a1] /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_ZNK6icinga13DaemonCommand3RunERKN5boost15program_options13variables_mapERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaISC_EE+0xbc9)[0x55c05066ec19] /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(+0x266446)[0x55c050499446] /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(main+0x49)[0x55c05043d889] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f3b5c9e42e1] /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2(_start+0x2a)[0x55c05045ad6a] ======= Memory map: ======== 55c050233000-55c050b59000 r-xp 00000000 08:01 10355102 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 55c050d59000-55c050d8d000 r--p 00926000 08:01 10355102 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 55c050d8d000-55c050d8f000 rw-p 0095a000 08:01 10355102 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 55c050d8f000-55c050d96000 rw-p 00000000 00:00 0 55c052c7b000-55c053242000 rw-p 00000000 00:00 0 [heap] 7f3ad4000000-7f3ad4021000 rw-p 00000000 00:00 0 7f3ad4021000-7f3ad8000000 ---p 00000000 00:00 0 7f3ad8000000-7f3ad8021000 rw-p 00000000 00:00 0 7f3ad8021000-7f3adc000000 ---p 00000000 00:00 0 7f3adc000000-7f3adc021000 rw-p 00000000 00:00 0 7f3adc021000-7f3ae0000000 ---p 00000000 00:00 0 7f3ae3f7e000-7f3ae3f7f000 ---p 00000000 00:00 0 7f3ae3f7f000-7f3ae3fbf000 rw-p 00000000 00:00 0 7f3ae3fbf000-7f3ae3fc0000 ---p 00000000 00:00 0 7f3ae3fc0000-7f3ae4000000 rw-p 00000000 00:00 0 7f3ae4000000-7f3ae4021000 rw-p 00000000 00:00 0 7f3ae4021000-7f3ae8000000 ---p 00000000 00:00 0 7f3ae8000000-7f3ae8021000 rw-p 00000000 00:00 0 7f3ae8021000-7f3aec000000 ---p 00000000 00:00 0 7f3aec000000-7f3aec021000 rw-p 00000000 00:00 0 7f3aec021000-7f3af0000000 ---p 00000000 00:00 0 7f3af0000000-7f3af0021000 rw-p 00000000 00:00 0 7f3af0021000-7f3af4000000 ---p 00000000 00:00 0 7f3af4000000-7f3af4021000 rw-p 00000000 00:00 0 7f3af4021000-7f3af8000000 ---p 00000000 00:00 0 7f3af8000000-7f3af8021000 rw-p 00000000 00:00 0 7f3af8021000-7f3afc000000 ---p 00000000 00:00 0 7f3afc000000-7f3afc021000 rw-p 00000000 00:00 0 7f3afc021000-7f3b00000000 ---p 00000000 00:00 0 7f3b00000000-7f3b00021000 rw-p 00000000 00:00 0 7f3b00021000-7f3b04000000 ---p 00000000 00:00 0 7f3b04000000-7f3b04021000 rw-p 00000000 00:00 0 7f3b04021000-7f3b08000000 ---p 00000000 00:00 0 7f3b08000000-7f3b08021000 rw-p 00000000 00:00 0 7f3b08021000-7f3b0c000000 ---p 00000000 00:00 0 7f3b0c000000-7f3b0c021000 rw-p 00000000 00:00 0 7f3b0c021000-7f3b10000000 ---p 00000000 00:00 0 7f3b10000000-7f3b10021000 rw-p 00000000 00:00 0 7f3b10021000-7f3b14000000 ---p 00000000 00:00 0 7f3b14000000-7f3b14021000 rw-p 00000000 00:00 0 7f3b14021000-7f3b18000000 ---p 00000000 00:00 0 7f3b18000000-7f3b18021000 rw-p 00000000 00:00 0 7f3b18021000-7f3b1c000000 ---p 00000000 00:00 0 7f3b1c000000-7f3b1c042000 rw-p 00000000 00:00 0 7f3b1c042000-7f3b20000000 ---p 00000000 00:00 0 7f3b20000000-7f3b20021000 rw-p 00000000 00:00 0 7f3b20021000-7f3b24000000 ---p 00000000 00:00 0 7f3b24000000-7f3b2406c000 rw-p 00000000 00:00 0 7f3b2406c000-7f3b28000000 ---p 00000000 00:00 0 7f3b28000000-7f3b2804c000 rw-p 00000000 00:00 0 7f3b2804c000-7f3b2c000000 ---p 00000000 00:00 0 7f3b2c000000-7f3b2c05e000 rw-p 00000000 00:00 0 7f3b2c05e000-7f3b30000000 ---p 00000000 00:00 0 7f3b30000000-7f3b30021000 rw-p 00000000 00:00 0 7f3b30021000-7f3b34000000 ---p 00000000 00:00 0 7f3b34000000-7f3b34056000 rw-p 00000000 00:00 0 7f3b34056000-7f3b38000000 ---p 00000000 00:00 0 7f3b38000000-7f3b38021000 rw-p 00000000 00:00 0 7f3b38021000-7f3b3c000000 ---p 00000000 00:00 0 7f3b3c000000-7f3b3c02b000 rw-p 00000000 00:00 0 7f3b3c02b000-7f3b40000000 ---p 00000000 00:00 0 7f3b40000000-7f3b40021000 rw-p 00000000 00:00 0 7f3b40021000-7f3b44000000 ---p 00000000 00:00 0 7f3b44000000-7f3b44021000 rw-p 00000000 00:00 0 7f3b44021000-7f3b48000000 ---p 00000000 00:00 0 7f3b48000000-7f3b48021000 rw-p 00000000 00:00 0 7f3b48021000-7f3b4c000000 ---p 00000000 00:00 0 7f3b4c000000-7f3b4c076000 rw-p 00000000 00:00 0 7f3b4c076000-7f3b50000000 ---p 00000000 00:00 0 7f3b50000000-7f3b50050000 rw-p 00000000 00:00 0 7f3b50050000-7f3b54000000 ---p 00000000 00:00 0 7f3b54000000-7f3b54054000 rw-p 00000000 00:00 0 7f3b54054000-7f3b58000000 ---p 00000000 00:00 0 7f3b58039000-7f3b5803a000 ---p 00000000 00:00 0 7f3b5803a000-7f3b5807a000 rw-p 00000000 00:00 0 7f3b5807a000-7f3b5807b000 ---p 00000000 00:00 0 7f3b5807b000-7f3b580bb000 rw-p 00000000 00:00 0 7f3b580bb000-7f3b580bc000 ---p 00000000 00:00 0 7f3b580bc000-7f3b580fc000 rw-p 00000000 00:00 0 7f3b580fc000-7f3b580fd000 ---p 00000000 00:00 0 7f3b580fd000-7f3b5813d000 rw-p 00000000 00:00 0 7f3b5813d000-7f3b5813e000 ---p 00000000 00:00 0 7f3b5813e000-7f3b5817e000 rw-p 00000000 00:00 0 7f3b5817e000-7f3b5817f000 ---p 00000000 00:00 0 7f3b5817f000-7f3b581bf000 rw-p 00000000 00:00 0 7f3b581bf000-7f3b581c0000 ---p 00000000 00:00 0 7f3b581c0000-7f3b58200000 rw-p 00000000 00:00 0 7f3b58200000-7f3b58201000 ---p 00000000 00:00 0 7f3b58201000-7f3b58241000 rw-p 00000000 00:00 0 7f3b58241000-7f3b58242000 ---p 00000000 00:00 0 7f3b58242000-7f3b58282000 rw-p 00000000 00:00 0 7f3b58282000-7f3b58283000 ---p 00000000 00:00 0 7f3b58283000-7f3b582c3000 rw-p 00000000 00:00 0 7f3b582c3000-7f3b582c4000 ---p 00000000 00:00 0 7f3b582c4000-7f3b58304000 rw-p 00000000 00:00 0 7f3b58304000-7f3b58305000 ---p 00000000 00:00 0 7f3b58305000-7f3b58345000 rw-p 00000000 00:00 0 7f3b58345000-7f3b58346000 ---p 00000000 00:00 0 7f3b58346000-7f3b58386000 rw-p 00000000 00:00 0 7f3b58386000-7f3b58387000 ---p 00000000 00:00 0 7f3b58387000-7f3b583c7000 rw-p 00000000 00:00 0 7f3b583c7000-7f3b583c8000 ---p 00000000 00:00 0 7f3b583c8000-7f3b58408000 rw-p 00000000 00:00 0 7f3b58408000-7f3b58409000 ---p 00000000 00:00 0 7f3b58409000-7f3b58449000 rw-p 00000000 00:00 0 7f3b58449000-7f3b5845d000 r-xp 00000000 08:01 2752536 /lib/x86_64-linux-gnu/libresolv-2.24.so 7f3b5845d000-7f3b5865c000 ---p 00014000 08:01 2752536 /lib/x86_64-linux-gnu/libresolv-2.24.so 7f3b5865c000-7f3b5865d000 r--p 00013000 08:01 2752536 /lib/x86_64-linux-gnu/libresolv-2.24.so 7f3b5865d000-7f3b5865e000 rw-p 00014000 08:01 2752536 /lib/x86_64-linux-gnu/libresolv-2.24.so 7f3b5865e000-7f3b58660000 rw-p 00000000 00:00 0 7f3b58660000-7f3b58665000 r-xp 00000000 08:01 2752529 /lib/x86_64-linux-gnu/libnss_dns-2.24.so 7f3b58665000-7f3b58864000 ---p 00005000 08:01 2752529 /lib/x86_64-linux-gnu/libnss_dns-2.24.so 7f3b58864000-7f3b58865000 r--p 00004000 08:01 2752529 /lib/x86_64-linux-gnu/libnss_dns-2.24.so 7f3b58865000-7f3b58866000 rw-p 00005000 08:01 2752529 /lib/x86_64-linux-gnu/libnss_dns-2.24.so 7f3b58866000-7f3b58867000 ---p 00000000 00:00 0 7f3b58867000-7f3b588a7000 rw-p 00000000 00:00 0 7f3b588a7000-7f3b588a8000 ---p 00000000 00:00 0 7f3b588a8000-7f3b588e8000 rw-p 00000000 00:00 0 7f3b588e8000-7f3b588e9000 ---p 00000000 00:00 0 7f3b588e9000-7f3b58929000 rw-p 00000000 00:00 0 7f3b58929000-7f3b5892a000 ---p 00000000 00:00 0 7f3b5892a000-7f3b5896a000 rw-p 00000000 00:00 0 7f3b5896a000-7f3b5896b000 ---p 00000000 00:00 0 7f3b5896b000-7f3b589ab000 rw-p 00000000 00:00 0 7f3b589ab000-7f3b589b6000 r-xp 00000000 08:01 2752532 /lib/x86_64-linux-gnu/libnss_nis-2.24.so 7f3b589b6000-7f3b58bb5000 ---p 0000b000 08:01 2752532 /lib/x86_64-linux-gnu/libnss_nis-2.24.so 7f3b58bb5000-7f3b58bb6000 r--p 0000a000 08:01 2752532 /lib/x86_64-linux-gnu/libnss_nis-2.24.so 7f3b58bb6000-7f3b58bb7000 rw-p 0000b000 08:01 2752532 /lib/x86_64-linux-gnu/libnss_nis-2.24.so 7f3b58bb7000-7f3b58bcb000 r-xp 00000000 08:01 2752527 /lib/x86_64-linux-gnu/libnsl-2.24.so 7f3b58bcb000-7f3b58dcb000 ---p 00014000 08:01 2752527 /lib/x86_64-linux-gnu/libnsl-2.24.so 7f3b58dcb000-7f3b58dcc000 r--p 00014000 08:01 2752527 /lib/x86_64-linux-gnu/libnsl-2.24.so 7f3b58dcc000-7f3b58dcd000 rw-p 00015000 08:01 2752527 /lib/x86_64-linux-gnu/libnsl-2.24.so 7f3b58dcd000-7f3b58dcf000 rw-p 00000000 00:00 0 7f3b58dcf000-7f3b58dd6000 r-xp 00000000 08:01 2752528 /lib/x86_64-linux-gnu/libnss_compat-2.24.so 7f3b58dd6000-7f3b58fd5000 ---p 00007000 08:01 2752528 /lib/x86_64-linux-gnu/libnss_compat-2.24.so 7f3b58fd5000-7f3b58fd6000 r--p 00006000 08:01 2752528 /lib/x86_64-linux-gnu/libnss_compat-2.24.so 7f3b58fd6000-7f3b58fd7000 rw-p 00007000 08:01 2752528 /lib/x86_64-linux-gnu/libnss_compat-2.24.so 7f3b58fd7000-7f3b58fd8000 ---p 00000000 00:00 0 7f3b58fd8000-7f3b59018000 rw-p 00000000 00:00 0 7f3b59018000-7f3b59019000 ---p 00000000 00:00 0 7f3b59019000-7f3b59059000 rw-p 00000000 00:00 0 7f3b59059000-7f3b5905a000 ---p 00000000 00:00 0 7f3b5905a000-7f3b5909a000 rw-p 00000000 00:00 0 7f3b5909a000-7f3b5909b000 ---p 00000000 00:00 0 7f3b5909b000-7f3b590db000 rw-p 00000000 00:00 0 7f3b590db000-7f3b590e5000 r-xp 00000000 08:01 2752530 /lib/x86_64-linux-gnu/libnss_files-2.24.so 7f3b590e5000-7f3b592e5000 ---p 0000a000 08:01 2752530 /lib/x86_64-linux-gnu/libnss_files-2.24.so 7f3b592e5000-7f3b592e6000 r--p 0000a000 08:01 2752530 /lib/x86_64-linux-gnu/libnss_files-2.24.so 7f3b592e6000-7f3b592e7000 rw-p 0000b000 08:01 2752530 /lib/x86_64-linux-gnu/libnss_files-2.24.so 7f3b592e7000-7f3b592ed000 rw-p 00000000 00:00 0 7f3b592ed000-7f3b59300000 r-xp 00000000 08:01 2752562 /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0 7f3b59300000-7f3b594ff000 ---p 00013000 08:01 2752562 /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0 7f3b594ff000-7f3b59500000 r--p 00012000 08:01 2752562 /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0 7f3b59500000-7f3b59501000 rw-p 00013000 08:01 2752562 /lib/x86_64-linux-gnu/libgpg-error.so.0.21.0 7f3b59501000-7f3b59573000 r-xp 00000000 08:01 2752551 /lib/x86_64-linux-gnu/libpcre.so.3.13.3 7f3b59573000-7f3b59772000 ---p 00072000 08:01 2752551 /lib/x86_64-linux-gnu/libpcre.so.3.13.3 7f3b59772000-7f3b59773000 r--p 00071000 08:01 2752551 /lib/x86_64-linux-gnu/libpcre.so.3.13.3 7f3b59773000-7f3b59774000 rw-p 00072000 08:01 2752551 /lib/x86_64-linux-gnu/libpcre.so.3.13.3 7f3b59774000-7f3b59787000 r-xp 00000000 08:01 2753036 /lib/x86_64-linux-gnu/libbsd.so.0.8.3 7f3b59787000-7f3b59987000 ---p 00013000 08:01 2753036 /lib/x86_64-linux-gnu/libbsd.so.0.8.3 7f3b59987000-7f3b59988000 r--p 00013000 08:01 2753036 /lib/x86_64-linux-gnu/libbsd.so.0.8.3 7f3b59988000-7f3b59989000 rw-p 00014000 08:01 2753036 /lib/x86_64-linux-gnu/libbsd.so.0.8.3 7f3b59989000-7f3b5998a000 rw-p 00000000 00:00 0 7f3b5998a000-7f3b599ac000 r-xp 00000000 08:01 2752566 /lib/x86_64-linux-gnu/libncurses.so.5.9 7f3b599ac000-7f3b59bab000 ---p 00022000 08:01 2752566 /lib/x86_64-linux-gnu/libncurses.so.5.9 7f3b59bab000-7f3b59bac000 r--p 00021000 08:01 2752566 /lib/x86_64-linux-gnu/libncurses.so.5.9 7f3b59bac000-7f3b59bad000 rw-p 00022000 08:01 2752566 /lib/x86_64-linux-gnu/libncurses.so.5.9 7f3b59bad000-7f3b59cb4000 r-xp 00000000 08:01 2752564 /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6 7f3b59cb4000-7f3b59eb4000 ---p 00107000 08:01 2752564 /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6 7f3b59eb4000-7f3b59eb6000 r--p 00107000 08:01 2752564 /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6 7f3b59eb6000-7f3b59ebd000 rw-p 00109000 08:01 2752564 /lib/x86_64-linux-gnu/libgcrypt.so.20.1.6 7f3b59ebd000-7f3b59ece000 r-xp 00000000 08:01 10224540 /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1 7f3b59ece000-7f3b5a0cd000 ---p 00011000 08:01 10224540 /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1 7f3b5a0cd000-7f3b5a0ce000 r--p 00010000 08:01 10224540 /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1 7f3b5a0ce000-7f3b5a0cf000 rw-p 00011000 08:01 10224540 /usr/lib/x86_64-linux-gnu/liblz4.so.1.7.1 7f3b5a0cf000-7f3b5a0f4000 r-xp 00000000 08:01 2752545 /lib/x86_64-linux-gnu/liblzma.so.5.2.2 7f3b5a0f4000-7f3b5a2f3000 ---p 00025000 08:01 2752545 /lib/x86_64-linux-gnu/liblzma.so.5.2.2 7f3b5a2f3000-7f3b5a2f4000 r--p 00024000 08:01 2752545 /lib/x86_64-linux-gnu/liblzma.so.5.2.2 7f3b5a2f4000-7f3b5a2f5000 rw-p 00025000 08:01 2752545 /lib/x86_64-linux-gnu/liblzma.so.5.2.2 7f3b5a2f5000-7f3b5a31a000 r-xp 00000000 08:01 2752601 /lib/x86_64-linux-gnu/libselinux.so.1 7f3b5a31a000-7f3b5a519000 ---p 00025000 08:01 2752601 /lib/x86_64-linux-gnu/libselinux.so.1 7f3b5a519000-7f3b5a51a000 r--p 00024000 08:01 2752601 /lib/x86_64-linux-gnu/libselinux.so.1 7f3b5a51a000-7f3b5a51b000 rw-p 00025000 08:01 2752601 /lib/x86_64-linux-gnu/libselinux.so.1 7f3b5a51b000-7f3b5a51d000 rw-p 00000000 00:00 0 7f3b5a51d000-7f3b5a6b1000 r-xp 00000000 08:01 10231348 /usr/lib/x86_64-linux-gnu/libicuuc.so.57.1 7f3b5a6b1000-7f3b5a8b0000 ---p 00194000 08:01 10231348 /usr/lib/x86_64-linux-gnu/libicuuc.so.57.1 7f3b5a8b0000-7f3b5a8c2000 r--p 00193000 08:01 10231348 /usr/lib/x86_64-linux-gnu/libicuuc.so.57.1 7f3b5a8c2000-7f3b5a8c3000 rw-p 001a5000 08:01 10231348 /usr/lib/x86_64-linux-gnu/libicuuc.so.57.1 7f3b5a8c3000-7f3b5a8c5000 rw-p 00000000 00:00 0 7f3b5a8c5000-7f3b5ab30000 r-xp 00000000 08:01 10231342 /usr/lib/x86_64-linux-gnu/libicui18n.so.57.1 7f3b5ab30000-7f3b5ad2f000 ---p 0026b000 08:01 10231342 /usr/lib/x86_64-linux-gnu/libicui18n.so.57.1 7f3b5ad2f000-7f3b5ad3c000 r--p 0026a000 08:01 10231342 /usr/lib/x86_64-linux-gnu/libicui18n.so.57.1 7f3b5ad3c000-7f3b5ad3e000 rw-p 00277000 08:01 10231342 /usr/lib/x86_64-linux-gnu/libicui18n.so.57.1 7f3b5ad3e000-7f3b5ad3f000 rw-p 00000000 00:00 0 7f3b5ad3f000-7f3b5c5bb000 r-xp 00000000 08:01 10231341 /usr/lib/x86_64-linux-gnu/libicudata.so.57.1 7f3b5c5bb000-7f3b5c7ba000 ---p 0187c000 08:01 10231341 /usr/lib/x86_64-linux-gnu/libicudata.so.57.1 7f3b5c7ba000-7f3b5c7bb000 r--p 0187b000 08:01 10231341 /usr/lib/x86_64-linux-gnu/libicudata.so.57.1 7f3b5c7bb000-7f3b5c7bc000 rw-p 0187c000 08:01 10231341 /usr/lib/x86_64-linux-gnu/libicudata.so.57.1 7f3b5c7bc000-7f3b5c7c3000 r-xp 00000000 08:01 2752537 /lib/x86_64-linux-gnu/librt-2.24.so 7f3b5c7c3000-7f3b5c9c2000 ---p 00007000 08:01 2752537 /lib/x86_64-linux-gnu/librt-2.24.so 7f3b5c9c2000-7f3b5c9c3000 r--p 00006000 08:01 2752537 /lib/x86_64-linux-gnu/librt-2.24.so 7f3b5c9c3000-7f3b5c9c4000 rw-p 00007000 08:01 2752537 /lib/x86_64-linux-gnu/librt-2.24.so 7f3b5c9c4000-7f3b5cb59000 r-xp 00000000 08:01 2752520 /lib/x86_64-linux-gnu/libc-2.24.so 7f3b5cb59000-7f3b5cd59000 ---p 00195000 08:01 2752520 /lib/x86_64-linux-gnu/libc-2.24.so 7f3b5cd59000-7f3b5cd5d000 r--p 00195000 08:01 2752520 /lib/x86_64-linux-gnu/libc-2.24.so 7f3b5cd5d000-7f3b5cd5f000 rw-p 00199000 08:01 2752520 /lib/x86_64-linux-gnu/libc-2.24.so 7f3b5cd5f000-7f3b5cd63000 rw-p 00000000 00:00 0 7f3b5cd63000-7f3b5cd79000 r-xp 00000000 08:01 2752516 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f3b5cd79000-7f3b5cf78000 ---p 00016000 08:01 2752516 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f3b5cf78000-7f3b5cf79000 r--p 00015000 08:01 2752516 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f3b5cf79000-7f3b5cf7a000 rw-p 00016000 08:01 2752516 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f3b5cf7a000-7f3b5d07d000 r-xp 00000000 08:01 2752524 /lib/x86_64-linux-gnu/libm-2.24.so 7f3b5d07d000-7f3b5d27c000 ---p 00103000 08:01 2752524 /lib/x86_64-linux-gnu/libm-2.24.so 7f3b5d27c000-7f3b5d27d000 r--p 00102000 08:01 2752524 /lib/x86_64-linux-gnu/libm-2.24.so 7f3b5d27d000-7f3b5d27e000 rw-p 00103000 08:01 2752524 /lib/x86_64-linux-gnu/libm-2.24.so 7f3b5d27e000-7f3b5d3f0000 r-xp 00000000 08:01 10229266 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 7f3b5d3f0000-7f3b5d5f0000 ---p 00172000 08:01 10229266 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 7f3b5d5f0000-7f3b5d5fa000 r--p 00172000 08:01 10229266 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 7f3b5d5fa000-7f3b5d5fc000 rw-p 0017c000 08:01 10229266 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 7f3b5d5fc000-7f3b5d600000 rw-p 00000000 00:00 0 7f3b5d600000-7f3b5d626000 r-xp 00000000 08:01 2752604 /lib/x86_64-linux-gnu/libtinfo.so.5.9 7f3b5d626000-7f3b5d825000 ---p 00026000 08:01 2752604 /lib/x86_64-linux-gnu/libtinfo.so.5.9 7f3b5d825000-7f3b5d829000 r--p 00025000 08:01 2752604 /lib/x86_64-linux-gnu/libtinfo.so.5.9 7f3b5d829000-7f3b5d82a000 rw-p 00029000 08:01 2752604 /lib/x86_64-linux-gnu/libtinfo.so.5.9 7f3b5d82a000-7f3b5d85b000 r-xp 00000000 08:01 10230780 /usr/lib/x86_64-linux-gnu/libedit.so.2.0.55 7f3b5d85b000-7f3b5da5b000 ---p 00031000 08:01 10230780 /usr/lib/x86_64-linux-gnu/libedit.so.2.0.55 7f3b5da5b000-7f3b5da5d000 r--p 00031000 08:01 10230780 /usr/lib/x86_64-linux-gnu/libedit.so.2.0.55 7f3b5da5d000-7f3b5da5e000 rw-p 00033000 08:01 10230780 /usr/lib/x86_64-linux-gnu/libedit.so.2.0.55 7f3b5da5e000-7f3b5da62000 rw-p 00000000 00:00 0 7f3b5da62000-7f3b5da6b000 r-xp 00000000 08:01 10233109 /usr/lib/x86_64-linux-gnu/libyajl.so.2.1.0 7f3b5da6b000-7f3b5dc6a000 ---p 00009000 08:01 10233109 /usr/lib/x86_64-linux-gnu/libyajl.so.2.1.0 7f3b5dc6a000-7f3b5dc6b000 r--p 00008000 08:01 10233109 /usr/lib/x86_64-linux-gnu/libyajl.so.2.1.0 7f3b5dc6b000-7f3b5dc6c000 rw-p 00009000 08:01 10233109 /usr/lib/x86_64-linux-gnu/libyajl.so.2.1.0 7f3b5dc6c000-7f3b5ded1000 r-xp 00000000 08:01 10229375 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 7f3b5ded1000-7f3b5e0d0000 ---p 00265000 08:01 10229375 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 7f3b5e0d0000-7f3b5e0ee000 r--p 00264000 08:01 10229375 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 7f3b5e0ee000-7f3b5e0fc000 rw-p 00282000 08:01 10229375 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 7f3b5e0fc000-7f3b5e0ff000 rw-p 00000000 00:00 0 7f3b5e0ff000-7f3b5e161000 r-xp 00000000 08:01 10229376 /usr/lib/x86_64-linux-gnu/libssl.so.1.1 7f3b5e161000-7f3b5e361000 ---p 00062000 08:01 10229376 /usr/lib/x86_64-linux-gnu/libssl.so.1.1 7f3b5e361000-7f3b5e365000 r--p 00062000 08:01 10229376 /usr/lib/x86_64-linux-gnu/libssl.so.1.1 7f3b5e365000-7f3b5e36b000 rw-p 00066000 08:01 10229376 /usr/lib/x86_64-linux-gnu/libssl.so.1.1 7f3b5e36b000-7f3b5e383000 r-xp 00000000 08:01 2752535 /lib/x86_64-linux-gnu/libpthread-2.24.so 7f3b5e383000-7f3b5e582000 ---p 00018000 08:01 2752535 /lib/x86_64-linux-gnu/libpthread-2.24.so 7f3b5e582000-7f3b5e583000 r--p 00017000 08:01 2752535 /lib/x86_64-linux-gnu/libpthread-2.24.so 7f3b5e583000-7f3b5e584000 rw-p 00018000 08:01 2752535 /lib/x86_64-linux-gnu/libpthread-2.24.so 7f3b5e584000-7f3b5e588000 rw-p 00000000 00:00 0 7f3b5e588000-7f3b5e589000 r-xp 00000000 08:01 10232429 /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.62.0 7f3b5e589000-7f3b5e788000 ---p 00001000 08:01 10232429 /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.62.0 7f3b5e788000-7f3b5e789000 r--p 00000000 08:01 10232429 /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.62.0 7f3b5e789000-7f3b5e78a000 rw-p 00001000 08:01 10232429 /usr/lib/x86_64-linux-gnu/libboost_atomic.so.1.62.0 7f3b5e78a000-7f3b5e799000 r-xp 00000000 08:01 10232625 /usr/lib/x86_64-linux-gnu/libboost_date_time.so.1.62.0 7f3b5e799000-7f3b5e999000 ---p 0000f000 08:01 10232625 /usr/lib/x86_64-linux-gnu/libboost_date_time.so.1.62.0 7f3b5e999000-7f3b5e99a000 r--p 0000f000 08:01 10232625 /usr/lib/x86_64-linux-gnu/libboost_date_time.so.1.62.0 7f3b5e99a000-7f3b5e99b000 rw-p 00010000 08:01 10232625 /usr/lib/x86_64-linux-gnu/libboost_date_time.so.1.62.0 7f3b5e99b000-7f3b5e9a0000 r-xp 00000000 08:01 10232433 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0 7f3b5e9a0000-7f3b5eba0000 ---p 00005000 08:01 10232433 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0 7f3b5eba0000-7f3b5eba1000 r--p 00005000 08:01 10232433 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0 7f3b5eba1000-7f3b5eba2000 rw-p 00006000 08:01 10232433 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0 7f3b5eba2000-7f3b5ecb4000 r-xp 00000000 08:01 10233097 /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0 7f3b5ecb4000-7f3b5eeb3000 ---p 00112000 08:01 10233097 /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0 7f3b5eeb3000-7f3b5eeb8000 r--p 00111000 08:01 10233097 /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0 7f3b5eeb8000-7f3b5eeba000 rw-p 00116000 08:01 10233097 /usr/lib/x86_64-linux-gnu/libboost_regex.so.1.62.0 7f3b5eeba000-7f3b5eebb000 rw-p 00000000 00:00 0 7f3b5eebb000-7f3b5ef36000 r-xp 00000000 08:01 10232627 /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0 7f3b5ef36000-7f3b5f136000 ---p 0007b000 08:01 10232627 /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0 7f3b5f136000-7f3b5f139000 r--p 0007b000 08:01 10232627 /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0 7f3b5f139000-7f3b5f13a000 rw-p 0007e000 08:01 10232627 /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.62.0 7f3b5f13a000-7f3b5f13d000 r-xp 00000000 08:01 10232431 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0 7f3b5f13d000-7f3b5f33c000 ---p 00003000 08:01 10232431 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0 7f3b5f33c000-7f3b5f33d000 r--p 00002000 08:01 10232431 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0 7f3b5f33d000-7f3b5f33e000 rw-p 00003000 08:01 10232431 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0 7f3b5f33e000-7f3b5f364000 r-xp 00000000 08:01 10233099 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0 7f3b5f364000-7f3b5f563000 ---p 00026000 08:01 10233099 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0 7f3b5f563000-7f3b5f565000 r--p 00025000 08:01 10233099 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0 7f3b5f565000-7f3b5f566000 rw-p 00027000 08:01 10233099 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0 7f3b5f566000-7f3b5f569000 r-xp 00000000 08:01 2752523 /lib/x86_64-linux-gnu/libdl-2.24.so 7f3b5f569000-7f3b5f768000 ---p 00003000 08:01 2752523 /lib/x86_64-linux-gnu/libdl-2.24.so 7f3b5f768000-7f3b5f769000 r--p 00002000 08:01 2752523 /lib/x86_64-linux-gnu/libdl-2.24.so 7f3b5f769000-7f3b5f76a000 rw-p 00003000 08:01 2752523 /lib/x86_64-linux-gnu/libdl-2.24.so 7f3b5f76a000-7f3b5f78d000 r-xp 00000000 08:01 2752515 /lib/x86_64-linux-gnu/ld-2.24.so 7f3b5f79b000-7f3b5f79c000 ---p 00000000 00:00 0 7f3b5f79c000-7f3b5f7dc000 rw-p 00000000 00:00 0 7f3b5f7dc000-7f3b5f7dd000 ---p 00000000 00:00 0 7f3b5f7dd000-7f3b5f81d000 rw-p 00000000 00:00 0 7f3b5f81d000-7f3b5f81e000 ---p 00000000 00:00 0 7f3b5f81e000-7f3b5f85e000 rw-p 00000000 00:00 0 7f3b5f85e000-7f3b5f85f000 ---p 00000000 00:00 0 7f3b5f85f000-7f3b5f89f000 rw-p 00000000 00:00 0 7f3b5f89f000-7f3b5f8a0000 ---p 00000000 00:00 0 7f3b5f8a0000-7f3b5f8f4000 rw-p 00000000 00:00 0 7f3b5f8f4000-7f3b5f978000 r-xp 00000000 08:01 2753196 /lib/x86_64-linux-gnu/libsystemd.so.0.17.0 7f3b5f978000-7f3b5f979000 ---p 00084000 08:01 2753196 /lib/x86_64-linux-gnu/libsystemd.so.0.17.0 7f3b5f979000-7f3b5f97c000 r--p 00084000 08:01 2753196 /lib/x86_64-linux-gnu/libsystemd.so.0.17.0 7f3b5f97c000-7f3b5f97d000 rw-p 00087000 08:01 2753196 /lib/x86_64-linux-gnu/libsystemd.so.0.17.0 7f3b5f97d000-7f3b5f97e000 rw-p 00000000 00:00 0 7f3b5f97e000-7f3b5f982000 rw-p 00000000 00:00 0 7f3b5f989000-7f3b5f98d000 rw-p 00000000 00:00 0 7f3b5f98d000-7f3b5f98e000 r--p 00023000 08:01 2752515 /lib/x86_64-linux-gnu/ld-2.24.so 7f3b5f98e000-7f3b5f98f000 rw-p 00024000 08:01 2752515 /lib/x86_64-linux-gnu/ld-2.24.so 7f3b5f98f000-7f3b5f990000 rw-p 00000000 00:00 0 7ffefe43b000-7ffefe45c000 rw-p 00000000 00:00 0 [stack] 7ffefe5d6000-7ffefe5d8000 r--p 00000000 00:00 0 [vvar] 7ffefe5d8000-7ffefe5da000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Caught SIGABRT. Current time: 2018-12-02 06:32:10 +0100

/var/log/icinga2/icinga2.log.1
[2018-12-02 06:28:39 +0100] information/WorkQueue: #16 (JsonRpcConnection, #7) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-12-02 06:28:40 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:29:00 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:29:30 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:29:40 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:29:59 +0100] information/WorkQueue: #12 (JsonRpcConnection, #3) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-12-02 06:30:00 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:30:23 +0100] information/WorkQueue: #13 (JsonRpcConnection, #4) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-12-02 06:30:28 +0100] information/WorkQueue: #11 (JsonRpcConnection, #2) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-12-02 06:30:30 +0100] information/WorkQueue: #14 (JsonRpcConnection, #5) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-12-02 06:30:30 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:30:40 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:31:00 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:31:15 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state' [2018-12-02 06:31:30 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:31:40 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min); [2018-12-02 06:31:40 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:32:00 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min); [2018-12-02 06:32:02 +0100] information/WorkQueue: #9 (JsonRpcConnection, #0) items: 0, rate: 0.15/s (9/min 45/5min 134/15min);

Let me know if you need more things.

fladna9 picture fladna9 on 3 Dec 2018

Just experienced (probably) the same crash on a fresh install (only two weeks old).
Due to its semi-production state, I didn't see that Icinga2 wasn't running anymore for the last 2 days. Only after a login into icingaweb2 the UI informed me that the backend is not running.

There was a report file created in /var/log/icinga2/crash:

# ls -la /var/log/icinga2/crash
total 24
drwxr-s--x 2 nagios adm 4096 Dec 22 06:25 .
drwxr-s--x 4 nagios adm 4096 Dec 23 06:25 ..
-rw-r--r-- 1 nagios adm 1529 Dec  8 13:06 report.1544270800.969342
-rw-r--r-- 1 nagios adm 1529 Dec  8 13:06 report.1544270814.937787
-rw-r--r-- 1 nagios adm 1529 Dec  8 13:07 report.1544270853.915306
-rw-r--r-- 1 nagios adm 2524 Dec 22 06:25 report.1545456308.298044

Content:

# cat /var/log/icinga2/crash/report.1545456308.298044 
  Application version: r2.10.2-1

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.18.0-0.bpo.1-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 6.3.0
  Build host: 486e413fb159

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Stacktrace:

    (0) libc.so.6: gsignal (+0xcf) [0x14cae1342fff]
    (1) libc.so.6: abort (+0x16a) [0x14cae134442a]
    (2) libc.so.6: <unknown function> (+0x70c00) [0x14cae1380c00]
    (3) libc.so.6: <unknown function> (+0x76fc6) [0x14cae1386fc6]
    (4) libc.so.6: <unknown function> (+0x7780e) [0x14cae138780e]
    (5) icinga2: icinga::StreamLogger::BindStream(std::ostream*, bool) (+0xa4) [0x5575de6591b4]
    (6) icinga2: icinga::FileLogger::ReopenLogFile() (+0xaa) [0x5575de60e45a]
    (7) icinga2: icinga::Application::RunEventLoop() (+0x7c2) [0x5575de5eb8f2]
    (8) icinga2: icinga::IcingaApplication::Main() (+0x1f4) [0x5575de7f4b64]
    (9) icinga2: icinga::Application::Run() (+0xc1) [0x5575de5e98a1]
    (10) icinga2: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) const (+0xbc9) [0x5575de7c4c19]
    (11) icinga2: <unknown function> (+0x266446) [0x5575de5ef446]
    (12) icinga2: main (+0x49) [0x5575de593889]
    (13) libc.so.6: __libc_start_main (+0xf1) [0x14cae13302e1]
    (14) icinga2: _start (+0x2a) [0x5575de5b0d6a]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

Failed to launch GDB: No such file or directory
Napsty picture Napsty on 24 Dec 2018

While not a pretty fix, this does help us sleep at night so perhaps it will help someone. We added the following file:

# /lib/systemd/system/icinga2.service.d/00-restart.conf
[Service]
Restart=on-failure

This will automatically restart Icinga2 while it crashes. Make sure to run systemctl daemon-reload after placing the file.

NielsH picture NielsH on 24 Dec 2018
👍1

@NielsH Thanks for sharing this. Please note that user-adjusted Systemd configs should reside in /etc/systemd, not /lib/systemd.
So I placed your workaround here: /etc/systemd/system/icinga2.service.d/00-restart.conf

Napsty picture Napsty on 24 Dec 2018
👍1

I have the same issue on Debian 8 and 9 systemd machines/VMs, with 2.10.2-1. The icinga2 instances on a few machines crash once in a while. Thanks @NielsH @Napsty I'll apply your workaround until the root cause of the issue is identified/fixed.

jandd picture jandd on 9 Jan 2019

We also encouter such issues with logrotate with icinga2 agents updated recently. At least the 2.9.0 seems to be fine but the 2.10 version we have around show the problem.
We have the issue on Centos 7 with systemd and Centos 6 without systemd.
We can see the crash happens diretly after logrotate created the new empty log file and probably sent SIGUSR1 immediately before crash.

From one Centos 6 system I now got a core dump after fixing abrtd config (but for some reason again no crash report from icinga2). I doubt that for this case (similiar to at least one of the above) the double-free fix will help but assume as it looks the crash happens during creation of new timer object. And I can see that the time module was heavily changed between 2.9.0 and 2.9.1 regarding initialiszation/threading etc. so it might be worth to take a look on that part.
```(gdb) bt

0 icinga::Timer::__base_dtor (this=0x31d6f60) at /usr/include/boost153/boost/signals2/detail/slot_groups.hpp:95

1 0x0000000000c4c729 in icinga::Timer::__deleting_dtor (this=0x31d6f60) at ../base/timer.cpp:83

2 0x0000000000c95f14 in __base_dtor (this=0x2ac164000cb0, stream=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/smart_ptr/intrusive_ptr.hpp:97

3 operator= (this=0x2ac164000cb0, stream=Unhandled dwarf expression opcode 0xf3

) at /usr/include/boost153/boost/smart_ptr/intrusive_ptr.hpp:135

4 BindStream (this=0x2ac164000cb0, stream=Unhandled dwarf expression opcode 0xf3

) at ../base/streamlogger.cpp:75

5 0x0000000000c96385 in ReopenLogFile (this=0x2ac164000cb0) at ../base/filelogger.cpp:75

6 0x0000000000cf7e91 in operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/function/function_template.hpp:767

7 m_invoke (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:89

8 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:78

9 m_invoke (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:114

10 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:107

11 dereference (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/slot_call_iterator.hpp:82

12 dereference (this=0x2ac130000a10) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:514

13 operator* (this=0x2ac130000a10) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:639

14 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/optional_last_value.hpp:55

15 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/result_type_wrapper.hpp:64

16 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:246

17 operator() (this=0x2ac130000a10) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:695

18 icinga::Application::RunEventLoop (this=0x2ac130000a10) at ../base/application.cpp:320

19 0x0000000000cf9659 in icinga::IcingaApplication::Main (this=0x2ac130000a10) at ../icinga/icingaapplication.cpp:119

20 0x0000000000d4e303 in icinga::Application::Run (this=0x2ac130000a10) at ../base/application.cpp:1033

21 0x0000000000e7ffc8 in Run (this=Unhandled dwarf expression opcode 0xf3

) at ../cli/daemoncommand.cpp:320

22 0x0000000000e8c99c in Main () at icinga.cpp:681

23 0x0000000000a34eb7 in main (argc=10, argv=0x7ffddae8f3e8) at icinga.cpp:975```

Elias481 picture Elias481 on 12 Feb 2019
👍1

Thanks for sharing your findings. I'm leaving this as feedback for others to share more insights. This is a nasty one, on my TODO list and I am working to figure out the underlaying problem.

dnsmichi picture dnsmichi on 14 Feb 2019
❤1 👍1

We had exactly same picture on next icinga2 "logrotate"-crash on one of the machines I monitor for this issue.
I now took some time to review changes to timer module but in general the workflow did not change in a way that could directly explain these now appearing errors as far as I can see.

But if I'm correct with my limit understanding of C++ and boost magic - and icinga2 code - the following can happen:

  • The timer thread TimerThreadProc is in the area before line 300 (so has no counted reference to the timer) where it pulls the old FlushLogTimer and determines it is ready for execution
  • The BindStream method decrements reference count of old timer when assigning the new created timer to the m_FlushLogTimer -> the ref count drops to zero and the icinga::intrusive_ptr_release in turn decides to delete the object instance which causes the main thread to call destruction sequence for the object
  • The timer thread TimerThreadProc acquires a reference to the old timer in line 300 incrementing it's use count to 1 and starts it asynchonously on a thread pool worker thread (possibly temporary incrementing refcount with the bind operation but realease it own ref at end of scope[loop] so it is at 1 so to say)
  • The old timer finishes eighther before the destructor acquires the mutex for the Stop method (unlikely as the destructor should went there much faster than it takes to call the tread asynchronously) or it finishes when the Stop method wait for the current time run to finish
  • Now, before the already running destructor finishes, the Ptr on the timer used for running the timer in thread pool is deallocated, the refcount drops to 0 and again a delete is triggered _this delete is fast (it shortcuts in the Stop method) and finishes before the destructor running in main thread finishes - and even if it wouldn't be very fast, it can take an undefined time for the waiting Stop to get a notify (the asynchronously run thread waited for will not issue such a notify in case it is already deactivated which is the case here - which also looks like a bug for me but it probably nearly never causes issues if there are timers running with short reschedule times as they issue notify-all on reschedule..) and possibly some more time as another thread running timer code could get the mutex before the destructor itself_
  • The destructor of old timer running in main thread continues running but and the end of the destructor the further cleanup actions may fail as a cleanup already took place by another thread

The same thing could happen with older code 2.9.0 etc. - but the thread timing and memory reuse pattern which can influence whether a hard crash happens on duplicate delete (or just some minor memory corruption, somehow gracful handled excpetion or even nothing bad) could be changed by many parts of the code. So if my understanding regarding the reference counting and magical destruction of unused object by reference count is correct this is a thing that needs to be fixed anyway as doing delete twice leads to undefined behaviour including such "random" crashes and it was just luck when it did not happen with older code.

It shouldn't be too nasty to fix that when knowing C++ and boost and icinga2 code structure. Without much knowledge about them I assume one can just omit the Line 321 in timer.cpp and do the bind in 332 directly to the plain pointer? The destructor will wait for the timer to finish anyway (first on the mutex hold by timer thread and afterwards on the m_Running field) so no need to have a instrusive_ptr for this - and without instrusive_ptr also no decrement/duplicate free not done.

Elias481 picture Elias481 on 17 Feb 2019
❤1

So and now we got few (3) more coredumps over the weekend.
Two of them are quite interesting as it looks like they shows exactly the issue as I assumed - but in the case the destructor from thread pool is a bit slower and the one that runs into error because abject already deleted. (The third is again the crash in main thread as above.)
```(gdb) bt

0 icinga::Timer::__base_dtor (this=0x23bcf00)

at /usr/include/boost153/boost/signals2/detail/slot_groups.hpp:95

1 0x0000000000c4c729 in icinga::Timer::__deleting_dtor (this=0x23bcf00)

at ../base/timer.cpp:83

2 0x0000000000d527a8 in __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3

)
at /usr/include/boost153/boost/smart_ptr/intrusive_ptr.hpp:97

3 __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3

)
at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/tuple:128

4 __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3

)
at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/tuple:229

5 __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3

)
at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/tuple:388

6 __base_dtor (__dest=Unhandled dwarf expression opcode 0xf3

)
at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:1280

7 std::_Function_base::_Base_manager)> >::_M_destroy (__dest=Unhandled dwarf expression opcode 0xf3

)
at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:1926

8 std::_Function_base::_Base_manager::_M_manager (__dest=Unhandled dwarf expression opcode 0xf3

)
at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:1950

9 0x0000000000cff25c in __base_dtor (this=0x131a9a8, queue=...)

at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:2030

10 __base_dtor (this=0x131a9a8, queue=...)

at /opt/rh/devtoolset-2/root/usr/include/c++/4.8.2/functional:2174

11 __base_dtor (this=0x131a9a8, queue=...) at ../base/threadpool.hpp:68

12 icinga::ThreadPool::WorkerThread::ThreadProc (this=0x131a9a8, queue=...)

at ../base/threadpool.cpp:184

13 0x00000034ee80c5c3 in ?? () from /usr/lib64/libboost_thread.so.1.53.0

14 0x00000034e9a07aa1 in start_thread () from /lib64/libpthread.so.0

15 0x00000034e96e8c4d in clone () from /lib64/libc.so.6

For reference we can see in such cases the Main Thread that is exactly at that line (but not the one catching the SEGFAULT):
```(gdb) bt
#0  boost::signals2::detail::signal_impl::connect (this=0x2388400, slot=..., position=boost::signals2::at_back) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:160
#1  0x0000000000c9606a in connect (this=0x2b8fc0002680, stream=Unhandled dwarf expression opcode 0xf3
) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:664
#2  BindStream (this=0x2b8fc0002680, stream=Unhandled dwarf expression opcode 0xf3
) at ../base/streamlogger.cpp:77
#3  0x0000000000c96385 in ReopenLogFile (this=0x2b8fc0002680) at ../base/filelogger.cpp:75
#4  0x0000000000cf7e91 in operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/function/function_template.hpp:767
#5  m_invoke (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:89
#6  operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:78
#7  m_invoke (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:114
#8  operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/variadic_slot_invoker.hpp:107
#9  dereference (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/slot_call_iterator.hpp:82
#10 dereference (this=0x2b8fc0000c80) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:514
#11 operator* (this=0x2b8fc0000c80) at /usr/include/boost153/boost/iterator/iterator_facade.hpp:639
#12 operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/optional_last_value.hpp:55
#13 operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/result_type_wrapper.hpp:64
#14 operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:246
#15 operator() (this=0x2b8fc0000c80) at /usr/include/boost153/boost/signals2/detail/signal_template.hpp:695
#16 icinga::Application::RunEventLoop (this=0x2b8fc0000c80) at ../base/application.cpp:320
#17 0x0000000000cf9659 in icinga::IcingaApplication::Main (this=0x2b8fc0000c80) at ../icinga/icingaapplication.cpp:119
#18 0x0000000000d4e303 in icinga::Application::Run (this=0x2b8fc0000c80) at ../base/application.cpp:1033
#19 0x0000000000e7ffc8 in Run (this=Unhandled dwarf expression opcode 0xf3
) at ../cli/daemoncommand.cpp:320
#20 0x0000000000e8c99c in Main () at icinga.cpp:681
#21 0x0000000000a34eb7 in main (argc=10, argv=0x7ffc16e17ae8) at icinga.cpp:975
Elias481 picture Elias481 on 18 Feb 2019

I found some time to verify the issue we have with icinga2 dying on SIGUSR1.
If the threads are synchronized "properly" (like so: https://github.com/Elias481/icinga2/compare/master...Elias481:bugverify/logrotate-segfault-6737) it is very reproducible (each SIGUSR1 leads to the SEGFAULT).

Also I verified that these can be fixed with different approaches:

  • Fix the logic that there is no intrusive_ptr_add_ref triggered from within timer thread/module itself (this implies a minor API change as the possible argument to OnTimerExpired callback has to be a plain pointer then): https://github.com/Elias481/icinga2/compare/master...Elias481:fix/logrotate-segfault-6737
    _The same with the other logical thread-synchronization error mentioned above somehow fixed: https://github.com/Elias481/icinga2/compare/master...Elias481:fix/logrotate-segfault-6737-complete_
  • Global Workaround for double-free because of such issues by patching intrusive_ptr_release: https://github.com/Elias481/icinga2/compare/master...Elias481:fix/logrotate-segfault-6737-globalworkaround ### _(not recommended as it avoids just double free but does not ensure the object are not deleted before they are really thrown away by all consumers that are in good believe they have a owning reference)_
  • Local Workaround by Stopping Timer before exposing it to intrusive_ptr_release: https://github.com/Elias481/icinga2/compare/master...Elias481:fix/logrotate-segfault-6737-localworkaround (this one cannot be verified with the thread synchronization patch from above but should also help because the timer thread will not trigger intrusive_ptr_add_ref for an inactive timer)

And there are probably some more ways to fix this issue..

Elias481 picture Elias481 on 3 Mar 2019

Hello guys and thank you very much for investigation!

Please could you test the linked PR?

Best,
AK

Al2Klimov picture Al2Klimov on 11 Mar 2019

I doubt that this will helps for this as the issue of possible re-incrementing refcount after it dropped to zero remains persistent regardless of the change visible in that in commit.
But I can merge it on my bugverify branch to verif that.

Elias481 picture Elias481 on 11 Mar 2019

Yes, please!

Al2Klimov picture Al2Klimov on 11 Mar 2019

As expected this changes nothing (the bind was not involved at all).

If You did what You wrote with Your commit (use C++11 lambda instead of shared pointer) it could have been a fix if You really removed the causing Ptr allocs as in https://github.com/Elias481/icinga2/compare/master...Elias481:fix/logrotate-segfault-6737 _but You can see that no change from bind to lambda is needed to archive that_.

But You only did what You told in Your PRs (Timer::TimerThreadProc(): use C++11 lambda instead of bind()) that changed nothing (Ptr's involved are still the same - or possibly one less, but if there is really one less the one that's omitted now never counted toward the problem).

If You want to verify Yourself feel free to use https://github.com/Elias481/icinga2/tree/bugverify/logrotate-segfault-6737-testproposal to do that.

Elias481 picture Elias481 on 11 Mar 2019

All the others will be changed as well: #7006

Al2Klimov picture Al2Klimov on 12 Mar 2019

But thats is not related to this issue...

Elias481 picture Elias481 on 12 Mar 2019

We now had another error with possibly/hopefully same cause, this time on a satellite with of course more concurrent activity as on the agents:

*** Error in `/usr/lib64/icinga2/sbin/icinga2': malloc(): memory corruption (fast): 0x00000000029f4320 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7f5d4)[0x2b8d02b385d4]
/lib64/libc.so.6(+0x82d10)[0x2b8d02b3bd10]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x2b8d02b3e1ac]
/usr/lib64/icinga2/sbin/icinga2[0x5f94bf]
/usr/lib64/icinga2/sbin/icinga2[0x5f955b]
/usr/lib64/icinga2/sbin/icinga2(yajl_parse+0x49)[0x82cd79]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga10JsonDecodeERKNS_6StringE+0x3e1)[0x82d551]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga7JsonRpc13DecodeMessageERKNS_6StringE+0x23)[0x879ed3]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga17JsonRpcConnection14MessageHandlerERKNS_6StringE+0x37)[0x8acc17]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga17JsonRpcConnection21MessageHandlerWrapperERKNS_6StringE+0x5b)[0x9873db]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga9WorkQueue15RunTaskFunctionERKSt8functionIFvvEE+0x29)[0x779ff9]
/usr/lib64/icinga2/sbin/icinga2(_ZN6icinga9WorkQueue16WorkerThreadProcEv+0x5e7)[0x780c57]
/lib64/libboost_thread-mt.so.1.53.0(+0xd27a)[0x2b8d0099027a]
/lib64/libpthread.so.0(+0x7dd5)[0x2b8d028a4dd5]
/lib64/libc.so.6(clone+0x6d)[0x2b8d02bb6ead]

Stacktrace:

        (0) libc.so.6: gsignal (+0x37) [0x2b8d02aef207]
        (1) libc.so.6: abort (+0x148) [0x2b8d02af08f8]
        (2) libc.so.6: <unknown function> (+0x78d27) [0x2b8d02b31d27]
        (3) libc.so.6: <unknown function> (+0x7f5d4) [0x2b8d02b385d4]
        (4) libc.so.6: <unknown function> (+0x82d10) [0x2b8d02b3bd10]
        (5) libc.so.6: __libc_malloc (+0x4c) [0x2b8d02b3e1ac]
        (6) /usr/lib64/icinga2/sbin/icinga2() [0x5f94bf]
        (7) /usr/lib64/icinga2/sbin/icinga2() [0x5f955b]
        (8) icinga2: yajl_parse (+0x49) [0x82cd79]
        (9) icinga2: icinga::JsonDecode(icinga::String const&) (+0x3e1) [0x82d551]
        (10) icinga2: icinga::JsonRpc::DecodeMessage(icinga::String const&) (+0x23) [0x879ed3]
        (11) icinga2: icinga::JsonRpcConnection::MessageHandler(icinga::String const&) (+0x37) [0x8acc17]
        (12) icinga2: icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) (+0x5b) [0x9873db]
        (13) icinga2: icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) (+0x29) [0x779ff9]
        (14) icinga2: icinga::WorkQueue::WorkerThreadProc() (+0x5e7) [0x780c57]
        (15) libboost_thread-mt.so.1.53.0: <unknown function> (+0xd27a) [0x2b8d0099027a]
        (16) libpthread.so.0: <unknown function> (+0x7dd5) [0x2b8d028a4dd5]
        (17) libc.so.6: clone (+0x6d) [0x2b8d02bb6ead]

Sadly no coredump or debug info available (core dumps not configured for suid processes and processes that cannot write to cwd and gdb not isntalled).

I assume in this case the same happended as in the other cases but before the double-free with 2nd delete occurred another thread tried to allocate some memory of still running thread. At least I do not find another valid reason for this to happen at exactly logrotate time.

Elias481 picture Elias481 on 9 Apr 2019

I believe that this originates from YAJL leaking memory. YAJL is the JSON parser library we've been using, and now have replaced in 2.11 with #7019. You might want to give the snapshot packages a try, to see whether this problem has been solved.

https://icinga.com/docs/icinga2/snapshot/doc/21-development/#snapshot-packages-nightly-builds

dnsmichi picture dnsmichi on 10 Apr 2019
👍1

For me it's hard to believe that this happens exactly during log-rotate time where this well-known issue exists and each day has 24 hours. So I believe that the original issue in #6684 happening at logrotate time could just be the same issue as this one as I have absolutely no prove for the opposite. :-) And for the other one reported in that issue (https://github.com/Icinga/icinga2/issues/6684#issuecomment-441987060) I believe... ...I believe there is another bug in icinga2 at reload logic that leaves room for similiar race condition... We also observed a crash (SIGBUS) at reload time but sadly without any information (beside the SIGBUS error in syslog) so I could not investigate.

I don't think I will give a 2.11 snapshots a try because a) we got this issue seldom (possibly first time) and b) I still wait for the three investigated segfault bugs in icinga2 from which two hit us regulary are fixed (all three should be simple to fix, at least for this one and for #6874 it's pretty clear what to do and for the third one the cause is also identified and a fix is simple to implement but not that abvious which is the straight-forward way for a bug-fix) - and finally c) I don't believe and we encountered no problems with UTF-8 special chars and anyway what I see is related to #7019 besides the things I don't belive are related. And I would prefer to mitigate this at the source or destination (just use correct UTF-8 where possibly some way)...

Elias481 picture Elias481 on 10 Apr 2019

Thanks for your thoughts and input. I can only offer you to test things already available, everything else is work in progress.

I tend to disagree on "should be simple to fix", as the analysis in the issue has proven that the problem exists, but is not an easy one to reproduce. Since you're experiencing this in your environment, and obviously have some more deep knowledge about C++ and signal handling, it might be a good effort to patch this and send in a PR with analysis before and after.

I'm not sure how the UTF8 things relate here, that's just a different issue having been fixed with the JSON library replacement. The main thing is that memory corruption and leaks are hard to find, and at the time this crashes, they may have influenced this but the root cause lies in a different scope.

I don't believe that this is the true cause for logrotate either, but testing the removed YAJL problem allows us to mitigate the problem further. The stacktrace on the crash will definitely look different, and hopefully gives more insights on this. Therefore I am asking you and everyone else experiencing the problem to test the snapshot packages.

If we don't find anything here in a reasonable amount of time, there's two options: a) Block 2.11 release, meaning to say everyone else that's waiting for the network stack etc. b) Remove this from 2.11 and leave it for later. That's bad for everyone else here.

Cheers,
Michael

dnsmichi picture dnsmichi on 11 Apr 2019

Regarding not wanting to test this: It's just not really testable for us because the last one, aborting in yayl, is not common in our environment and as the other ones. And if it is a separate issue, it thus had lower priority.
As the other ones are verified (including possible fix and some other ways to mitigate this) I just don't believe the yayl at logrotate time is a yayl fault. Accessing an already deleted object or it's memory while the object is still accessed (as it is the case often as verified) can possibly lead to such memory allocation errors in random other scopes, as You confirmed.
So testing the yayl replacement from #7019 is something that could make sense for miso231 who claimed in #6684 to get this error condition on logrotate frequently. Not sure as I did not verify but I think it's quite likely he then will get these errors (which directly point to the roots) or similiar error from other json parser if it does some similiar memory allocation. But this is only what I assume seeing relation logotate time which does not trigger any json parsing related stuff or influence it.

For the crashes that are reported in this issue I did a verification that it is related to unsafe refcounting. And I made one clean fix/patch changing as less as possible just fixing the issue with only a small "interface" change (OnTimerExpired callback does not return a Ptr object with reference but just a plain pointer - and thus had to be careful not to increment it's refcount or copy it byside for later use without knowing the object is still alive then - but this parameter to OnTimerExpired is nowhere used beside test case).
Another possibility would be a local fix in StreamLogger destructor but this does not fix the bug with the Timer module refcounting so I would not recommend.
The third option is now, seeing the other errors, something that I would strike-through (it's anyway a dirty solution and would not avoid such errors as can be seen at json parsing if they are related - it was just an example to verify bug).

One could prefer to do a bigger rework to keep the Timer-Scheduler keeping references on its Timers to not have the interface change described above, but that's not as simple. Sure I could do if that would be Your requirement for a fix but it would be some efford I would only spent if really needed.

Sure I can put it in a PR with analysis before and after - the one I have.

I could also try prove whether the yayl abort is reproducible if synchronizing that part to the other I already synchronized for the verification in such a way that the malloc from yayl occurs after one of the parties finished destruction completely (freed memory) but before the other one does the same and raises the double-free SEGV (so the one that is not the first destructor waits for a nitification from yayl).
Some efford and it has the side effect that even more extra code would need to be added so that it would be even more difficult to verify that it's just showing a race condition of original code and not an error introduced by these synchronization mechanisms.
So I would prefer to just implement a fix for this (the small and punctual one I already did or a bigger more complicated one that causes the timer-scheduler to be the Owner of all Timers that are in it's queue or currently executed by the scheduler) and see whether there are more - or if someone has the yayl error more often he could test with that version.

So what's Your requirement? Just a PR from what I already did or does it need to verify whether the yayl error is covered or is it required to provide OnTimerExpired callback with a Ptr object with valid refcount while at the same time keeping the auto-desctruction behaviour of the Timer objects which would make a fix much more complex?

I don't have a deep knowledge of C++ and signal-handling is also nothing I'm really deep in. But this is not really related to signal handling that far and I have quite deep understanding of programming/threading logic which is the same across all languages at the end/behind the curtins for the hardware we speak about (we don't speak about quantum-computers and such stuff where I expect some major differences).

So for the other Issues I can comment there (a bit shorter).

Elias481 picture Elias481 on 11 Apr 2019

Hello @Elias481 and thank you for the inspiration!

With #7129 (I think) I've found the root cause.
Please could you test #7130?

Best,
AK

Al2Klimov picture Al2Klimov on 18 Apr 2019

review done - #7130 does not change anything and is not addressing rootcause. the root-cause is fixed with #7129.

if you want a workarounf like that 7130 which does not address the root-cause I posted something that is working instead in comment above:
https://github.com/Elias481/icinga2/commit/9917e3dd12a83e5253c94c4774bba5caa68dbe81

As I'm not a fan to not fix the root-cause but just implement a local workaround I will not make a PR for that. But You could indeed make a merge of what's show there (it's already tested - and for the problem observiced within the issue the second one change would be the least needed - but then one more place stays unfixed....)

Elias481 picture Elias481 on 18 Apr 2019

Hi,

thanks for the critical section patch to force the application crash, this really helped in analysing all the possible patches.

Here's what we've tested today.

kill -USR1 $(pidof icinga2)

Nothing

Git Master, 02db12ae024fe8704b24715402d2c475fce0efbb

Result: Segfault.

Only initialize Flush Timer once

From #7130.

Result: Icinga doesn't crash, but the log flush is running into a lock, enforced by the CS patch and different circumstances.

Timer const*

From #7129.

Result: Icinga doesn't crash and continues to log on stdout and mainlog.

Combine flush timer once with timer const*

Result: Works fine, doesn't have any other influences.

Conclusion, first part

Using a const pointer instead of an intrusive ptr fixes the problem, still the flush timer "re-newal" is totally unnecessary and needs to be fixed as well.

lambda function instead of std::bind

From #7004.

Result: Doesn't fix, crashes too.

lambda function and flush

Result: Doesn't crash.

Conclusion, second part

Somehow std::bind is involved here with intrusive_ptr, starting with the changes in 2.9.

A const* pointer or using a lambda bind is effectively keeping the pointer reference, and the call continues, no application crashes involved.

icinga2_logrotate_crash_analysis.patch.txt

dnsmichi picture dnsmichi on 18 Apr 2019

After some research within #7006 I've come to the following conclusion:

  • lambdas capture by reference, still holding a reference on the timer object
  • lambdas are more likely to be inlined by the compiler than function objects with std::bind
  • std::bind resp. std::function require a copy or move constructor, always binding the value and using that when invoke happens later.
  • Timer::Call happens just once inside the application thread pool, there's no need to keep the returned function object around and re-use it multiple times.
  • The function object copies the pointer, and while another thread may already have freed it, it tries to access the address resulting in a crash
  • Lambdas hold a reference on the passed pointer and actually capture its value on execution (later in the TP).
  • Best practice are lambdas over std::bind with C++11 and future versions.

You can avoid the intrusive_ptr going out of scope by either making this a const raw pointer, or by passing its reference into a lambda function.

Since the Timer class explicitly uses raw pointers instead of intrusive_ptr's, I'd stay there and combine the three patches with 1) const Timer* 2) lambdas 3) avoid re-newing the timer.

In terms of moving away from std::bind to lambdas, follow my research here: https://github.com/Icinga/icinga2/issues/7006#issuecomment-486098608

dnsmichi picture dnsmichi on 24 Apr 2019

Snapshot packages for Debian/Ubuntu need to be fixed in our build infra, the rest is available.

dnsmichi picture dnsmichi on 25 Apr 2019

ref/IP/13853

dnsmichi picture dnsmichi on 25 Apr 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

lamaral picture lamaral  ·  4Comments
ggzengel picture ggzengel  ·  4Comments
ak2766 picture ak2766  ·  7Comments
adamparker picture adamparker  ·  5Comments
seventh-chord picture seventh-chord  ·  5Comments