Icinga2: Run icinga2 in foreground in containers

[jkroepke]

Running icinga2 in foreground will be terminated on config reload ...

Expected Behavior

icinga2 runs in foreground

Current Behavior

starting /usr/sbin/icinga2 daemon (on a satellite host) will start in the process in the foreground. On a config reload, like

[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//.timestamp
[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/001-director-basics.conf
[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/commands.conf
[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/host_templates.conf
[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/hostgroups.conf
[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/service_apply.conf
[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/service_templates.conf
[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/servicegroups.conf
[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/timeperiods.conf
[2017-05-05 10:49:17 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/user_templates.conf
[2017-05-05 10:49:17 +0000] information/ApiListener: Restarting after configuration change.
[2017-05-05 10:49:19 +0000] information/Application: Got reload command: Starting new instance.
[2017-05-05 10:49:19 +0000] information/Application: Received request to shut down.
[2017-05-05 10:49:20 +0000] information/Application: Shutting down...

icinga starts a new instance in the background while the initial process would be terminated.

Possible Solution

Run a init process like supervisor. But thats not how docker works.

Steps to Reproduce (for bugs)

1. Create a icinga2 master
2. Create a icinga2 satellite
3. Run icinga2 daemon from shell
4. Wait for a cluster sync w/ new changes
5. Icinga will be terminated on the shell.

Context

Running an icinga2 agent on a docker-only OS like Atomic, CoreOS. ...

Your Environment

• Version used (icinga2 --version): v2.6.3
• Operating System and version: CentOS 7
• Enabled features (icinga2 feature list): api mainlog
• Icinga Web 2 version and modules (System - About): N/A
• Config validation (icinga2 daemon -C): Fine
• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

[lazyfrosch]

This can be tricky. We currently have a problem that we don't hand over STDIN / STDERR to the new process.

But I'm not sure if that would be enough since we start the new daemon in background...

[jkroepke]

In bash, if I start new process with exec the new process will be replace the current one.. This is mandory for docker, since the container will be killed if the pid 1 gets died. Maybe icinga2 can use this logic, too.

I think the STDOUT problem can be easy workaround (for docker is good enough) with a symlink, like /var/log/icinga2/icinga2.log -> /proc/1/fd/1 .

[lazyfrosch]

I think that might be a bit more complex internally...

[dnsmichi]

Use something like supervisord as a workaround for now.

[jkroepke]

supervisord doen't work here for same reasons. Its untypical that daemons
forks itself and die.
Michael Friedrich [e-mail] schrieb am Mo. 8. Mai 2017 um
10:34:

Use something like supervisord as a workaround for now.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Icinga/icinga2/issues/5230#issuecomment-299806368,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABfQC8I4UICebtScwtYdVo_sbtQTBXbXks5r3tN4gaJpZM4NRyC_
.

[dnsmichi]

The daemon doesn't die, it shuts down and hands over to the child process which is spawned for reloads. That's a built-in reload mechanism and cannot be changed that easy.

The issue is valid, but I don't have any idea nor solution for that.

[jkroepke]

The issue is valid, but I don't have any idea nor solution for that.
I'm open for any workarounds, 3rd party tools, e.g.

I tried supervisord but supervisord checks if the pid is alive.

[jkroepke]

@dnsmichi do you have any status for me?

[Crunsher]

There is currently nobody working on this issue.

[bodsch]

@jkroepke I had experimented a lot with supervisord and s6 in docker containers.
Both worked satisfactorily - in my test cases.
In my current containers, however, I no longer use them, but use a small script that controls certificates, configurations and processes.

[Al2Klimov]

I had this problem too while building Icinga 2 Docker containers. I worked around this e.g. here by telling supervisord not to restart Icinga 2. This way on reload supervisord thinks Icinga 2 has shut down and just does nothing at all. Actually Icinga 2 keeps running – in the background. To see the logs, I do while cat /var/log/icinga2/icinga2.log >/proc/1/fd/1; do true; done by supervisord.

[Al2Klimov]

Thought about how this could be solved for all OSes with as less hacks as possible:

The daemon command should become an umbrella process ASAP (if not invoked with -C). This umbrella process spawns the actual daemon which loads the config etc.. Once the daemon wants to be reloaded, it tells the umbrella process to start yet another daemon. Then the two daemon processes interact as before except no one closes the standard I/O so both of the processes log to stdout/err as if they run in foreground.

This would also solve the reload problem on Windows – @LordHepipud had already suggested the same solution for that.

Problem:

I'd like to have as less different behaviors as possible – i.e. there shall always be an umbrella process, whether -d is given or not. But at the moment -d is handled after config load.

@dnsmichi Is there a reason why -d is being handled after config load?