Icinga2: Problem while safe-reload

[Duffkess]

When I reload icinga2 deamon using systemctl reload icinga2 I get the following error:

-- Unit icinga2.service has begun reloading its configuration Feb 13 12:07:02 a.monitoring.node safe-reload[32018]: Validating config files: /usr/lib/icinga2/safe-reload: line 27: /tmp/tmp.QAlEpw7H9w: Permission denied Feb 13 12:07:02 a.monitoring.node safe-reload[32018]: Failed

When I set setenforce 0 it just works fine.

Its a completly new installation of icinga2 without any modifications yet.

• icinga2.service is running under icinga user, the reload command is started by root
• Version used (icinga2 --version): 2.11.2-1
• Operating System and version: Centos 8
• Enabled features (icinga2 feature list):
  
  
  
  • Disabled features: command compatlog debuglog elasticsearch gelf graphite opentsdb statusdata syslog
  
  
  • Enabled features: api checker ido-mysql influxdb livestatus mainlog notification perfdata
  
  
• Icinga Web 2 version and modules (System - About): 2.7.3

• Config validation (icinga2 daemon -C):
  [2020-02-13 14:31:44 +0100] information/cli: Icinga application loader (version: 2.11.2-1)
  [2020-02-13 14:31:44 +0100] information/cli: Loading configuration file(s).
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Committing config item(s).
  [2020-02-13 14:31:44 +0100] information/ApiListener: My API identity: a.monitoring.node
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 1 InfluxdbWriter.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 1 LivestatusListener.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 1 FileLogger.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 7 NotificationCommands.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 10 Hosts.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 1 PerfdataWriter.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 6 Zones.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 4 Endpoints.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 2 ApiUsers.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 2 UserGroups.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 1 ApiListener.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 265 CheckCommands.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 10 TimePeriods.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 2 Users.
  [2020-02-13 14:31:44 +0100] information/ConfigItem: Instantiated 131 Services.
  [2020-02-13 14:31:44 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
  [2020-02-13 14:31:44 +0100] information/cli: Finished validating the configuration file(s).

• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.

Permissions on /tmp are just normal:
drwxrwxrwt. root root

After the reload icinga2 deamon has been failed and needs to be started again. stopping and starting is no problem.

It looks like there is a problem with impersonification not enabled in the selinux policy (just a gues, not an expert in this..)

Thanks for the great work.

[dgoetz]

Can you please provide:

• Output of semodule -l | grep -e icinga2 -e nagios -e apache
• Output of ps -eZ | grep icinga2
• Output of ls -laZ /tmp
• Output of audit2allow -li /var/log/audit/audit.log

Could be related to CentOS 8 specific changes.

[Duffkess]

[root@a ~]# semodule -l | grep -e icinga2 -e nagios -e apache
apache
icinga2
nagios

[root@a ~]# ps -eZ | grep icinga2
system_u:system_r:icinga2_t:s0 10013 ? 00:00:11 icinga2
system_u:system_r:icinga2_t:s0 10069 ? 00:00:10 icinga2
system_u:system_r:icinga2_t:s0 10070 ? 01:07:15 icinga2

[root@a ~]# ls -laZ /tmp
total 68
drwxrwxrwt. 14 root root system_u:object_r:tmp_t:s0 4096 Feb 14 10:19 .
dr-xr-xr-x. 17 root root system_u:object_r:root_t:s0 224 Feb 3 16:03 ..
drwxrwxrwt. 2 root root system_u:object_r:user_fonts_t:s0 4096 Feb 4 11:17 .font-unix
drwxrwxrwt. 2 root root system_u:object_r:user_tmp_t:s0 4096 Feb 4 11:17 .ICE-unix
drwxrwxrwt. 2 root root system_u:object_r:unlabeled_t:s0 16384 Feb 4 10:49 lost+found
-rwxrwxrwt. 1 root root unconfined_u:object_r:user_tmp_t:s0 3951 Feb 13 12:06 mktemp
drwxrwxrwt. 3 root root system_u:object_r:tmp_t:s0 4096 Feb 10 16:26 systemd-private-23229a75108b4dde943cf2d15866ca14-httpd.service-ZQszox
drwxrwxrwt. 3 root root system_u:object_r:tmp_t:s0 4096 Feb 10 11:41 systemd-private-23229a75108b4dde943cf2d15866ca14-php-fpm.service-t6Yzcj
drwxrwxrwt. 2 root root system_u:object_r:tmp_t:s0 4096 Feb 4 11:17 .Test-unix
drwxrwxrwt. 2 root root system_u:object_r:user_tmp_t:s0 4096 Feb 4 11:17 .X11-unix
drwxrwxrwt. 2 root root system_u:object_r:tmp_t:s0 4096 Feb 4 11:17 .XIM-unix
I guess you are only interessted in mktemp but I give you all output.

[root@a ~]# audit2allow -li /var/log/audit/audit.log

============= init_t ==============

allow init_t httpd_tmp_t:dir read;

Thanks

[dgoetz]

Some things that make me wonder:

• The required permission allow init_t httpd_tmp_t:dir read; would mean icinga2's start script has not transitioned and is already trying to read a file in a directory with type httpd_tmp_t. Something I can not change and I can not give the permission as it would be a problem of the base policy.
• safe_reload creates a file in /tmp with mktemp which is then chcon to icinga_tmp_t and written with the output from the config validation and not icinga2 itself which would explain init_t
• If the file would be created by the daemon itself it should be icinga_tmp_t according to the rule in the policy files_tmp_filetrans(icinga2_t, icinga2_tmp_t, { dir file }) (meaning if a process with type icinga2_t creates a file or directory in a directory with tmp_t the file's type should be icinga2_tmp_t)

I tried to recreate the problem on a vagrant box and there I get allow init_t icinga2_tmp_t:file write; which I can explain to myself. This can I solve by changing the context of safe_reload and give icinga2 some more permissions.

Hint for myself:
icinga2.te

allow icinga2_t icinga2_exec_t:file execute_no_trans;
allow icinga2_t self:capability kill;

icinga.fc

/usr/lib/icinga2/safe-reload        --  gen_context(system_u:object_r:icinga2_exec_t,s0)

Test also on el7.

@Duffkess: If you are familiar enough with SELinux you can also try with this changes on your own and give me feedback. If not I will test myself in the near future and then prepare a pull request.

[Duffkess]

Hi @dgoetz
I'm not quite familiar with SELinux yet so I think I'm not able to test this properly right now. I will try to dig deeper into SELinux the next view weeks, I will post if I find something interessteing. Thanks for you feedback so far!

[dgoetz]

I tested it also on EL7 and my suggested fix works fine, so I created a pull request for it.
Can one of the devs merge it and put it on the list for 2.11.3 and 2.12? Thanks!

[mmartinello]

I'm having this problem on CentOS 8 even if using version 2.12.1:

TEST root@proxy3 ~# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.12.1-1)

Icinga does not reload due to permission denied into /tmp:

TEST root@proxy3 ~# systemctl reload icinga2
Job for icinga2.service failed.
See "systemctl status icinga2.service" and "journalctl -xe" for details.

TEST root@proxy3 ~# systemctl status icinga2.service
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2020-11-23 15:13:51 CET; 1min 16s ago
  Process: 1454 ExecReload=/usr/lib/icinga2/safe-reload /etc/sysconfig/icinga2 (code=exited, status=1/FAILURE)
  Process: 914 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/sysconfig/icinga2 (code=exited, status=0/SUCCESS)
 Main PID: 945 (icinga2)
    Tasks: 17 (limit: 23958)
   Memory: 29.5M
   CGroup: /system.slice/icinga2.service
           ├─ 945 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
           ├─1026 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
           └─1031 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log

[...]
Nov 23 15:14:17 proxy3 safe-reload[1454]: Validating config files: /usr/lib/icinga2/safe-reload: line 27: /tmp/tmp.OoeRPndaQS: Permission denied
Nov 23 15:14:17 proxy3 safe-reload[1454]: Failed
Nov 23 15:14:17 proxy3 systemd[1]: icinga2.service: Control process exited, code=exited status=1
Nov 23 15:14:17 proxy3 systemd[1]: Reload failed for Icinga host/service/network monitoring system.

If I disable SELinux it reloads:

TEST root@proxy3 ~# setenforce 0
TEST root@proxy3 ~# systemctl reload icinga2
TEST root@proxy3 ~#