Icinga2: Hostname change not recognized

[strixBE]

icinga2 api setup does not respect hostname --fqdn

Expected Behavior

When I change the hostname of my system (e.g. by using hostnamectl set-hostname myhost.mydomain.net) i expect the api setup to regenerate keys. As documented in constants.conf, it should use the output of hostname --fqdn when generating the NodeName.

Current Behavior

As is known by issue #5353 there is a problem with autogenerated hostnames being too long on Azure VMs. By changing the hostname, I hoped to be able to circumvent this behaviour. As mentioned in #5763, the certs are based on NodeName which itself should simply be the output of hostname --fqdn. But performing icinga2 api setup still takes the "original" (Azure-given) hostname from somewhere:

[root@monhost01 ~]# hostname --fqdn
monhost01.mydomain.org
[root@monhost01 ~]#  hostnamectl 
   Static hostname: monhost01.mydomain.org
         Icon name: computer-vm
           Chassis: vm
        Machine ID: <redacted>
           Boot ID: <redacted>
    Virtualization: microsoft
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-862.el7.x86_64
      Architecture: x86-64
[root@monhost01 ~]# icinga2 api setup
information/cli: Generating new CA.
critical/cli: CA files '/var/lib/icinga2/ca//ca.crt' and '/var/lib/icinga2/ca//ca.key' already exist.
warning/cli: Found CA, skipping and using the existing one.
information/cli: Generating new CSR in '/var/lib/icinga2/certs//my-hostname-which-is-long.gozujlgpqoigetrkoveznwndndaras.ax.internal.cloudapp.net.csr'.
information/base: Writing private key to '/var/lib/icinga2/certs//my-hostname-which-is-long.gozujlgpqoigetrkoveznwndndaras.ax.internal.cloudapp.net.key'.
information/base: Writing certificate signing request to '/var/lib/icinga2/certs//my-hostname-which-is-long.gozujlgpqoigetrkoveznwndndaras.ax.internal.cloudapp.net.csr'.
information/cli: Signing CSR with CA and writing certificate to '/var/lib/icinga2/certs//my-hostname-which-is-long.gozujlgpqoigetrkoveznwndndaras.ax.internal.cloudapp.net.crt'.
critical/SSL: Error with x509 NAME getting text by NID: 218603671, "error:0D07A097:asn1 encoding routines:ASN1_mbstring_ncopy:string too long"
critical/Application: Error: std::exception

Steps to Reproduce (for bugs)

1. Set up Azure VM
2. Change Hostname, make DNS changes in portal and on your DNS server
3. Try icinga2 api setup

Your Environment

• Version used (icinga2 --version):

icinga2 - The Icinga 2 network monitoring daemon (version: r2.9.2-1)

Copyright (c) 2012-2018 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-862.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: unknown

• Operating System and version:

[root@monhost01 ~]# cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core) 

• Enabled features (icinga2 feature list):

Disabled features: compatlog elasticsearch gelf graphite influxdb opentsdb perfdata statusdata syslog
Enabled features: api checker command debuglog ido-mysql livestatus mainlog notification

• Icinga Web 2 version and modules (System - About): None
• Config validation (icinga2 daemon -C):

[root@monhost01 ~]# icinga2 daemon -C
[2018-10-02 15:58:31 +0200] information/cli: Icinga application loader (version: r2.9.2-1)
[2018-10-02 15:58:31 +0200] information/cli: Loading configuration file(s).
[2018-10-02 15:58:31 +0200] information/ConfigItem: Committing config item(s).
[2018-10-02 15:58:31 +0200] critical/SSL: Error on bio X509 AUX reading pem file '/var/lib/icinga2/certs//my-hostname-which-is-long.gozujlgpqoigetrkoveznwndndaras.ax.internal.cloudapp.net.crt': 33558530, "error:02001002:lib(2):func(1):reason(2)"
[2018-10-02 15:58:31 +0200] critical/config: Error: Cannot get certificate from cert path: '/var/lib/icinga2/certs//my-hostname-which-is-long.gozujlgpqoigetrkoveznwndndaras.ax.internal.cloudapp.net.crt'.
Location: in /etc/icinga2/features-enabled/api.conf: 5:1-5:24
/etc/icinga2/features-enabled/api.conf(3):  */
/etc/icinga2/features-enabled/api.conf(4): 
/etc/icinga2/features-enabled/api.conf(5): object ApiListener "api" {
                                           ^^^^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/features-enabled/api.conf(6):   accept_config = true
/etc/icinga2/features-enabled/api.conf(7):   accept_commands = true

[2018-10-02 15:58:31 +0200] critical/config: 1 error

• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.
  It's a fresh setup:

/*
 * Endpoint and Zone configuration for a cluster setup
 * This local example requires `NodeName` defined in
 * constants.conf.
 */

object Endpoint NodeName {
  host = NodeName
}

object Zone ZoneName {
  endpoints = [ NodeName ]
}

[Crunsher]

The NodeName is set when Icinga2 is installed, the easiest would be to change your NodeName in the constants.conf and re-run the setup. If you are using automation, you'll have to change the hostname before installing icinga :(

[strixBE]

I tried changing the value in /etc/icinga2/constants.conf. This does not change anything, it is being overwritten by api setup:

[root@monhost01 ~]# grep 'NodeName' /etc/icinga2/constants.conf
const NodeName = "blubbblubb"
[root@monhost01 ~]# icinga2 api setup
information/cli: Generating new CA.
critical/cli: CA files '/var/lib/icinga2/ca//ca.crt' and '/var/lib/icinga2/ca//ca.key' already exist.
warning/cli: Found CA, skipping and using the existing one.
information/cli: Private key file '/var/lib/icinga2/certs//monhost01.mydomain.org.key' already exists, not generating new certificate.
information/cli: API user config file '/etc/icinga2/conf.d/api-users.conf' already exists, not creating config file.
information/cli: Enabling the 'api' feature.
warning/cli: Feature 'api' already enabled.
information/cli: Updating 'NodeName' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
information/cli: Updating 'ZoneName' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
Done.

Now restart your Icinga 2 daemon to finish the installation!

I finally did, what you already mentioned in your comment: I killed the VM and redid the setup with setting the hostname before installing icinga2. But shouldn't I be able to change the hostname without redeploying?

[Crunsher]

I think I found the issue: GetVariable("NodeName") does not return what it should, ie something different from what's set in the constants.conf.

[dnsmichi]

constants.conf needs icinga2 daemon -C as validation and as such updating the variables cache which is used again in icinga2 api setup. @MrStrix try that please.

[strixBE]

i tried, it worked :smirk:

Steps to reproduce:

1. Install icinga2 using yum install icinga2. At this time, my hostname is still the generated one from Azure:

[root@vm-icinga2test-01 ~]# hostnamectl 
   Static hostname: vm-icinga2test-01
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 97da09219a2d42489c8b8f748e6d2fb7
           Boot ID: cd88789023514e32a18a6b843068a1d2
    Virtualization: microsoft
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-862.11.6.el7.x86_64
      Architecture: x86-64
[root@vm-icinga2test-01 ~]# hostname --fqdn
vm-icinga2test-01.1fejhddejz1ulibi4ibt1rvwid.ax.internal.cloudapp.net

constants.conf looks like this:

[root@vm-icinga2test-01 ~]# grep Name /etc/icinga2/constants.conf 
//const NodeName = "localhost"
const ZoneName = NodeName

When I try to run icinga2 api setup at this stage, it will fail because the hostname is too long for x509.

1. Change var in constants.conf and change hostname

[root@vm-icinga2test-01 ~]# hostnamectl set-hostname vm-icinga2test-01.example.com
[root@vm-icinga2test-01 ~]# hostname --fqdn  
vm-icinga2test-01.example.com
[root@vm-icinga2test-01 ~]# grep Name /etc/icinga2/constants.conf
const NodeName = "vm-icinga2test-01.example.com"
const ZoneName = NodeName

1. Perform icinga2 api setup which fails

[root@vm-icinga2test-01 ~]# icinga2 api setup
information/cli: Generating new CA.
information/base: Writing private key to '/var/lib/icinga2/ca//ca.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/ca//ca.crt'.
information/cli: Generating new CSR in '/var/lib/icinga2/certs//vm-icinga2test-01.1fejhddejz1ulibi4ibt1rvwid.ax.internal.cloudapp.net.csr'.
information/base: Writing private key to '/var/lib/icinga2/certs//vm-icinga2test-01.1fejhddejz1ulibi4ibt1rvwid.ax.internal.cloudapp.net.key'.
information/base: Writing certificate signing request to '/var/lib/icinga2/certs//vm-icinga2test-01.1fejhddejz1ulibi4ibt1rvwid.ax.internal.cloudapp.net.csr'.
information/cli: Signing CSR with CA and writing certificate to '/var/lib/icinga2/certs//vm-icinga2test-01.1fejhddejz1ulibi4ibt1rvwid.ax.internal.cloudapp.net.crt'.
critical/SSL: Error with x509 NAME getting text by NID: 218603671, "error:0D07A097:asn1 encoding routines:ASN1_mbstring_ncopy:string too long"
critical/Application: Error: std::exception


Additional information is available in '/var/log/icinga2/crash/report.1550475507.211740'

Aborted

1. Run icinga2 daemon -C:

[root@vm-icinga2test-01 ~]# icinga2 daemon -C
[2019-02-18 07:38:42 +0000] information/cli: Icinga application loader (version: r2.10.2-1)
[2019-02-18 07:38:42 +0000] information/cli: Loading configuration file(s).
[2019-02-18 07:38:42 +0000] information/ConfigItem: Committing config item(s).
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 11 Services.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 1 Host.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 12 Notifications.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 3 Zones.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 1 Endpoint.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 1 User.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 215 CheckCommands.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2019-02-18 07:38:42 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2019-02-18 07:38:42 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2019-02-18 07:38:42 +0000] information/cli: Finished validating the configuration file(s).

1. Re-run the setup

[root@vm-icinga2test-01 ~]# icinga2 api setup
information/cli: Generating new CA.
critical/cli: CA files '/var/lib/icinga2/ca//ca.crt' and '/var/lib/icinga2/ca//ca.key' already exist.
warning/cli: Found CA, skipping and using the existing one.
information/cli: Generating new CSR in '/var/lib/icinga2/certs//vm-icinga2test-01.example.com.csr'.
information/base: Writing private key to '/var/lib/icinga2/certs//vm-icinga2test-01.example.com.key'.
information/base: Writing certificate signing request to '/var/lib/icinga2/certs//vm-icinga2test-01.example.com.csr'.
information/cli: Signing CSR with CA and writing certificate to '/var/lib/icinga2/certs//vm-icinga2test-01.example.com.crt'.
information/pki: Writing certificate to file '/var/lib/icinga2/certs//vm-icinga2test-01.example.com.crt'.
information/cli: Copying CA certificate to '/var/lib/icinga2/certs//ca.crt'.
information/cli: Adding new ApiUser 'root' in '/etc/icinga2/conf.d/api-users.conf'.
information/cli: Enabling the 'api' feature.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Updating 'NodeName' constant in '/etc/icinga2/constants.conf'.
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
information/cli: Updating 'ZoneName' constant in '/etc/icinga2/constants.conf'.
information/cli: Backup file '/etc/icinga2/constants.conf.orig' already exists. Skipping backup.
Done.

Now restart your Icinga 2 daemon to finish the installation!

It seems to work this way. Also constants.conf is rewritten with correct vars set:

[root@vm-icinga2test-01 ~]# grep Name /etc/icinga2/constants.conf
const NodeName = "vm-icinga2test-01.example.com"
const ZoneName = "vm-icinga2test-01.example.com"

Works for me

Tl;DR:
Run icinga2 daemon -C before icinga2 api setup after hostname change

[dnsmichi]

Thanks for the steps, I'll assign this to myself for updating the docs and/or add a CLI parameter for optional cn overrides similar to other CLI commands.

[dnsmichi]

I'll implement this myself, I need a break from Boost ASIO.

[dnsmichi]

Some tests.

$ icinga2 api setup -DDataDir=/tmp/a/var -DConfigDir=/tmp/a/etc --cn bumsti
Closed FD 3 which we inherited from our parent process.
Closed FD 4 which we inherited from our parent process.
Closed FD 5 which we inherited from our parent process.
information/cli: Generating new CA.
information/base: Writing private key to '/tmp/a/var/ca//ca.key'.
information/base: Writing X509 certificate to '/tmp/a/var/ca//ca.crt'.
information/cli: Generating new CSR in '/tmp/a/var/certs//bumsti.csr'.
information/base: Writing private key to '/tmp/a/var/certs//bumsti.key'.
information/base: Writing certificate signing request to '/tmp/a/var/certs//bumsti.csr'.
information/cli: Signing CSR with CA and writing certificate to '/tmp/a/var/certs//bumsti.crt'.
information/pki: Writing certificate to file '/tmp/a/var/certs//bumsti.crt'.
information/cli: Copying CA certificate to '/tmp/a/var/certs//ca.crt'.
warning/cli: Path '/tmp/a/etc/conf.d' do not exist.
information/cli: Creating path '/tmp/a/etc/conf.d'.
information/cli: Adding new ApiUser 'root' in '/tmp/a/etc/conf.d/api-users.conf'.
information/cli: Reading '/tmp/a/etc/icinga2.conf'.
information/cli: Updating '"conf.d/api-users.conf"' include in '/tmp/a/etc/icinga2.conf'.
information/cli: Enabling the 'api' feature.
critical/cli: Cannot parse available features. Path '/tmp/a/etc/features-available' does not exist.
information/cli: Updating 'NodeName' constant in '/tmp/a/etc/constants.conf'.
information/cli: Updating 'ZoneName' constant in '/tmp/a/etc/constants.conf'.
information/cli: Created backup file '/tmp/a/etc/constants.conf.orig'.
Done.

Now restart your Icinga 2 daemon to finish the installation!

[dnsmichi]

michi@mbpmif ~/dev/icinga/icinga2 (feature/api-setup-cn) $ icinga2 api setup -DDataDir=/tmp/b/var -DConfigDir=/tmp/b/etc
Closed FD 3 which we inherited from our parent process.
Closed FD 4 which we inherited from our parent process.
Closed FD 5 which we inherited from our parent process.
information/cli: Generating new CA.
information/base: Writing private key to '/tmp/b/var/ca//ca.key'.
information/base: Writing X509 certificate to '/tmp/b/var/ca//ca.crt'.
information/cli: Generating new CSR in '/tmp/b/var/certs//mbpmif.int.netways.de.csr'.
information/base: Writing private key to '/tmp/b/var/certs//mbpmif.int.netways.de.key'.
information/base: Writing certificate signing request to '/tmp/b/var/certs//mbpmif.int.netways.de.csr'.
information/cli: Signing CSR with CA and writing certificate to '/tmp/b/var/certs//mbpmif.int.netways.de.crt'.
information/pki: Writing certificate to file '/tmp/b/var/certs//mbpmif.int.netways.de.crt'.
information/cli: Copying CA certificate to '/tmp/b/var/certs//ca.crt'.
warning/cli: Path '/tmp/b/etc/conf.d' do not exist.
information/cli: Creating path '/tmp/b/etc/conf.d'.
information/cli: Adding new ApiUser 'root' in '/tmp/b/etc/conf.d/api-users.conf'.
information/cli: Reading '/tmp/b/etc/icinga2.conf'.
information/cli: Updating '"conf.d/api-users.conf"' include in '/tmp/b/etc/icinga2.conf'.
information/cli: Enabling the 'api' feature.
critical/cli: Cannot parse available features. Path '/tmp/b/etc/features-available' does not exist.
information/cli: Updating 'NodeName' constant in '/tmp/b/etc/constants.conf'.
information/cli: Updating 'ZoneName' constant in '/tmp/b/etc/constants.conf'.
information/cli: Created backup file '/tmp/b/etc/constants.conf.orig'.
Done.

Now restart your Icinga 2 daemon to finish the installation!

[dnsmichi]

10 minutes for the code and some tests.

[lrconsult]

for me it helped to do:
rm /var/lib/icinga2/ca/*
rm /var/lib/icinga2/certs/*
rm /var/cache/icinga2/*
icinga2 api setup

[paulb-opusvl]

This was a really helpful find, it lead me to the resolution after I noticed I had typo'ed my domain name.

The important part was using daemon -C before hand, however it failed because the api feature was enabled and the config would not succeed. I had to disable the feature and then things worked as expected and I got a correct name.

icinga2 daemon -C

[2020-06-03 17:59:23 +0100] critical/SSL: Error on bio X509 AUX reading pem file '/var/lib/icinga2/certs//icinga-test.domain.co.uk.crt': 33558530, "error:02001002:system library:fopen:No such file or directory"
[2020-06-03 17:59:23 +0100] critical/config: Error: Cannot get certificate from cert path: '/var/lib/icinga2/certs//icinga-test.domain.co.uk.crt'.
Location: in /etc/icinga2/features-enabled/api.conf: 5:1-5:24
/etc/icinga2/features-enabled/api.conf(3):  */
/etc/icinga2/features-enabled/api.conf(4): 
/etc/icinga2/features-enabled/api.conf(5): object ApiListener "api" {
                                           ^^^^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/features-enabled/api.conf(6):   //accept_config = false
/etc/icinga2/features-enabled/api.conf(7):   //accept_commands = false

vi /etc/icinga2/constants.conf
icinga2 feature disable api
icinga2 daemon -C
icinga2 api setup

[apagr]

confirm paulb-opusvl version. In my case i enabled api, realized hostname was wrong, changed it, and hit the same place - no matter what, old hostname beeing remembered and put in constants.conf.
Had to disable api, remove certs, run icinga2 daemon -C and setup again.

It is fresh icinga2 on Buster.