Icinga2: Client unable to obtain certificate

[andrew-beak-property-logic]

Describe the bug

On fresh agent install on new Ubuntu 18.04 from current release repositories the node wizard fails to complete.

To Reproduce

• Have working master node (mine is known good because it's monitoring other hosts)
• Spin up fresh Ubuntu 18.04 VM (mine is on AWS)
• Add repository and install Icinga2
• Copy ca certificate from master to /var/lib/icinga2/certs
• Restart Icinga2
• Try to run node wizard to add client node
• Get valid ticket from master node
• Enter ticket

Expected behavior

Node wizard is able to obtain a certificate

Screenshots

The graphite node is failing certificate verification. The log line looks like this:

[2019-08-22 11:24:56 +0000] information/ApiListener: New client connection for identity 'graphite.mydomain.com' from [ip4 address]:35318 (certificate validation failed: code 18: self signed certificate)
[2019-08-22 11:39:47 +0000] warning/ApiListener: No data received on new API connection for identity 'graphite.mydomain.com'. Ensure that the remote endpoints are properly configured in a cluster setup.

On the agent I get an error like this:

critical/cli: Could not fetch valid response. Please check the master log.

Your Environment

Include as many relevant details about the environment you experienced the problem in

icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)

Copyright (c) 2012-2019 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
Platform: Ubuntu
Platform version: 18.04.3 LTS (Bionic Beaver)
Kernel: Linux
Kernel version: 4.15.0-1044-aws
Architecture: x86_64

Build information:
Compiler: GNU 7.4.0
Build host: 8b8700ecb474

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker mainlog

Zones.conf on the agent is as deployed with default distribution

Additional context

I’ve installed Icinga2 and am using Ansible to deploy client nodes.

I’m using a single master with no satellites.

I have two agents - “sonarqube” and “graphite”. My sonarqube node is working correctly and is added to my Icinga2 master correctly.

The graphite node is failing certificate verification. The log line looks like this:

[2019-08-22 11:39:47 +0000] warning/ApiListener: No data received on new API connection for identity 'graphite.mydomain.com'. Ensure that the remote endpoints are properly configured in a cluster setup.
On the agent I get an error like this:

critical/cli: Could not fetch valid response. Please check the master log.

The commands that Ansible produces look like this:

icinga2 pki new-cert --cn graphite.mydomain.com --key /etc/icinga2/pki/graphite.mydomain.com.key --cert /etc/icinga2/pki/graphite.mydomain.com.crt
icinga2 pki save-cert --key /etc/icinga2/pki/graphite.mydomain.com.key --cert /etc/icinga2/pki/graphite.mydomain.com.crt --trustedcert /etc/icinga2/pki/trusted-master.crt --host mymasterip4
icinga2 pki request --host mymasterip4 --port 5665 --ticket validtickethash --key /etc/icinga2/pki/graphite.mydomain.com.key --cert /etc/icinga2/pki/graphite.mydomain.com.crt --trustedcert /etc/icinga2/pki/trusted-master.crt --ca /etc/icinga2/pki/ca.key
On the failing (graphite) node there is no CA certificate at /var/lib/icinga2/certs

If I run openssl x509 -in graphite.mydomain.com -text then I can see it is self-signed.

How do I obtain the CA cert and get the agent node to use it when generating the certificate?

It appears that JsonRpcConnection is not running when I try to get the certificate from the server?

Manual workaround

I found a way to work around this manually, but this isn’t going to work for automated deployments.

• Change the master to connect to the agent machine (and stop agent from connecting to master)
• Wait for the ca list to get populated and manually sign the certificate
• Revert the change you made in step 1 and try running the key generation again (it works)

[Al2Klimov]

Hello @andrew-beak-property-logic and thank you for reporting!

Please try v2.11rc1: https://icinga.com/2019/07/25/icinga-2-11-release-candidate/

Best,
AK

[dnsmichi]

When using Ansible, I would recommend to use the automated pki ticket generation logic found in many roles. This absorbs the requirement to use the pending ca list which is sort of asynchronous too.

[andrew-beak-property-logic]

@Al2Klimov Thanks very much, this appears to work for me - I can't reproduce the problem with the new rc version.

[dnsmichi]

Thanks for testing! 👍

[wonder75]

I had the same problem with 2.10. Upgrading to testing fixed it. Problem with this is, that the bugged version is the one being installed on Debian Buster, so this will happen over and over again to people. I'd say the default apt package is broken on Debian and maybe on other distros too. A client connection is not possible with the currently shipped stable package.

[dnsmichi]

What Debian upstream provides, is not in our influence - we can only recommend to use packages.icinga.com and use the releases from there. Once 2.11 reaches its final release soon, it will be available for our user base. If case you want to convince Debian upstream to update their packages, file a bug report at their tracker.

[andrew-beak-property-logic]

Personally I'll just keep an eye on the upstream package and when 2.11 is available I'll just swap my play. Dunno if it helps anybody, but my playbook looks like this:

- name: work around issue 7438 (https://github.com/Icinga/icinga2/issues/7438)
  block:
  - name: Add icinga repository
    shell: >
      echo "deb https://packages.icinga.com/ubuntu icinga-{{ ansible_distribution_release }} main" > /etc/apt/sources.list.d/{{ ansible_distribution_release }}-icinga.list;
      echo "deb-src https://packages.icinga.com/ubuntu icinga-{{ ansible_distribution_release }} main" >> /etc/apt/sources.list.d/{{ ansible_distribution_release }}-icinga.list

  - name: Add icinga testing repository
    shell: >
      echo "deb https://packages.icinga.com/ubuntu icinga-{{ ansible_distribution_release }}-testing main" > /etc/apt/sources.list.d/{{ ansible_distribution_release }}-icinga-testing.list;
      echo "deb-src https://packages.icinga.com/ubuntu icinga-{{ ansible_distribution_release }}-testing main" >>  /etc/apt/sources.list.d/{{ ansible_distribution_release }}-icinga-testing.list;

  - name: run apt update
    apt:
      update_cache: yes

  - name: install icinga from testing repository
    shell: "apt-get install -y -t icinga-{{ ansible_distribution_release }}-testing icinga2"