Icinga2: WORKAROUND - mail-service-notifications.sh script failes to send notification - errors out with "Requirement parameters are missing."

Created on 8 Aug 2017 · 7Comments · Source: Icinga/icinga2

When a apply Service... block is applied to a host with assign where..., when the service goes down the front end indicates a message has been sent but nothing gets to my Inbox. However, while still waiting for my email, another host went down and I got that email. This prompted me to believe that the issue was with the mail-service-notification script.

When I ran the command icinga2 object list --name 'mail-service-notification', I got the following output:

Object 'mail-service-notification' of type 'NotificationCommand':
  % declared in '/etc/icinga2/conf.d/commands.conf', lines 21:1-21:54
  * __name = "mail-service-notification"
  * arguments = null
  * command = [ "/etc/icinga2/scripts/mail-service-notification.sh" ]
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 22:3-22:76
  * env
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 24:3-37:3
    * HOSTADDRESS = "$address$"
    * HOSTALIAS = "$host.display_name$"
    * HOSTDISPLAYNAME = "$host.display_name$"
    * LONGDATETIME = "$icinga.long_date_time$"
    * NOTIFICATIONAUTHORNAME = "$notification.author$"
    * NOTIFICATIONCOMMENT = "$notification.comment$"
    * NOTIFICATIONTYPE = "$notification.type$"
    * SERVICEDESC = "$service.name$"
    * SERVICEDISPLAYNAME = "$service.display_name$"
    * SERVICEOUTPUT = "$service.output$"
    * SERVICESTATE = "$service.state$"
    * USEREMAIL = "$user.email$"
  * execute
    % = modified in 'methods-itl.conf', lines 48:3-48:40
    * arguments = [ "notification", "user", "cr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginNotification"
    * side_effect_free = false
    * type = "Function"
  * name = "mail-service-notification"
  * package = "_etc"
  * source_location
    * first_column = 1
    * first_line = 21
    * last_column = 54
    * last_line = 21
    * path = "/etc/icinga2/conf.d/commands.conf"
  * templates = [ "mail-service-notification", "plugin-notification-command" ]
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 21:1-21:54
    % = modified in 'methods-itl.conf', lines 47:2-47:83
  * timeout = 60
  * type = "NotificationCommand"
  * vars = null
  * zone = ""

I noticed there was no SERVICENAME = "$service.name$" defined in the list above but yet it existed in the mail-service-notification.shscript. As such, I changed all occurences of SERVICENAME to SERVICEDESC in the mail-service-notification.sh script and I finally received the notification email!

The logs had this (replaced my servername with and my affected service name with :

[2017-08-08 07:41:48 +1000] information/Checkable: Checking for configured notifications for object '<FQDN>!<Service>'
[2017-08-08 07:41:48 +1000] information/Notification: Sending 'Problem' notification '<FQDN>!<Service>!mail-icingaadmin' for user 'icingaadmin'
[2017-08-08 07:41:48 +1000] information/Notification: Completed sending 'Problem' notification '<FQDN>!<Service>!mail-icingaadmin' for checkable 'FQDN!<Service>' and user 'icingaadmin'.
[2017-08-08 07:41:48 +1000] warning/PluginNotificationTask: Notification command for object '<FQDN>!<Service>' (PID: 8076, arguments: '/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 1, output: Requirement parameters are missing.

Required parameters:
  -4 HOSTADDRESS ($address$)
  -d LONGDATETIME ($icinga.long_date_time$)
  -e SERVICENAME ($service.name$)
  -l HOSTNAME ($host.name$)
  -n HOSTDISPLAYNAME ($host.display_name$)
  -o SERVICEOUTPUT ($service.output$)
  -r USEREMAIL ($user.email$)
  -s SERVICESTATE ($service.state$)
  -t NOTIFICATIONTYPE ($notification.type$)
  -u SERVICEDISPLAYNAME ($service.display_name$)

Optional parameters:
  -6 HOSTADDRESS6 ($address6$)
  -b NOTIFICATIONAUTHORNAME ($notification.author$)
  -c NOTIFICATIONCOMMENT ($notification.comment$)
  -i ICINGAWEB2URL ($notification_icingaweb2url$, Default: unset)
  -f MAILFROM ($notification_mailfrom$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
  -v ($notification_sendtosyslog$, Default: false)

Expected Behavior

I expected to receive an email in my Inbox.

Current Behavior

No email was received in my Inbox relating to the service now in a CRITICAL state.

Possible Solution

Change all occurences of SERVICENAME to SERVICEDESC in the /etc/icinga2/scripts/mail-servicenotification.sh script.

Steps to Reproduce (for bugs)

Setup a host configuration

object Host "server.domain.com" {
  import "generic-host"
  address = "server.domain.com"
  vars.cipherise = true
  vars.os = "Linux"
  vars.wload = "5,3,3"
  vars.cload = "10,6,6"
}

2. Setup a service
apply Service "Testing" {
  import "generic-service"
  check_command = "cipherise-access-point"
  vars.service.name = "My Test Service"
  vars.apiurl = "https://logserver.domain.com:9880"
  vars.accesspoint = "Some access point"
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }

  assign where match("server.domain.com", host.name)
}

Force service failure by blocking access to the APIURL port 9880
I used iptables to block access to port 9880 from server.domain.com
Check email after Notification is purportedly sent out
No email received in Inbox.

Context

Your Environment

Version used (icinga2 --version):

icinga2 - The Icinga 2 network monitoring daemon (version: r2.7.0-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 16.04.3 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 4.4.0-77-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 5.3.1
  Build host: 6f646e1a58f5

Operating System and version: Description (lsb_release -a):

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

Enabled features (icinga2 feature list):

Disabled features: compatlog debuglog gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

Icinga Web 2 version and modules (System - About):

2.4.1
Copyright
© 2013-2017 The Icinga Project

Config validation (icinga2 daemon -C):

information/cli: Icinga application loader (version: r2.7.0-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: <COMPANY NAME>-Monitor
warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 30:1-30:41) for type 'Dependency' does not match anywhere!
information/ConfigItem: Instantiated 1 ApiUser.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 3 Zones.
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 1 Endpoint.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ConfigItem: Instantiated 2 NotificationCommands.
information/ConfigItem: Instantiated 100 Notifications.
information/ConfigItem: Instantiated 214 CheckCommands.
information/ConfigItem: Instantiated 1 Downtime.
information/ConfigItem: Instantiated 9 Hosts.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 2 HostGroups.
information/ConfigItem: Instantiated 3 Comments.
information/ConfigItem: Instantiated 1 UserGroup.
information/ConfigItem: Instantiated 3 TimePeriods.
information/ConfigItem: Instantiated 1 User.
information/ConfigItem: Instantiated 91 Services.
information/ConfigItem: Instantiated 3 ServiceGroups.
information/ConfigItem: Instantiated 1 ScheduledDowntime.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes.
N/A

areconfiguration arenotifications no-issue

Source

ak2766

Most helpful comment

Update your notification commands too when updating the notification scripts. These changes are required when upgrading to v2.7.0.

https://github.com/Icinga/icinga2/blob/master/etc/icinga2/conf.d/commands.conf

Quote from https://www.icinga.com/2017/08/02/icinga-2-v2-7-0-released/

Make sure to checkout the documentation for the updated example configuration.

https://www.icinga.com/docs/icinga2/latest/doc/03-monitoring-basics/#notification-commands

dnsmichi on 8 Aug 2017

👍2

All 7 comments

For completeness sake, adding the following lines:

for u in HOSTADDRESS LONGDATETIME HOSTNAME HOSTDISPLAYNAME SERVICEDESC SERVICEDISPLAYNAME SERVICEOUTPUT SERVICESTATE USEREMAIL NOTIFICATIONTYPE; do
  echo "${u} : ${!u}"
done

just before:

if [ ! "$HOSTADDRESS" ] || [ ! "$LONGDATETIME" ] \
|| [ ! "$HOSTNAME" ] || [ ! "$HOSTDISPLAYNAME" ] \
|| [ ! "$SERVICEDESC" ] || [ ! "$SERVICEDISPLAYNAME" ] \
|| [ ! "$SERVICEOUTPUT" ] || [ ! "$SERVICESTATE" ] \
|| [ ! "$USEREMAIL" ] || [ ! "$NOTIFICATIONTYPE" ]; then
  Error "Requirement parameters are missing."
fi

helped a lot in my troubleshooting steps for this issue. Had it not been a BASH script, it would definitely have taken a while longer to identify the root cause.

The output (found in the log file /var/log/icinga2/icinga2.log) looked like:

[2017-08-08 09:59:23 +1000] information/Notification: Completed sending 'Problem' notification '<FQDN>!<Service>!mail-icingaadmin' for checkable '<FQDN>!<Service>' and user 'icingaadmin'.
[2017-08-08 09:59:23 +1000] warning/PluginNotificationTask: Notification command for object '<FQDN>!<Service>' (PID: 20784, arguments: '/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 1, output: HOSTADDRESS : server.domain.com
LONGDATETIME : 2017-08-08 09:59:23 +1000
HOSTNAME : icinga2
HOSTDISPLAYNAME : server.domain.com
SERVICENAME : 
SERVICEDISPLAYNAME : <Service>
SERVICEOUTPUT : !! Device <device> has been disconnected for too long !!
SERVICESTATE : CRITICAL
USEREMAIL : [email protected]
NOTIFICATIONTYPE : PROBLEM
Requirement parameters are missing.

As can be seen above, the SERVICENAME reported a blank!

ak2766 on 8 Aug 2017

I can confirm this issue and that the workaround/possible solution ist working.

Version used (icinga2 --version)

icinga2 - The Icinga 2 network monitoring daemon (version: r2.7.0-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 16.04.3 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 4.4.0-87-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 5.3.1
  Build host: 6f646e1a58f5

Operating System and version: Description (lsb_release -a)

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:    16.04
Codename:   xenial

enabled features (icinga2 feature list)

Disabled features: compatlog debuglog gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

hnnnglmbrg on 8 Aug 2017

Possible Solution

Change all occurences of SERVICENAME to SERVICEDESC in the /etc/icinga2/scripts/mail-servicenotification.sh script.

Worked for me, too (Ubuntu14.04), thanks a lot @ak2766

ronator on 8 Aug 2017

👍1

Update your notification commands too when updating the notification scripts. These changes are required when upgrading to v2.7.0.

https://github.com/Icinga/icinga2/blob/master/etc/icinga2/conf.d/commands.conf

Quote from https://www.icinga.com/2017/08/02/icinga-2-v2-7-0-released/

Make sure to checkout the documentation for the updated example configuration.

https://www.icinga.com/docs/icinga2/latest/doc/03-monitoring-basics/#notification-commands

dnsmichi on 8 Aug 2017

👍2

Sorry for any possible rough tone, but this does not work for me, either. As I said, I use zones.d/ not conf.d/ so I do not see how that applies to users with cluster/satellite setup. No mails are send again after making the suggested changes.

My /etc/icinga2/conf.d/commands.conf is already exactly like https://github.com/Icinga/icinga2/blob/master/etc/icinga2/conf.d/commands.conf. I was trying to tell you that but we got off the wrong foot.

ronator on 8 Aug 2017

The problem for me was, that the HOSTADDRESS was missing, even after updating my commands.conf.

I'm using the deprecated way of satellite hosts and they simply have no address, the never had. But with this update the need to...

I simply added HOSTADDRESS="127.0.0.1" before

## Check required parameters (TODO: better error message)
## Keep formatting in sync with mail-host-notification.sh
if [ ! "$HOSTADDRESS" ] || [ ! "$LONGDATETIME" ] \
|| [ ! "$HOSTNAME" ] || [ ! "$HOSTDISPLAYNAME" ] \
|| [ ! "$SERVICENAME" ] || [ ! "$SERVICEDISPLAYNAME" ] \
|| [ ! "$SERVICEOUTPUT" ] || [ ! "$SERVICESTATE" ] \
|| [ ! "$USEREMAIL" ] || [ ! "$NOTIFICATIONTYPE" ]; then
  Error "Requirement parameters are missing."
fi

and everything works as before.

fdellwing on 21 Sep 2017

I am using icinga 2.10 , icingaweb2 2.7 and director 1.6.2 module. many values are missing while executing notification.
HOSTDISPLAYNAME :
SERVICENAME :
SERVICEDISPLAYNAME :
SERVICEOUTPUT :
SERVICESTATE :
USEREMAIL :
NOTIFICATIONTYPE :

I have checked using the method given by @ak2766.

Also, icinga2 object list --name 'mail-service-notification' output is

Object 'mail-service-notification' of type 'NotificationCommand':
% declared in '/var/lib/icinga2/api/packages/director/965ac676-e2e7-432a-8583-179ceb9cd3a5/zones.d/icinga2-newmaster.example.com/commands.conf', lines 1:0-1:53

__name = "mail-service-notification"
arguments = null
command = [ "/etc/icinga2/scripts/mail-service-notification.sh" ]
% = modified in '/var/lib/icinga2/api/packages/director/965ac676-e2e7-432a-8583-179ceb9cd3a5/zones.d/icinga2-newmaster.example.com/commands.conf', lines 3:5-3:69
env = null
execute
% = modified in 'methods-itl.conf', lines 40:3-40:30
% = modified in 'methods-itl.conf', lines 40:3-40:30
- arguments = [ "notification", "user", "cr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
- deprecated = false
- name = "Internal#PluginNotification"
- side_effect_free = false
- type = "Function"
name = "mail-service-notification"
package = "director"
source_location
- first_column = 0
- first_line = 1
- last_column = 53
- last_line = 1
- path = "/var/lib/icinga2/api/packages/director/965ac676-e2e7-432a-8583-179ceb9cd3a5/zones.d/icinga2-newmaster.example.com/commands.conf"
templates = [ "mail-service-notification", "plugin-notification-command", "plugin-notification-command" ]
% = modified in '/var/lib/icinga2/api/packages/director/965ac676-e2e7-432a-8583-179ceb9cd3a5/zones.d/icinga2-newmaster.example.com/commands.conf', lines 1:0-1:53
% = modified in 'methods-itl.conf', lines 39:2-39:122
% = modified in 'methods-itl.conf', lines 39:2-39:122
timeout = 60
% = modified in '/var/lib/icinga2/api/packages/director/965ac676-e2e7-432a-8583-179ceb9cd3a5/zones.d/icinga2-newmaster.example.com/commands.conf', lines 4:5-4:16
type = "NotificationCommand"
vars = null
zone = "icinga2-newmaster.example.com"

sunilmbhoi on 12 Sep 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Icinga 2.11.0 RC1/Icingaweb 2.6.3: Icinga Web 2 does not properly detect Icinga 2

peteeckel · 4Comments

[dev.icinga.com #13215] Newly configured already running ScheduledDowntime not put into effect

icinga-migration · 6Comments

Acknowledgement comments with newlines break emails using default notify scripts.

BuildTheRobots · 6Comments

Servicegroup table for livestatus is not refreshed when service is removed

mickenordin · 5Comments

When using a perl plugin within /usr/lib64/nagios/plugins/ directory, selinux is blocking access to it.

adamparker · 5Comments