Icinga2: Reminder notifications get send out even with interval=0 when forcing CUSTOM notification via webinterface

[Mikesch-mp]

Reminder notifications get send out even with interval=0 when forcing notification via webinterface

Expected Behavior

When sending a notification via webinterface (with or without force) the notification is send with type custom only once.

Current Behavior

When sending a notification via webinterface (with or without force) the notification is send with type custom and a reminder is send with type problem etc.

Possible Solution

Do not send reminders with notification interval set to 0.

Steps to Reproduce (for bugs)

1. Create notification rule with interval=0 for a service
2. Send a custom forced or not forced notification

Context

Some notifications going to a ticket system and we dont want to create Tickets when users are sending custom notifications via the web interface, so we filter out CUSTOM types, but the reminder that is send after the custom notification is not from type custom :(

Your Environment

• Version used (icinga2 --version): 2.7.2
• Operating System and version: SLES12SP2
• Icinga Web 2 version and modules (System - About): 2.5.0

Log output

[2018-05-24 11:23:46 +0200] information/Notification: Sending 'Custom' notification 'Host007!agent-health!TICKET-SERVICES' for user 'ticketsystem'
[2018-05-24 11:23:46 +0200] information/Notification: Completed sending 'Custom' notification 'Host007!agent-health!TICKET-SERVICES' for checkable 'Host007!agent-health' and user 'ticketsystem'.
[2018-05-24 11:23:48 +0200] information/Notification: Sending reminder 'Problem' notification 'Host007!agent-health!TICKET-SERVICES' for user 'ticketsystem'
[2018-05-24 11:23:48 +0200] information/Notification: Completed sending 'Problem' notification 'Host007!agent-health!TICKET-SERVICES' for checkable 'Host007!SN-agent-health' and user 'ticketsystem'.

[dnsmichi]

A custom notification is a different type than actually a problem notification. This type of notification does not influence interval=0 or anything related to the state change logic.

This reminder notification must source from a different configuration, e.g. a notification object which has times.begin set to anything other than 0 and interval=0. Without the configuration details it is hard to tell though.

Oh, and 2.7.2 isn't supported anymore. Please retry with 2.8.4.

[dnsmichi]

Please re-test this with 2.9.0.

[Mikesch-mp]

Next week i will do it in test environment.

[dnsmichi]

Any updates here?

[Mikesch-mp]

As the TLS fix will be in 2.10.0 i have to wait for that before i can update and test again.
This will not happen before 2019 :(
But i will test it next week in my testing environment at home

[dnsmichi]

I don't mind, I can also stall the issue and remove it from our feedback cycle.

[Mikesch-mp]

I will test this next week, i have some spare time :) (if i dont play ps4)

[Mikesch-mp]

Just tested with 2.10.0 and it behaves the same :(
From the log

[2018-10-12 13:54:29 +0200] information/Notification: Sending 'Custom' notification 'pve.fritz.box!updates!mail-service' for user 'ckoebke'
[2018-10-12 13:54:29 +0200] information/Notification: Completed sending 'Custom' notification 'pve.fritz.box!updates!mail-service' for checkable 'pve.fritz.box!updates' and user 'ckoebke'.
[2018-10-12 13:54:29 +0200] information/HttpServerConnection: HTTP client disconnected (from [::1]:53470)
[2018-10-12 13:54:30 +0200] information/Notification: Sending reminder 'Problem' notification 'pve.fritz.box!updates!mail-service' for user 'ckoebke'
[2018-10-12 13:54:30 +0200] information/Notification: Completed sending 'Problem' notification 'pve.fritz.box!updates!mail-service' for checkable 'pve.fritz.box!updates' and user 'ckoebke'.
````

I would expect to see only a "Completed sending" for the **Custom** notification, and not a **reminder Problem** notification too.

The notification configuration is:

[root@koelin01 ~]# icinga2 object list --type notification --name 'pve.fritz.box!updates!mail-service'
Object 'pve.fritz.box!updates!mail-service' of type 'Notification':
% declared in '/etc/icinga2/zones.d/Master/notification/hosts.conf', lines 30:1-30:44

• __name = "pve.fritz.box!updates!mail-service"
• command = "mail-service-notification"
  % = modified in '/etc/icinga2/zones.d/global-config/templates/notifications.conf', lines 29:3-29:39
• command_endpoint = ""
• host_name = "pve.fritz.box"
  % = modified in '/etc/icinga2/zones.d/Master/notification/hosts.conf', lines 30:1-30:44
• interval = 0
  % = modified in '/etc/icinga2/zones.d/Master/notification/hosts.conf', lines 35:3-35:14
• name = "mail-service"
• package = "_etc"
  % = modified in '/etc/icinga2/zones.d/Master/notification/hosts.conf', lines 30:1-30:44
• period = "24x7"
  % = modified in '/etc/icinga2/zones.d/global-config/templates/notifications.conf', lines 42:3-42:17
• service_name = "updates"
  % = modified in '/etc/icinga2/zones.d/Master/notification/hosts.conf', lines 30:1-30:44
• source_location
  
  
  
  • first_column = 1
  
  
  • first_line = 30
  
  
  • last_column = 44
  
  
  • last_line = 30
  
  
  • path = "/etc/icinga2/zones.d/Master/notification/hosts.conf"
  
  
• states = [ "OK", "Warning", "Critical", "Unknown" ]
  % = modified in '/etc/icinga2/zones.d/global-config/templates/notifications.conf', lines 31:3-31:45
• templates = [ "mail-service", "mail-service-notification" ]
  % = modified in '/etc/icinga2/zones.d/Master/notification/hosts.conf', lines 30:1-30:44
  % = modified in '/etc/icinga2/zones.d/global-config/templates/notifications.conf', lines 28:1-28:49
• times = null
• type = "Notification"
• types = [ "Problem", "Acknowledgement", "Recovery", "Custom", "FlappingStart", "FlappingEnd", "DowntimeStart", "DowntimeEnd", "DowntimeRemoved" ]
  % = modified in '/etc/icinga2/zones.d/global-config/templates/notifications.conf', lines 32:3-34:57
• user_groups = [ "icingaadmins" ]
  % = modified in '/etc/icinga2/zones.d/Master/notification/hosts.conf', lines 33:3-33:34
• users = null
• vars
  % = modified in '/etc/icinga2/zones.d/global-config/templates/notifications.conf', lines 36:3-40:3
  
  
  
  • notification_from = "Icinga 2 Service Monitoring [e-mail]"
  
  
  • notification_icingaweb2url = "https://www.koebbes.de/icingaweb2"
  
  
  • notification_logtosyslog = false
  
  
• zone = "Master"
  % = modified in '/etc/icinga2/zones.d/Master/notification/hosts.conf', lines 30:1-30:44
  [root@koelin01 ~]#
  ```

And it downt matter if interval = 0' or atimes.begin` is set.

[mig5]

I reproduce this on Icinga 2.11.1. I have a host that is already in a HARD state and for which has sent out a notification already (with interval set to 0 in the Notification object).

I want to send a Custom notification. After I do, I get the Custom notification, but then 45 seconds later I also get another PROBLEM notification which appears as a 'reminder' in the Icinga log.

I should not get the reminder if a notification already went out earlier when it transitioned to HARD state, when the Notification's interval is set to 0.

[dnsmichi]

Similar thing noted here: https://github.com/Icinga/icinga2/issues/7758#issuecomment-582983166 Apart from that, the whole interval=0 and times.begin logic needs an overhaul, there's too many possibilities and bugs in this behavior.

[Al2Klimov]

Hello @Mikesch-mp!

After these one is done:

• [ ] #7818

... we'll let you test the respective version. (Likely v2.13)

Best,
AK