Icinga2: Modifed attributes is applied in the wrong order at start

[theseal]

Expected Behavior

Changes to variable data used in Apply rules should take effect when restarting Icinga2.
You should be able to create objects and change variables during the objects lifetime and always get a consistent answer when fetching the object. Changes in configuration related to those variables should take effect.

Current Behavior

Its seems like the original configuration is read first and then Apply rules are applied and lastly modified attributes are applied rendering it impossible to modify objects and using Apply rules.

This also causing the livestatus and http api to give outdated answers when Icinga2 is starting up.

Steps to Reproduce (for bugs)

1. Create two notification rules and two users

apply Notification "mail-host-queue1" to Host {
  import "mail-host-notification"

  types = [ Problem ]

  interval = 0
  period = "24x7"
  users = [ "user1" ]
  assign where host.vars.ticket_queue == "queue1"
}
apply Notification "mail-host-queue2" to Host {
  import "mail-host-notification"

  types = [ Problem ]

  interval = 0
  period = "24x7"
  users = [ "user2" ]
  assign where host.vars.ticket_queue == "queue2"
}

object User "user1" {
    display_name = "User1"
    email = "[email protected]"
    states = [ OK, Warning,Critical, Unknown ]
    types = [ Problem ]
}
object User "user2" {
    display_name = "User2"
    email = "[email protected]"
    states = [ OK, Warning,Critical, Unknown ]
    types = [ Problem ]
}

1. Create a host with the API like

{
   "attrs" : {
      "address" : "127.0.0.1",
      "check_command" : "hostalive",
      "check_interval" : 60,
      "check_period" : "24x7",
      "display_name" : "example-prod-srv01.example.com",
      "retry_interval" : 30,
      "vars" : {
         "ticket_queue" : "queue1"
      },
      "zone" : "master"
   }
}

1. Change the ticket_queue variable to queue2 via the API and see this appears in /var/lib/icinga2/modified-attributes.conf

var obj = get_object("Host", "example-prod-srv01.example.com")
if (obj) {
        obj.modify_attribute("address", "127.0.0.1")
        obj.modify_attribute("check_command", "hostalive")
        obj.modify_attribute("check_interval", 60.000000)
        obj.modify_attribute("display_name", "example-prod-srv01.example.com")
        obj.modify_attribute("max_check_attempts", 0.000000)
        obj.modify_attribute("retry_interval", 30.000000)
        obj.modify_attribute("vars", {
ticket_queue = "queue2"
})
        obj.version = 1494316089.004014
}

1. Restart Icinga2.
2. Be fast and fetch the object when Icinga2 is starting. Notice that the original object is delivered.
3. Fetch the object again and see the diffrence. Notice that the modified object is delivered.
4. Notice that the notifications for the host are sent to user1.

Context

Inconsistent data, notifications sent to the wrong address and Apply rules can not be used on variable data.

Your Environment

# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.6.2-1)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 16.04.2 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 4.4.0-72-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 5.4.0
  Build host: lgw01-08

# icinga2 feature list
Disabled features: debuglog gelf influxdb mainlog opentsdb perfdata statusdata
Enabled features: api checker command compatlog graphite ido-mysql livestatus notification syslog

# icinga2 daemon -C
information/cli: Icinga application loader (version: r2.6.2-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: icinga-test-srv01.example.com
information/ConfigItem: Instantiated 2 ApiUsers.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 1 SyslogLogger.
information/ConfigItem: Instantiated 1 Zone.
information/ConfigItem: Instantiated 2 Endpoints.
information/ConfigItem: Instantiated 4 NotificationCommands.
information/ConfigItem: Instantiated 9746 Notifications.
information/ConfigItem: Instantiated 273 CheckCommands.
information/ConfigItem: Instantiated 1124 Hosts.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 43 HostGroups.
information/ConfigItem: Instantiated 7538 Dependencies.
information/ConfigItem: Instantiated 7 UserGroups.
information/ConfigItem: Instantiated 307 Users.
information/ConfigItem: Instantiated 7538 Services.
information/ConfigItem: Instantiated 5 TimePeriods.
information/ConfigItem: Instantiated 45 ServiceGroups.
information/ConfigItem: Instantiated 1 CompatLogger.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 GraphiteWriter.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 LivestatusListener.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

[dnsmichi]

Probably related to #4696 - please test the snapshot packages.

[theseal]

Tested with the latest snapshot

# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: v2.6.3-217-gc611a31)

Copyright (c) 2012-2017 Icinga Development Team (https://www.icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

System information:
  Platform: Ubuntu
  Platform version: 16.04.2 LTS (Xenial Xerus)
  Kernel: Linux
  Kernel version: 4.4.0-77-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 5.3.1
  Build host: 85bd2cef4e43

and we can still reproduce this issue. 😔

Anything else we can test?

[dnsmichi]

Then is is a different issue, solely related to apply rules and object creation. I thought it could be related to the linked change.

[mickenordin]

I think that Icinga2 simply reads the configuration in the wrong order, original configuration should be read first, secondly modified attributes should be read and thirdly apply rules should be applied. Can someone confirm if this is the way it is done currently or if it like I fear that original configuration is read first, then aply rules are applied and thirdly modified attributes are applied?

[dnsmichi]

2.6.x compiles config objects, evaluates apply rules, activates objects and then applies modified attributes. That could lead into the described issue with #4696 where notifications are sent too early.
The change in git master does not activate objects immediately but does that after modified attributes have been applied. Without a peek in the code I would guess that apply rules are still evaluated before modified attributes are applied.

[brianmcardle]

I'm seeing what I believe is this issue also - apply rules working perfectly on data in hostname.conf files, but not for data in the modified-attributes.conf file. Is there an ETA on a fix, or where would I start contributing one?

[htriem]

@Al2Klimov

[Yonas-net]

@Al2Klimov correct me if I'm wrong.

The problem here is that the modified attributes - after all host, service, and child objects are validated - are loaded and thus Icinga will take the previous value of the modified attribute because it doesn't know yet that an attribute of that object has been modified. A possible solution to this would be to not store the modified attributes in a normal variable e.g. (var obj) but pack them all into a lambda function and store these values in the variable modified_attributes. I.e. for each modified attribute, (typeName, objectName and attrs) are stored as Dictionary::Ptr&. So that we can later access the variable and check if an attribute for that object has been modified, get its new value after the host and service have been validated and before Icinga validates the apply rules.