Icinga2: Recurring Downtimes/TimePeriods and DST changes

[terra-nova]

Describe the bug

We have defined a number of recurring downtimes in our Icinga configuration. On 2019-03-31, many of the downtimes were off by one hour. I suspect this is because of the change from winter time to summer time in out timezone (from UTC+1 to UTC+2).

I am not sure if this is a bug in Icinga or a faulty configuration in our MariaDB IDO database.

To Reproduce

Probably need to setup a VM and modify system time, I didn't have the time to do so as of now, unfortunmately.

1.
2.
3.
4.

Expected behavior

Scheduled downtimes should account for changes from summer time to winter time (and vice versa).

Screenshots

Time zone configuration for Icinga system (timedatectl):

      Local time: Mon 2019-04-01 08:40:04 CEST
  Universal time: Mon 2019-04-01 06:40:04 UTC
        RTC time: Mon 2019-04-01 06:40:04
       Time zone: Europe/Berlin (CEST, +0200)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2019-03-31 01:59:59 CET
                  Sun 2019-03-31 03:00:00 CEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2019-10-27 02:59:59 CEST
                  Sun 2019-10-27 02:00:00 CET

Time zone configuration for Database (MariaDB):

+--------------------+---------------------+
| @@global.time_zone | @@session.time_zone |
+--------------------+---------------------+
| SYSTEM             | SYSTEM              |
+--------------------+---------------------+

Contents of IDO table icinga_scheduleddowntimes. Note that the scheduled_start_time column is one hour off from the time listed in the comment_data column.

MariaDB [icinga2_ido]> select scheduleddowntime_id, entry_time, scheduled_start_time, scheduled_end_time, comment_data, o.name1 from icinga_scheduleddowntime dt inner join icinga_objects o on (dt.object_id=o.object_id and o.object_type=1) where scheduled_start_time between '2019-03-31 20:00:00' and '2019-04-01 00:00:00'; 
+----------------------+---------------------+----------------------+---------------------+--------------------------------------------------------------------------+-------------------+
| scheduleddowntime_id | entry_time          | scheduled_start_time | scheduled_end_time  | comment_data                                                             | name1             |
+----------------------+---------------------+----------------------+---------------------+--------------------------------------------------------------------------+-------------------+
|               342232 | 2019-03-24 20:00:56 |  2019-03-31 21:00:00 | 2019-03-31 21:15:00 | Wöchentlicher Neustart So 20:00-20:15 Uhr                               | xxx               |
|               342234 | 2019-03-24 20:00:56 |  2019-03-31 21:00:00 | 2019-03-31 21:15:00 | Wöchentlicher Neustart So 20:00-20:15 Uhr                               | xyz               |
...

When rescheduling a downtime with identical start / end times through Icingaweb, the times look different in the database (note that the scheduled_start_time column matches the times detaield in the comment_data column):

+----------------------+---------------------+----------------------+---------------------+--------------------------------------------------------------------------+----------------------------------------+
| scheduleddowntime_id | entry_time          | scheduled_start_time | scheduled_end_time  | comment_data                                                             | name1                                  |
+----------------------+---------------------+----------------------+---------------------+--------------------------------------------------------------------------+----------------------------------------+
|               349403 | 2019-03-31 20:51:44 |  2019-03-31 22:10:00 | 2019-04-01 04:00:00 | Wöchentliche Sicherung So 22:10 - 04:00 Uhr                             | xyz                                    |

Your Environment

Include as many relevant details about the environment you experienced the problem in

• Version used (icinga2 --version): 2.10.3-1
• Operating System and version: CentOS 7.6 x64
• Enabled features (icinga2 feature list): api checker command graphite ido-mysql notification syslog
• Icinga Web 2 version and modules (System - About): 2.6.2
• Config validation (icinga2 daemon -C): valid
• If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes. I don't think this matters here

Additional context

Can't think of anything else?

[terra-nova]

This issue is possibly related to the problem described in #7069 (not the lack of documentation, but the mishandling of DST changes with regards to future timepoints in Icinga state).

If this is indeed an issue with Icinga and not the way our environment is configured, then one possible way to counter this problem would be to always convert timepoints to UTC when storing them and to always convert timepoints to the local time zone when retrieving them, I think

It might be a good idea to make use of the Boost Date_Time library to offload all sorts of date / time arithmetics.

Just do _something_ before October 27th, please... our on-call-duty guy was almost in tears last weekend 😃

[dnsmichi]

Icinga uses UTC internally, and just dumps these timestamps into the schema, without any extra timezone or offsets. Maybe your schema is different to what we use upstream, older versions had timestamp with timezone and required a schema migration.

Or, the scheduled_start_time was provided by the caller of an API action, and this is done the wrong way. Icinga doesn't verify these timestamps and expects them to be UTC after all.

There's ideas to rewrite the timeperiod related code, but I doubt that this will happen in 2019. There's too many other important things on the plate already. Hopefully EU drops DST at all in the future.

[NotAProfessionalDeveloper]

Does it know what DST changes are at all?
I had a notification go out at 11:31 and in the Notification Event Details it shows Start time | 2019-04-04 10:31:28.
Since we are currently CEST (+0200), I figured it would show the event as having happened at 09:31 (UTC). However it shows a one-hour difference as if we were still CET (+0100). (Even after icinga reload)
The record in the DB is 2019-04-04 11:31:28 BTW

[terra-nova]

@dnsmichi Thanks for the suggestions, this helps to clear things up a bit.

Icinga uses UTC internally, and just dumps these timestamps into the schema, without any extra timezone or offsets. Maybe your schema is different to what we use upstream, older versions had timestamp with timezone and required a schema migration.

Database schema seems to be ok; also, I understand from your reply that the database is just used to dump current icinga state and is not actively being queried to determine the state of objects. So I think I can rule that one out as a source of the error.

Or, the scheduled_start_time was provided by the caller of an API action, and this is done the wrong way. Icinga doesn't verify these timestamps and expects them to be UTC after all.

We generate scheduled downtime with apply rules and predefined TimePeriod objects. An object definition might look like this:

object TimePeriod "Downtime XYZ" {
    import "legacy-timeperiod"
    ranges = {
        "sunday" = "00:00-06:00"
    }
}

And the actual downtimes are generated with an apply rule like this:

apply ScheduledDowntime for (downtime in host.vars.scheduled_downtimes) to Host {
    var timeperiod = get_time_period(downtime)
    ranges = timeperiod.ranges.clone()
    ignore where downtime == ""
}

Now it seems that TimePeriods don't have any notion of time zones at all (see issue #5374), and I take your answer to mean that the time "ranges" definition in TimePeriod objects is also based on UTC time (unfortunately the documentation doesn't state so clearly).

Which begs the question... how are we supposed to define scheduled downtimes in a environment with DST like Germany (UTC+1 winter time, UTC+2 summer time)? If using external tools to do the date arithmetics and then storing downtimes using the API is the only way to do it, then so be it...

[dnsmichi]

You're welcome to look into the code regions on calculating the time periods and create a patch. I won't look into this anytime soon, my schedule is full.

[dnsmichi]

Blocked by

• [x] #7288.

[Mikesch-mp]

1 year and still no progress, do we have to expect the same proplem next year again?

[lippserd]

1 year and still no progress, do we have to expect the same proplem next year again?

Nah, we wait for the European Union to fix this problem for some of us 😆

Anyway, this issue was one of the topics in a call this morning and we're working on a solution.

[Al2Klimov]

How to reproduce in < 1/2y:

1. Linux + TZ=Europe/Berlin + libfaketime + Icinga
2. ScheduledDowntime 12:30-23:30
3. LD_PRELOAD=/usr/lib64/faketime/libfaketimeMT.so.1 FAKETIME='@2020-03-29 12:00:00 x60' /usr/local/sbin/icinga2 daemon
4. ^C
5. Get the fresh Downtime's *nix TSs and interpret them as at 2020-03-29 12:00:00:

[root@localhost ~]# LD_PRELOAD=/usr/lib64/faketime/libfaketimeMT.so.1 FAKETIME='@2020-03-29 12:00:00 x60' date -d '@1585481400'
So 29. Mär 13:30:00 CEST 2020
[root@localhost ~]# LD_PRELOAD=/usr/lib64/faketime/libfaketimeMT.so.1 FAKETIME='@2020-03-29 12:00:00 x60' date -d '@1585521000'
Mo 30. Mär 00:30:00 CEST 2020
[root@localhost ~]#