Icinga2: 2.11 Release Tasks

[dnsmichi]

Release

Scheduled for CW38, 19.9.2019
Live feed: https://twitter.com/icinga/status/1174311275234504704

• [x] Changelog
• [x] Git Tag
• [x] https://github.com/Icinga/icinga2/blob/master/RELEASE.md
• [x] Packages
  
  
  
  • [x] rpm-icinga2 commit & tag
  
  
  • [x] deb-icinga2 commit & tag
  
  
  • [x] windows-icinga2
  
  
  • [x] raspbian-icinga
  
  
• [x] Blogpost @dnsmichi & press (JuH)
  
  
  
  • [x] https://icinga.com/?p=139847&preview=true
  
  
  • [x] Release graphics
  
  
  • [x] Press
  
  
• [x] Online doc redirects @dnsmichi
  
  
  
  • [x] Prepared in puppet, testing branch
  
  
  • [x] go live
  
  
  • [x] Edit stable link, remove RC docs at https://icinga.com/docs/
  
  

RC Feedback

• [x] Wait for RC feedback

General Feedback

• [x] Windows Agents ref/NC/627739
• [x] Reload icinga2 process inside container https://community.icinga.com/t/reload-icinga2-process-inside-container/1873/4
• [x] Stalled CSR signing requests fixed (2.10.5 vs 2.11 RC1): https://github.com/Icinga/icinga2/issues/7363#issuecomment-517138440
• [x] Icinga adds another LC_NUMERIC=C environment variable everytime a reload is fired. #7405
• [x] Fork errors with 200k services, under investigation if related to the RC. ref/NC/628417
  
  
  
  • [x] raise number of open files
  
  
  • [x] Analyse check execution performance #7425 - currently it works as it is, PoCs haven't proven anything better. Re-scheduled for after 2.11
  
  
  • [x] Our CIB's current_concurrent_checks are wrong #7416
  
  
  • [x] Expose metric current_pending_callbacks #7421
  
  
• [x] https://community.icinga.com/t/create-hosts-in-the-director-without-creating-a-zone-and-endpoint-manually-first/1931/9
• [x] Client unable to obtain certificate #7438

Reported Issues

• [x] Cipher lists RHEL7 agents #7366
• [x] Windows agent 2.10.4 -> 2.11.0 RC1 master: no shared cipher #7386
• [x] downtime start/end time API validation #7384 (I've reverted the incomplete fix)
• [x] R2.11.0-rc1-1 endless config sync loop master-satellite #7382 - this syncs unsupported binaries - updated the docs in #7390
• [x] Problems with changing "run as user" on Windows with 2.11 RC1 - Was: Check command 'powershell' does not exist. with new Agent: v2.11.0-rc1 #7387 Turns out that the Powershell module doesn't change this accordingly. Out of our support scope, tested this extensively w/o the module.
• [x] v2.11.0-rc1 Windows Agent does not create debug.log #7388 - cannot be reproduced with win10 nor win2012. The PS module works differently thus creating some trouble here.
• [x] Deny non-utf8 files other than .conf for the cluster config sync #7391
• [x] 2.11.0 rc1 - An error occurred while upgrading the database #7393 - now using upgrade safe procedure
• [x] New umbrella system seems to break startup logging under systemd #7394
• [x] Improve logging of Downtimes #7374 - to mitigate future loops better
• [x] 2.11 rc1: built-in check command "icinga" doesn't work with version compare #7415
• [x] Same downtime being created repeatedly in a cluster loop #7198
  
  
  
  • [x] Fix and improve logging for runtime object sync #7423
  
  
• [x] 2.11 RC1: Nessus Scan crash the Windows-Client. #7431
  
  
  
  • [x] https://github.com/Icinga/icinga2/issues/7431#issuecomment-529543471
  
  
• [x] No reconnect after connection loss with 2.11 RC1 #7444
• [x] RHEL6: IcingaWeb2 (v2.7.1) via API to Icinga2 (v2.11) => SSL Error / no shared cipher #7501

Docs

• [x] Updates release blogpost with cipher lists: https://icinga.com/2019/07/25/icinga-2-11-release-candidate/
• [x] Docs: Enhance Troubleshooting with TLS handshake, CSR signing, checks, notifications, cluster sync #7392
• [x] Enhance Agent best practices throughout the documentation #7403
• [x] Docs: Add cluster message routing details #7420
• [x] Update docs: Fix online URL, cipher list #7451
• [x] Docs: Prefer the Powershell commands on Windows #7463
• [x] Docs: Update supported Windows agent versions #7464
• [x] Docs: Update Linux and Windows Dev instructions for 2.11 #7467
• [x] Dev Docs: Update Windows to Visual Studio 2019 #7480

[dnsmichi]

@mcktr @Al2Klimov @Crunsher @htriem @lippserd @bobapple The master is now fully frozen. Do not merge anything except for small typo fixes. All remaining PRs are on hold.

Waiting for RC/snapshot customer and user feedback.

[widhalmt]

Just to let you know: We got the first replies of customers who started test runs of 2.11 RC on Monday. I'll let you know ASAP when I get feedback how the tests went.

[dnsmichi]

ref/NC/627739

[dnsmichi]

The "no shared cipher" problem with Windows agents was successfully mitigated and fixed with one of our customers.

Next up, is #7382 with a possible upgrade & config sync loop.

[dnsmichi]

The sync loop was from binary files which we don't support. Adding detection is hard, and not reasonable for the 99.9% of users who already do use the config sync just for config files. Therefore a doc fix only.

Two new issues with a missing check command - in feedback loop, and debuglog on Windows missing.

[Thomas-Gelf]

@dnsmichi: couldn't we be strict and refuse to work with anything but 100% valid UTF-8?

[Al2Klimov]

@Thomas-Gelf We already auto-sanitize JSON I/O (as otherwise our new JSON lib would complain).

[Thomas-Gelf]

Auto-sanitation has it's place. It is required to deal with unclean plugin output and (eventually) configuration "from /etc". I would not apply it to data from "trusted" sources. Read: invalid (non-UTF-8) data in "/var" should lead to an error log message followed by an immediate process shutdown. Invalid data via Netstring should lead to a terminated connection. In these contexts auto-sanitation doesn't help and instead makes part of the problem.

[dnsmichi]

Please move this discussion into #7391 - I've been working on this offline with Tom's help already.

This issue is solely for tracking the tasks left open for 2.11, to keep @lippserd & @bobapple updated.

[dnsmichi]

Cluster config sync is done, the missing powershell command turns into wrong permissions and not really being a bug, the Windows debuglog issue remains non-reproducible.

New to the party is the systemd logging which is part of this week's fixing.

[dnsmichi]

• v2.11.0-rc1 Windows Agent does not create debug.log #7388

is solved. The Powershell module is being used, which doesn't support icinga2 feature list and variants. It also collides with our graphical setup wizard using the default configuration layout instead of a single icinga2.conf file.

TL;DR - don't use the Powershell module for RC tests.

[dnsmichi]

The reload logging with failed config validation in systemd #7394 now logs this correctly. Alex and myself also decided to add an additional log line to point users to running icinga2 daemon -C afterwards.

Aug 07 11:53:43 icinga2-centos7-dev.vagrant.demo.icinga.com icinga2[22031]: [2019-08-07 11:53:43 +0200] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.

While testing the Windows agent I was looking at something in the docs and decided to restructure the agents chapter. That's following the updates for the distributed monitoring chapter. Done.

The Windows permission problem in #7387 turned into a problem with the Powershell module, and @LordHepipud pointed me to a Director issue. https://github.com/Icinga/icingaweb2-module-director/issues/1297 - was released with 1.6.1 already 🕺

[dnsmichi]

The network stack with Boost Asio may create fifo pipes visible with lsof. If there's too much, fork errors with too many open files may occur. Under investigation at a customer, raising the nofiles limits as a first shot.

[dnsmichi]

It is not the network stack, it has to do with the check process execution. While mitigating the issue, we've raised the number of open files.

systemctl edit icinga2

LimitNOFILE=50000
LimitNPROC=50000
TasksMax=infinity

vim /etc/default/icinga2

ICINGA2_RLIMIT_FILES=50000

systemctl daemon-reload
systemctl restart icinga2

for p in $(pidof icinga2); do echo -e "$p\n" && ps -ef | grep $p && echo && cat /proc/$p/limits | grep 'open files' && echo; done

for p in $(pidof icinga2); do echo -e "$p\n" && ps -ef | grep $p && echo && lsof -p $p && echo; done

This increased the number of pipes in the main process and fork errors are now gone. Still under investigation why check execution rate may drop - 1000/s vs current_concurrent_checks=10k.

[dnsmichi]

MaxConcurrentChecks is under investigation in our cleanup sprint week, same as the downtime loop. Team @Al2Klimov @bobapple @dnsmichi.

Small version parse fix incoming for the icinga check.

[dnsmichi]

Fork errors are resolved with raising the number of open files, as described in the troubleshooting docs. The general performance is analysed and tested once more.

[dnsmichi]

Coming late to the party, the downtime create/delete loop in HA clusters has been fixed this week with #7198. A nearly 4 year old problem.

[dnsmichi]

The Windows scan crash parts #7431 only exist in RC1 and git master coming from the network stack rewrite. Interestingly enough, this only causes problems on Windows, but not Linux/Unix. It is timing point related and must be fixed prior the release.

[dnsmichi]

The Windows thingy is still going on. Already cost too much time, nerves and sleep.

[dnsmichi]

The fixes for Nessus on Windows are done and already tested. A small fix inside has been reverted causing trouble with the HTTP API. The snapshot packages from today are fully working.

Release is scheduled for CW38.

[dnsmichi]

• RHEL6: IcingaWeb2 (v2.7.1) via API to Icinga2 (v2.11) => SSL Error / no shared cipher #7501

adds another compatible cipher suite for el6.

[dnsmichi]

armhf/raspbian & the branch/tag thing on GitLab did cost me some time, therefore the build is running for the next 3 hours coming after the announcement: https://git.icinga.com/packaging/raspbian-icinga2/pipelines/5458

[dnsmichi]

Everything else is done thus far, coordinating the announcement with Julia.