Collectd: Next collectd release

[maryamtahhan]

Hi Folks
just wondering at a high level is there any plans for another release before EOY?
My team [ @rkorynkx @serhiyx @vmytnykx @taraschornyi ] and I would be more than happy to help out in any way we can.

Thanks
Maryam

TODO items

• [X] #2309 - snmp: Fix double free / handle missing variables - it fixes a bug, which causes Collectd crash.
• [X] #2449 – snmp plugin: Added res->errstat check
• [X] #2377 – openldap: Fix crash on shutdown with openldap connection failed - it fixes a bug, which causes Collectd crash.
• [X] #2385 – memcached: Fix hitratio reporting, add connections rate report
• [X] #2473 - memcached: Fix CPU usage reporting
• [X] #2388 – memcached: Persistent connections with IO timeouts
• [X] #1944 – Allow to set custom plugin name in 'generic' plugins. ("umbrella issue")
• [X] #1558 – Curl_XML: Custom names for plugin/plugin instances
• [X] #1563 – Curl_JSON: Custom plugin name; PHP-FPM configuration example
• [X] #1681 – Allow custom plugin name for 'curl', 'memcachec', 'tail' and 'tail_csv' plugins.
• [X] #1707 – Allowed custom plugin name in 'dbi', 'oracle' and 'postgresql' plugins.
• [X] #1979 – Filecount plugin: Custom values for reported plugin, type and type instance
• [X] #2435 – Table plugin: Allow custom plugin name to be reported
• [X] #2449 – snmp plugin: Added res->errstat check
• [X] #2091 – ipmi: add support for system event log (SEL).
• [X] #2071 – ipmi plugin: add more analog sensors support
• [X] #2135 – python: Extend Notification class
• [X] #2398 – intel_pmu: add metadata with information about scaling
• [X] #2024 – ipmi plugin: Implemented IPMI LAN connection and plugin improved

[rpv-tomsk]

There is also many my PR which have been improved after reviews and ready to merge/ used in my production env: https://github.com/collectd/collectd/pulls/rpv-tomsk
Some of them are pending for more than year. They needs to be reviewed/merged (or rejected) before the new Collectd release.

I also will be happy to make things, required for new release.

[f1-outsourcing]

ceph luminous is out for a while, and still we are having these json errors. https://github.com/collectd/collectd/issues/2343 If I can be of any service with dumps I am happy to assist.

[octo]

+1 for a feature release sometime soon. Since this would mean ending support for 5.6, we should do a patch release for 5.6 first, though. Luckily, those are not as much work. Still, EOY should easily be possible.

@rpv-tomsk, could you compile a list of PRs that you think are ready to be merged? I'll make an effort to work through those next week.

The most time-consuming part of a new release is writing a (curated) change log. The wiki is the source of truth there and all other places where the change log is shown (Github release, ChangeLog file, email, …) are generated from that. @mfournier has been a tremendous help in the past in compiling that. If anybody else would like to help, they're more than welcome :)

Best regards,
—octo

[rpv-tomsk]

Patches which I consider important and which are ready to go in:

• [X] #2309 - snmp: Fix double free / handle missing variables - it fixes a bug, which causes Collectd crash.
• [X] #2449 - snmp plugin: Added res->errstat check
• [X] #2377 - openldap: Fix crash on shutdown with openldap connection failed - it fixes a bug, which causes Collectd crash.
• [X] #2385 - memcached: Fix hitratio reporting, add connections rate report
• [X] #2388 - memcached: Persistent connections with IO timeouts

Once one of #2385/#2388 will be merged, the other will cause small merge conflict.
I will rebase it if/when needed.

Series of patches related to #1944 are useful and awaiting of merge for a long time:

• [X] #1944 Allow to set custom plugin name in 'generic' plugins. ("umbrella issue")
• [X] #1558 Curl_XML: Custom names for plugin/plugin instances
• [X] #1563 Curl_JSON: Custom plugin name; PHP-FPM configuration example
• [X] #1681 Allow custom plugin name for 'curl', 'memcachec', 'tail' and 'tail_csv' plugins.
• [X] #1707 Allowed custom plugin name in 'dbi', 'oracle' and 'postgresql' plugins.
• [X] #1979 Filecount plugin: Custom values for reported plugin, type and type instance
• [X] #2435 Table plugin: Allow custom plugin name to be reported

Also I can work on PR from other authors, not only mine, just point that.

[rpv-tomsk]

Let's add

• [X] #2449 - snmp plugin: Added res->errstat check

into this list too?
I don't like to postpone its review to another months before next release. )
I will forget again it's details by this time. )))

[maryamtahhan]

Hi Folks
Sorry for the delay:

Reviewed:

• [x] ipmi: add support for system event log (SEL). https://github.com/collectd/collectd/pull/2091

Needs a final review

• [ ] ipmi plugin: add more analog sensors support https://github.com/collectd/collectd/pull/2071 --> @rpv-tomsk if you could review this I would be very grateful.
• [ ] Plugin for PCIe Advanced Error Reporting https://github.com/collectd/collectd/pull/2382

Needs review:

• [x] python: Extend Notification class https://github.com/collectd/collectd/pull/2135
• [x] intel_pmu: add metadata with information about scaling https://github.com/collectd/collectd/pull/2398

We can rebase the 2 ipmi ones against the latest master as it's been a while since they were rebased.

[rpv-tomsk]

Hi Maryam!
Of course, I will look into IPMI-related PRs. If we want to put these into new release, can you please look into mine PR also:

• [X] #2024 ipmi plugin: Implemented IPMI LAN connection and plugin improved

Thanks!

[octo]

As mentioned above, one of the prerequisites is that we bring the 5.6 branch up to date one last time. I've written a preliminary change log. It'd be great if you could review.

[maryamtahhan]

@octo I'm about half way through the commits atm...

A couple of typos in existing change log:
BIND plugin: Fix parsing of the sample time provided by BIND. Previously, the time was assumed to be in the local timezone when in fact it was in UTC. Thanks to Ed Ravin. #1268 --> time zone
RRDtool plugin: Incorrect handling of the flush'es timeout option has been fixed. Handling of the RandomTimeout has been fixed. Thanks to Pavel Rochnyack. #2363 --> flushes

_Missing:_

• Notify_email plugin: no longer explicitly links against libssl and libcrypto, relies on libesmtp.Thanks to Marc Fournier.
• Collectd: fix for memory leak in match utility. Thanks to krzysztofx https://github.com/collectd/collectd/pull/2192

[octo]

Thanks @maryamtahhan, much appreciated! I've fixed the typos and added a note about notify email. The memory leak (#2192) was already mentioned under "memcachec, Tail plugins".

[octo]

FTR, 5.6.3 is out, we can now focus on 5.8.0.

[octo]

I moved all outstanding PRs to the initial comment so the task list renders nicely.

@maryamtahhan I dropped #2382 Plugin for PCIe Advanced Error Reporting – due to its size I don't think I can review it thoroughly in time.

[octo]

I got started on https://collectd.org/wiki/index.php/Version_5.8 (but didn't get very far yet). Any help is welcome :)

[rpv-tomsk]

I moved all outstanding PRs to the initial comment so the task list renders nicely.

Why not to add #2024 (or its parts) too?

[octo]

@rpv-tomsk oversight on my part. I'll add it when I'm sitting in front of a computer :-)

[maryamtahhan]

@octo Cool, no stress. Thank you.

[octo]

FYI, the 5.8 change log is slowly materializing here: https://collectd.org/wiki/index.php/Version_5.8#Release_5.8.0 – I'm about halfway done.

So. Many. Changes. :open_mouth:

[maryamtahhan]

@octo I will put some time away tomorrow morning to help go through the change log and add more if possible. sorry for the delay in helping out on this one.

[maryamtahhan]

Some updates for now:

To be updated:
DPDKEvents plugin: This new plugin reports link status and keep alive events. Thanks to Maryam Tahhan, Harry van Haaren, Serhiy Pshyk, Kim-Marie Jones, Krzysztof Matczak, Przemyslaw Szczerbik, Christian Ehrhardt and Luca Boccassi. #2157, #2348, #2400, #2405, #2417
DPDKStat plugin: The plugin has been refactored to make DPDK related utility functions reusable. Thanks to Krzysztof Matczak, Przemyslaw Szczerbik, Christian Ehrhardt and Luca Boccassi. #2130, #2348, #2400, #2405, #2417
Intel PMU plugin: This new plugin collects CPU performance metrics using Intel's Performance Monitoring Unit (PMU). Scaling information added to metadata. Thanks to Serhiy Pshyk and Roman Korynkevych. #2276, #2398, #2374
mcelog plugin: This new plugin subscribes to Machine Check Exceptions (MCE) and dispatches notifications. Metadata reset. Thanks to Maryam Tahhan, Volodymyr Mytnyk, Taras Chornyi, Krzysztof Matczak and Roman Korynkevych. #2003, #2246, #2380

To be added:
Collectd as well as apache, memcached, openldap, perl, routeros, snmp, tail_csv plugins : free userdata for plugin_register_complex_read(). Thanks to Pavel Rochnyack. #2349
Collectd client library: added parsing and server code. Thanks to Florian Forster. #2258
Ipc plugin: Fixed failed compilation on AIX. Thanks to Pavel Rochnyack. #2357
Memcached: Implemented persistent connections. Thanks to Pavel Rochnyack. #2388
Perl plugin: Bugfix, added check of proper interpreter initialization. Thanks to Pavel Rochnyack. #2391
Uptime plugin: changed implementation to read from /proc/uptime instead of /proc/stat. Thanks to Ivan Kurnosov. #2431
Synproxy plugin: This new plugin provides statistics for Linux SYNPROXY. Thanks to Marek Bečka. #2381

[maryamtahhan]

To be added:
Collectd: Added daemon option to avoid making BaseDir. Thanks to Nathaniel Wesley Filardo and Florian Forster. #2422
Cpu plugin: add linux-specific "guest" states. Thanks to Xavier G. #2439
Filecount plugin: Custom values for reported plugin, type and type instance. Thanks to Pavel Rochnyack. #1979
Mqtt plugin: Add support for TLS in Subscriber blocks. Thanks to Florian Forster. #2434
Nfs plugin: Add config options to ignore versions and . Thanks to Christian Bartolomäus. #2430
Openvpn plugin: Added support for status files from latest OpenVPN-2.4 and possible from future versions. Thanks to Pavel Rochnyack. #2352
Table plugin: Allow custom plugin name to be reported. Thanks to Pavel Rochnyack. #2435
Tail plugin: Allow the calculation of several distributions in one 'tail' instance through a 'bucket' type. Thanks to Pavel Rochnyack. #2442

[maryamtahhan]

@octo I would add them to the wiki myself - but no privileges :( I could start updating the changelog though if that helped?

[octo]

@maryamtahhan I've created an account for you – an email with a random password should have been sent to your address. Let me know if anyone else from your team needs an account.

[maryamtahhan]

@octo perfect got it. I will have a stab at an update tomorrow morning. Also any interest in getting some wiki pages for some of the plugins like DPDK, OVS, PMU, RDT written up? I maintain a lot of documentation around them in the barometer project in OPNFV, through the wiki and documentation portal:
https://wiki.opnfv.org/display/fastpath <-- some design wikis and other bits and bobs here that might be useful
http://docs.opnfv.org/en/latest/submodules/barometer/docs/release/userguide/feature.userguide.html --> setup and configuration instructions for a bunch of different plugins...
There's probably good opportunity to re-use or reference some of what's been done there. Would be more than happy to do this.

[octo]

They sounds amazing. Ideally there would be a Wiki topic for each plugin, and having the contributors write the topic is the gold standard. Unfortunately I don't have the resources to write all missing topics myself.

[maryamtahhan]

Hi @octo I think we are up to date on the changelog in the wiki, the only 1 left to be added is:

2300

[octo]

Awesome, next would be to freeze 5.8 and create a version branch for it. How does November 1st sound for a freeze date?

[octo]

FTR: freeze announcement