Netdata: [RFC] Enabling the perf plugin by default

[vlvkobal]

Question summary

The perf plugin was introduced in the PR #6225 as a resolution for the issue #3232. It uses the perf_event_open system call. Thus, for every hardware monitoring event for every CPU core, a separate file descriptor should be opened. For example, on a 4 core CPU:

...
lrwx------ 1 root root    64 Jun 19 20:50 98 -> 'anon_inode:[perf_event]'
lrwx------ 1 root root    64 Jun 19 20:50 99 -> 'anon_inode:[perf_event]'
$ sudo ls -al /proc/2158/fd | wc -l
182

Another problem is that the number of PMUs in a system is usually quite limited. For example, modern Intel processors typically have only 4 PMUs per core. It means that we have multitasking on PMUs much like on CPUs.

Events share the same PMUs, and every counter runs only during a small fraction of time. If we run another PMC monitoring software, it will share the PMUs with Netdata, and every counter will get even less time to run.

It doesn't affect performance, but the less time the counter has to run the less data it gets. So, the estimation of the total is less precise.

The question is, what policy are we going to choose?

• Enable the plugin by default.
• Keep it as is - disabled.
• Enable it but restrict the resource usage somehow.

Component Name

perf plugin

[Saruspete]

Hello,

Being used to PCM in netdata, that kind of detail is very interesting. However, it get its full potential when you have a per-cpu (or at least per socket) detail.
Having the overview of a system with 72 CPUs is not that helpful. You need this kind of data when one of your application is running slow (or not as fast as usual), and need to understand why. So you already have isolated cores dedicated for your app, and need only to check those.

I'm afraid having this kind of data with only system-wide details are not precise enough for people who may want this kind of monitoring.
For the less tech-savy users, the very technical status of these metrics may ultimately confuse them: I already have users that see issues where there is none, just be mis-reading the graphs (eg, a spike without reading the scale, a normal drop, etc...)

There's also some bugs in some versions: on rhel6 (yeah, old but still quite used in many corp) PCM doesn't work correctly with perf support, so it must be disabled at compile time.

It will also break some developers tools that uses perf or direct PMU (like Intel VTune).

As a feature, maybe you can check / ask to disable sysctl "kernel.nmi_watchdog" as it consumes another pmu.

I would go with "disabled by default". Enable it if users understands the requirements and the use case. That would need a lot more documentation to be added