Beats: Metricbeat doesnt give network io and disk io per process

[indu-sharma-guavus]

Metricbeat is lacking with big feature i.e network io and disk io. These are the most critical metrics which need to be considered. Can somebody take a look at this ?

[andrewkroh]

Metricbeat can collect this information with its system module.

• system
  
  
  
  • diskio
  
  
  • network
  
  

[andrewkroh]

Oh, I see in the title only that you also mention "per process".

I'm not aware of per process network metrics on Linux in the /proc unless the process has been put into its own netns. There's an open issue around grabbing this data. Additionally I do think you could get per process data with eBPF (it would just take a bit of work).

So I think this is a duplicate?

There's an open issue for process IO too. https://github.com/elastic/beats/issues/4241

[indu-sharma-guavus]

Thank you very much @andrewkroh . eBPF is just the best idea, just that integrating its stats with metricbeat may be tedious, and metricbeat is something we can't live without because of its easy compatibility with ELK stack. There are other tools out there just for network i/o and diskio nethugs and iotop respectively.

By the way, i can see you've committed code in your private branch. What shall i do to consume your branch and achieve the one i've been looking for ?

[andrewkroh]

To get the netns metrics integrated I would checkout elastic/beats from master and port over the changes from https://github.com/andrewkroh/beats/commit/7dd6ba880f9d577e134c1be225818276cf9e3a17 by hand. It's been sitting on that branch for close to 2 years and lot's of things in Metricbeat have changed.

The main change being that code for collecting process data was moved over to libbeat so that will be where a lot the changes are needed now. https://github.com/elastic/beats/tree/master/libbeat/metric/system/process

[javadevmtl]

I had similar debugging issue recently and I had to use a tool like atop to get that info. So I could see disk spikes on the io dashboard, but couldn't associate it back to a process.

[mochiman33]

Will the disk io and network io per process stats be released on a version of metricbeat in the near future?

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[willemdh]

Imho, this shouldn't be closed, see also https://discuss.elastic.co/t/metricbeat-not-getting-diskio/251606/8