Beats: System diskio on windows - io.time metricset appeared and iit is mapped to 0, also events are coming in empty

[narph]

It looks like one of the linux metrics is visible in the windows diskio events io.time and it is mapped to 0. This was not the case previously.

Also, empty events are being logged (every second event is empty).
Examples:

{
        "_index" : "metricbeat-7.10.0-2020.10.19-000001",
        "_type" : "_doc",
        "_id" : "HKNwaXUBE_RnunIh9dJv",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2020-10-27T09:43:30.962Z",
          "event" : {
            "dataset" : "system.diskio",
            "module" : "system"
          },
          "metricset" : {
            "name" : "diskio",
            "period" : 10000
          },
          "service" : {
            "type" : "system"
          },
          "system" : {
            "diskio" : {
              "name" : "C:",
              "read" : {
                "count" : 7490802,
                "time" : 4324271,
                "bytes" : 176962782208
              },
              "write" : {
                "count" : 3733897,
                "time" : 3186026,
                "bytes" : 124662326784
              },
              "io" : {
                "time" : 0
              }
            }
          },
          "host" : {
          ...
          },
          "agent" : {
            "type" : "metricbeat",
            "version" : "7.10.0",
            "hostname" : "DESKTOP-RFOOE09",
            "ephemeral_id" : "57c038b5-8388-436e-a974-b7f214e37aea",
            "id" : "636db912-3d5a-449f-a5a2-daf0c0e5d853",
            "name" : "DESKTOP-RFOOE09"
          },
          "ecs" : {
            "version" : "1.6.0"
          }
        },
        "sort" : [
          1603791810962
        ]
      },
      {
        "_index" : "metricbeat-7.10.0-2020.10.19-000001",
        "_type" : "_doc",
        "_id" : "HqNwaXUBE_RnunIh9dJv",
        "_score" : null,
        "_source" : {
          "@timestamp" : "2020-10-27T09:43:30.962Z",
          "host" : {
            ...
              "disk" : {
              "read.bytes" : 2709504,
              "write.bytes" : 2906624
            }
          },
          "event" : {
            "dataset" : "system.diskio",
            "module" : "system"
          },
          "metricset" : {
            "name" : "diskio",
            "period" : 10000
          },
          "service" : {
            "type" : "system"
          },
          "agent" : {
            "type" : "metricbeat",
            "version" : "7.10.0",
            "hostname" : "DESKTOP-RFOOE09",
            "ephemeral_id" : "57c038b5-8388-436e-a974-b7f214e37aea",
            "id" : "636db912-3d5a-449f-a5a2-daf0c0e5d853",
            "name" : "DESKTOP-RFOOE09"
          },
          "ecs" : {
            "version" : "1.6.0"
          }
        },
        "sort" : [
          1603791810962
        ]
      },

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[fearful-symmetry]

It looks like one of the linux metrics is visible in the windows diskio events io.time and it is mapped to 0.

For whatever reason, we marked that as "Not implemented, and it also doesn't show 0 value" on windows in our spreadsheet, which is why it wasn't removed. No part of that has been touched recently. I can fix it.

Also, empty events are being logged (every second event is empty).

I suspect this is the host.* events created by @kaiyan-sheng . Can you post an example event without the host.* fields redacted?

[kaiyan-sheng]

So the empty event being logged only has host.* fields?

[fearful-symmetry]

@kaiyan-sheng Yah, the new host.* fields are sent as their own event, with no other metrics

[kaiyan-sheng]

@fearful-symmetry Ahh yes this is for diskio. host.* fields aggregate all disks together and report one event.

[narph]

@kaiyan-sheng , was this the desired effect to send an empty diskio event containing those 2 host values?
Wouldn't this be confusing for users, sending every second event empty, they might think something is wrong (that is what I assumed)?
Also, wouldn't this affect existing visualizations for diskio values or pose issues when users create visualizations as they will see gaps in the data?
cc @exekias

[exekias]

I assume this doesn't affect visualizations, right? By empty you mean the system.diskio.* fields are not present, visualization will not take these events into account, having that the field is not present.

I don't think the current behavior is necessarily wrong, were you able to reproduce any gap in a graph because of this?

@kaiyan-sheng I understand this is a separated event, instead of piggybacking host.* fields to the existing one because it's aggregating several disks, right?

[narph]

I have not tried yet when it comes to visualizations, it could work since I see both events have the same timestamp.
But sending an diskio event with no system.diskio properties does not sound as a correct event (at least to me, maybe this is just a preference).
Couldn't we just add the host fields to the existing event (I understand those are aggregated values) but might be less confusing, and we won't be needlessly increasing the storage space.

[kaiyan-sheng]

@kaiyan-sheng I understand this is a separated event, instead of piggybacking host.* fields to the existing one because it's aggregating several disks, right?

Yes, it's aggregated value among all disks/network interfaces so we can't attach these fields into an existing event which belong to one specific disk/network.

[narph]

Yes, it's aggregated value among all disks/network interfaces so we can't attach these fields into an existing event which belong to one specific disk/network.

I don't think that creating a separate event with no diskio.* data makes this more clear. If we look at the example above we only see:

 "host" : {
            ...
              "disk" : {
              "read.bytes" : 2709504,
              "write.bytes" : 2906624
            }
          },

I do not know if this is aggregated or not. Can we rename it to disk.total.read_bytes or something similar and still include it in the existing event?
If not (which is understandable), what about cases as the one I am testing with, where there is one disk only, can we add the host data to that event and not sent a second one with the same values in the host data?
Thank you all for the clarifications.

[exekias]

I'm not sure we should do an exception for the case where there is only one disk. I think of these metrics as the individual metrics per disk + a summary event aggregating all of them.

I see how for the special case of having a single disk we could put all metrics together, but that:

• Would make the code more complex as we need to account for this case, with its own logic
• Would actually be misleading, as the event has a dimension for the disk name. I wouldn't know anymore if the host.* metrics are aggregated or refer only to that disk

[narph]

@exekias , all right makes sense, I just wanted to explore all avenues in order to minimize the number of documents needed.

[fearful-symmetry]

Fixes have been merged, closing this.