Kibana: Metricbeat disk usage visualization not working correctly with interval:auto (TSVB)

[stacey-gammon]

I'm not sure if this is a TSVB error with the interval: auto setting, or a case of metricbeat default visualizations not setting the right defaults, but I am able to repro the issue mentioned in this discuss ticket: https://discuss.elastic.co/t/disappearing-disk-usage/129095

cc @simianhacker @elastic/beats

[simianhacker]

This has to do with the frequency of the data vs. the auto bucket size. If your data is coming in every 1 minute but your auto bucket size ends up being 10 seconds. Then when your page loads it will sometimes be zero ( 83% of the time it WILL be zero), just depends if the data shows up in the last bucket we display or not. When you customize the data frequency for Metricbeat to 1m you also need to update all the TSVB based visualization to use >=1m for the interval. At this time there is no way to fix this other than setting the interval under panel options to >=1m without some kind of communication from the Metricbeat process notifying Kibana's visualization that the frequency is lower then the visualizations, maybe the "data sources" feature will fix this.

I'm inclined to close this issue because TSVB is working as expected, if anything this should be a feature enhancement for having mechanism for Kibana where Metricbeat can update all the intervals when it detects a change in the publish interval config.

[stacey-gammon]

I think the bug then is that metricbeat doesn't set this interval in their default dashboard. Seems like their dashboard should work out of the box, and if the metricbeat default interval is 1m then so should the default visualization use 1m interval.

[simianhacker]

Yeah... the question is, if they run metricbeat --setup then change the interval because they realized they don't have enough storage to use the default interval, should they run metricbeat --setup again?

Probably a question for @monicasarbu or @tsg

[stacey-gammon]

Hm, yea I just don't recall touching system.yml to some custom interval, so it seemed to me that they are mismatched out of the box. Once a user changes the default .yml settings, then I can see it being reasonable to tell them they have to modify the visualization as well (though would also be bevery cool if metricbeat --setup updated it for them)

[simianhacker]

I wish there was a way to move this issue to the beats repo, this more of an issue with the defaults and less of an issue with anything in Kibana. I guess I will just create a new issue and link to this one.

[ruflin]

I agree on the part that our "dashboards" should set work well with our defaults and we must fix that. But it would be great to have a feature in Kibana that detects the interval and does the right thing. I'm thinking to send up the interval of each event under a certain field like event.interval. Based on this field Kibana could make a decision on what bucket size to use or show errors when a bucket size too small is used. Our users even change the interval over time so not requiring a setup call (which will overwrite all changes) would be great. Otherwise we probably would need an API to set the interval.

As this is out of scope of this issue I wonder if we should close this and open a new one for an enhancement request?

[exekias]

Would it make sense to take the last known value in this cases? I understand the last bucket is empty for the given interval and bucket size, but some of the previous buckets must contain the value we want.

[cdvv7788]

Any news on this issue? I still have issues with some visualizations (specially on disk). If I select "today" for the range, everything appears blank (My processes have been running for around 1 hour), so I have to select something smaller...the process to actually get data is kinda confusing.