Beats: Create rollup job examples for Metricbeat modules

[ruflin]

In https://github.com/elastic/beats/pull/7220 automatic generation of rollup jobs was tried out. As the rollup jobs should not necessary be generated on the Beats side I decided to close this PR and rather focus on documentation / examples for rollup jobs. This issue is to track this.

[JonnyBGod]

It would be awesome to have base curated templates for metricbeat. filebeat, auditbeat based on all new kibana's apps like Infrastructure and Logs including kubernetes and docker.

These could serve as a reasonable starting point with reasonable parameters, not too strict but enough to provide a good clean up and maintenance experience.

I would also include APM related rollup templates but this is probably better for another story.

The point is that it is now vary easy to bootstrap and elastic cluster and setup all beats for kubernetes. But this data grows very fast and it is very time consuming for a small team to maintain these elastic clusters. For now I see all my teams falling back to curator and just deleting all old data which is far from a good solution.

[jeff-cook]

What if each beat was to save/export a JSON file with a recommended rollup config? Then each admin would have something to use with the rollup API to create a rollup. Beats would not sending this to the cluster, so the beats agent doesn't need permissions. But it would give the admins someplace to start.

[JonnyBGod]

Any news on this topic?

Also would it be possible to use rolled up indexes with curated apps like APM, Logs, ML and Infrastructure?

[florian-asche]

Have a look at elasticsearch-curator.

[rayafratkina]

@ruflin I was wondering if there are plans to create any of these soon?
We merged changes to add rollup support to TSVB, so there is a chance to create some interesting (and useful) examples.

cc @AlonaNadler

[ruflin]

@rayafratkina Good to know TSVB now supports rollups as this was one of the blockers in https://github.com/elastic/beats/pull/7220 There is still the problem on the timing of loading rollup jobs from the Beats side as it can only be loaded after data is there. We should perhaps rather schedule this for when we have modules in Kibana?

[rayafratkina]

@ruflin I defer to you on this, just wanted to check if there are plans to do this.

[ruflin]

@rayafratkina I would say in the current module structure there are no plans but hopefully in the new one. I definitively see value in having this.

[florian-asche]

again... Have a look at elasticsearch-curator. also available as chef cookbook.

[rozling]

@florian-asche AFAIK curator doesn't support the rollup api, and even if it did it wouldn't relate to this issue: "documentation / examples for rollup jobs".

@ruflin a quick google doesn't reveal anything for a new beats module structure. For those of us not so close to the beats dev roadmap, could you point to more info on this? I'd like to throw in my vote for this feature, FWIW, wherever the discussion is going on.

I'm currently trying to create rollup jobs on metricbeat data, and the sheer number of fields / options is pretty intimidating.

I'm sure there are people in the Elastic team who could point users towards sensible / common options and help us get the most out of this feature; it would be hugely appreciated!

[ruflin]

@rozling There is not a single issue at the moment I can point you to but this repository here should give you the direction this is heading: https://github.com/elastic/integrations-registry

For the fields explosions, one of the issue can be found here: https://github.com/elastic/kibana/issues/24709 A few other issues are linked in it and some ideas are tried out, but I would not say we fully solve this yet.

[willemdh]

@ruflin Any progress on this? We are also looking into rolling up metricbeat data. Some sane rollup defaults for metricbeat indices provided by Elastic would ease the pain a bit imho.

[ruflin]

Yes and no. The project I mention above is getting close to a first release, but it will not have any rollup jobs in yet. The goods news is the foundation will be here to build these in, in the future.

[bleed2002]

Found this issue while searching for a way to limit disk space usage for my elastic stack. I currently use it mainly for monitoring data collected by metricbeat and we see storage requirements of about 500 MB per host (default config of system and nginx modules) which quite sums up when more hosts are under monitoring. Rollup jobs seem to be the best solution but currently I do not know an easy way to get this setup. What I would like to achive is something like: Keep data for last 2 weeks as it is, from 2 weeks to 4 weeks change resolution to 1 minute (from 10s), after 1 month to 3 month change it to 10 minutes, afterwards delete the data alltogether. Could you give an estimate when this functionality should be available in a convienent way? Thanks!