Beats: [Metricbeat] add resource discover metricset for AWS

[kaiyan-sheng]

It would be nice to add resource discover for AWS platform. For example, when user gives credentials to a specific AWS account, Metricbeat can discover all resources that exist in this AWS account. This will be helpful to give user an overview on what are the resources currently exist and this can be useful for overall infrastructure management, billing, budgeting and etc. Or with unique ARN for each resource, we can use it to even drill down to see metrics/logs that belong to one specific resource based on the ARN.

This idea came from what Google Cloud Platform has on their monitoring portal:

It would be good to add a similar inventory overview dashboard for AWS as well in Kibana as the first step. Later on, we can extend this to the rest of the public cloud platforms and also make the overview table clickable so it extends to show all ARNs. With these ARNs, we can also improve the current metricsets(For example add ARN to ec2 metricset) to link the resource to its monitoring metrics/logs.

cc @sorantis @exekias

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[exekias]

Do we have the ARNs for all the resources that we monitor right now?

I understand this would be a metricset reporting resources, right? Perhaps resources is a good name for it

[kaiyan-sheng]

resources sounds good to me too.
I believe we are only collecting ARNs for rds metricset. If this is useful, we can investigate how to add ARN into cloudwatch metricset(so it works for the light modules) and the rest. Also need to concider how much extra cost it will bring if it requires another API call. I believe we are already using the same API call to get tags in most metricsets so we shouldn't be adding new API call.

[sorantis]

Got another validation that autodisovering all services/resources used by an AWS account and automatically monitoring them is a wanted feature especially in multi-account cases supervised from a superuser account.

[ChenTsungYu]

It would be good to add a similar inventory overview dashboard for AWS as well in Kibana as the first step. Later on, we can extend this to the rest of the public cloud platforms and also make the overview table clickable so it extends to show all ARNs. With these ARNs, we can also improve the current metricsets(For example add ARN to ec2 metricset) to link the resource to its monitoring metrics/logs.

Hello, support

I want to ask is there any module of metricbeat for google cloud to help count all monitored resources used because I need to collect the information about the total quantity in each resource.

Thanks!

[kaiyan-sheng]

@ChenTsungYu Hello! We don't have it for google cloud right now. But it would be very helpful to have this functionality. Do you mind creating a github issue for this enhancement and explain about your use case please? Thank you!

[ChenTsungYu]

@ChenTsungYu Hello! We don't have it for google cloud right now. But it would be very helpful to have this functionality. Do you mind creating a github issue for this enhancement and explain about your use case please? Thank you!

@kaiyan-sheng
Hi support,

It's OK! Thanks a lot 😄
BTW. If I want to provide the other feature suggestions in metricbeat, where should I send the ticket?

Best

[kaiyan-sheng]

@ChenTsungYu I believe https://github.com/elastic/beats/issues/new?template=feature-request.md is a good spot. Thank you!