Telegraf: Input: CloudWatch - Partial Wildcard or Regexp support?

[mooyoul]

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Support Partial Wildcard (e.g. *prod*, like glob) or RegExp support (e.g. /prod/) to filter out unwanted metrics

For example, below configuration will pull production metrics only:

Using RegExp:

[[inputs.cloudwatch]]
  namespace = "AWS/Lambda"
  [[inputs.cloudwatch.metrics]]
    names = ["Invocations", "Errors", "Duration"]
    [[inputs.cloudwatch.metrics.dimensions]]
      name = "FunctionName"
      value = "/prod/"

... or partial wildcard (globbing):

[[inputs.cloudwatch]]
  namespace = "AWS/Lambda"
  [[inputs.cloudwatch.metrics]]
    names = ["Invocations", "Errors", "Duration"]
    [[inputs.cloudwatch.metrics.dimensions]]
      name = "FunctionName"
      value = "*prod*"

Current behavior:

There's no easy way to select production-only metrics without specifying them.

Desired behavior:

Users can filter metrics using Glob or RegExp

Use case:

Since our architecture of our application is microservices, We have 300+ lambda functions and we have 3 stages (production, staging, development) and number of functions are growing.
We replicate cloudwatch metrics to our InfluxDB using Telegraf with Cloudwatch Input Plugin to detect anomaly metrics (e.g. Latency Spikes). When something goes wrong, we receive alarms from Bosun.

Anyway currently filtering production resources is so hard. everytime we release new "service" to production, we need to change bunch line in the telegraf configuration to filter non-production metrics.

[danielnelson]

We currently have support only for * in the dimension value, but I think this can be extended to support all globs. This method would require following the code path that runs ListMetrics.

[jivanpuri]

We are also waiting for this feature. Is there any plan to include this in the near future?

[danielnelson]

in the near future

There is no work being done on this by us right now, but we have some other fairly large improvemnts coming to this plugin in #5544.

[glinton]

Can you show an aws list metrics command with input that produces your desired output?

[CpuID]

Can you show an aws list metrics command with input that produces your desired output?

Pretty sure the request was for a variation of aws cloudwatch list-metrics. Doing some testing, pretty sure there's no wildcard support on the API call :(

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_ListMetrics.html

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_DimensionFilter.html

I suspect this will only be possible using client side filtering (within telegraf), due to a lack of AWS API functionality for wildcard matching.

[glinton]

It would definitely save requests by making this part of the plugin, but in the meantime, using telegraf metric filtering should be a suitable workaround.

[PmeyerSwiftpage]

Regex on dimension is very important to us. For example, we have thousands of Application Load Balancers, and need to get those metrics for a certain set (hundreds). Pulling metrics for all, takes a long time (and sometimes run into REST API throttling limits).
So, instead of pulling all and using '.tagdrop' to remove the thousands of lb's we don't want, would rather use a regex to define the ones we do want (and not have to list hundreds of lb names, which also change dynamically).