Beats: [Heartbeat]: Cloud provider target autodiscovery

[pmoust]

Proposing to add target autodiscovery using Cloud Providers' APIs for Heartbeat monitors.

Example configuration

• monitors.autodiscovery.provider (ex aws, gcp, azure etc)
• monitors.autodiscovery.resource (ex ec2, elb, gcs, nlb etc)
• monitors.autodiscovery.credentials (fields vary per provider, ex. for aws type it would be something like secret_access_key, access_key_id etc, env vars taking precedence)
• monitors.autodiscovery.credentials.path (where applicable, i.e. gcp json token blob)
• monitors.autodiscovery.filters (accepting multiple filters, haven't given much thought to notation. Filters would all need to match or use a notation like the event post-processor (?))
• monitors.autodiscovery.interval (how often the autodiscovery mechanism will get triggered, in seconds)
• monitors.hosts accepting patterns based on autodiscovery results

heatbeat.monitors:
   - type: http
     schedule: '*/5 * * * * * *'

     autodiscovery.provider: "aws"
     autodiscovery.resource: "ec2"
     autodiscovery.credentials.secret_access_key: "F0000000000000000000000000000000"
     autodiscovery.credentials.access_key_id: "BAAAAAAAAAAR"

     autodiscovery.interval: 60

     autodiscovery.filters: 
       - name: "filter by tag"
         type: "tag"
         key: "environment"
         value: "production"
       - name: "exclude us-west-1"
         type: "region"
         value: "! us-west-1"
       - name: "Target the haproxy security group"
         type: "security_group"
         value: "haproxy"

     hosts: ["http://${autodiscovered.private_ipv4}:80/status"]

[pmoust]

I could have come up with a generic proposal to add cloud provider autodiscovery in libbeat (as we do with docker, k8s etc) and provide a list of first targets and exported fields of autodiscovered entities.

However I feel that would increase scope unnecessarily for a first stab. My first candidate to use Cloud Provider autodiscovery would be Heartbeat anyway :)

Happy if I see it developed properly in libbeat autodiscovery provider package of course!

[ruflin]

I definitively like the idea of a cloud autodiscovery provider. I think adding it to the generic providers instead of trying to get it into Heartbeat would be simpler.

[exekias]

That's my feeling too, once https://github.com/elastic/beats/pull/8023 is in, adding autodiscover to heartbeat should be fairly easy, which would be a win-win situation. Many beats could benefit from this provider

[andrewvc]

Yes, confirming what @exekias wrote, #8023 is a blocker for this. It should be merged relatively soon.

[andrewvc]

As an update here, I've begun work on an AWS ELB autodiscovery provider in #8680

[elasticmachine]

Pinging @elastic/uptime

[andrewvc]

As an update, I'm going to revive https://github.com/elastic/beats/pull/9122 soon

[andrewvc]

This was (partially) merged in https://github.com/elastic/beats/pull/12401 . I think we can safely consider this the start of this experiment. I'm going to close this issue for now. If it's successful we can consider adding EC2 and other integrations.