Beats: [Functionbeat] Allow to automatically subscribe to new cloudwatch/sqs/kinesis trigger.

[ph]

The description below is relevent for SQS, Kinesis and cloudwatch log group.

The current workflow of Functionbeat requires you to know beforehand the name of the cloudwatch log group, you cannot register a wildcard. This cause problem when you have a really large number of log groups or you are creating them and removing dynamically.

We should add an autodiscover mode to Functionbeat that will allow you to listen from events coming from your AWS infrastructure to automatically subscribe new log group to your Functionbeat lambda.

[ph]

@kvch This is really an interesting feature that might interest you, I am not sure of the implementation yet, I presume that the permissions will be hardest part here.

[tfendt]

+1 on this. Please allow a wild card options for log_group_name. We have 1000+ log groups in one environment. They get added automatically (and removed) when a new serverless service is created and deployed which in our dev environment happens frequently.

[spectorar]

+1 to the wildcard solution. One thing I dislike about AWS's Cloudwatch -> Elasticsearch Service feature is that it has to be configured per log group. As others have said, there could be a lot of log groups. Allowing wildcards/prefixes in the config for FunctionBeat would greatly simplify this. An autodiscover feature feels too broad. Every new cloudwatch log group would be too much for our use case.

[phillpafford]

+1 for wildcard

[jasonslater2000]

+1 on this as well, without wildcard/autodiscover, configuration would be untenable

[jderose9]

I'm struggling to see how functionbeat can be useful at scale without this feature.

[johnnydimas]

This is a must-have!

[alphachai]

We were planning to use functionbeat, but without this option, I'm not sure how it's even possible at scale.

[cluggas]

Rather than wildcard the log groups, would you consider removing the need for the triggers section in the functionbeat.yml file entirely?

AWS treats the Log Group Subscription and the Subscription Target as different resources, and only couples them in one direction allowing Subscriptions to scale up independently of Targets. By requiring the triggers list in functionbeat.yml the coupling goes in the other direction from Target to Subscription, which makes for an inflexible and unscalable architecture.

Could triggers be made optional? I realise it is required if you are using the Manager to deploy the Log Group Subscriptions. But anyone using this at scale probably won't be using Manager.

[kvch]

@cluggas You raise a valid point. Could you please open a separate issue for your request?

[kaykhancheckpoint]

Did anything come of this? has anyone found an alternative solution to deal with auto discovering cloud watch log groups,. whilst we wait for an official solution ?

[greendad]

@kaykhancheckpoint One way would be standardising all the log groups that I am interested in with the same prefix in name and I am injecting the log groups dynamically into functionbeat configuration when deploying it. So when a new log group is added, I would just trigger the functionbeat deployment again(through a trigger) automatically. I am not sure if it exactly fits your use case. I haven't implemented the trigger yet FYI.

[swiftmas]

We have been looking at a solution that would be functional with @cluggas suggestion.

If Functionbeat did not require a triggers list and instead dealt with any triggers assigned to the lambda you could create hooks in your service deployments which attach their log groups to functionbeat. Permissions are the only thing in the configuration that can take a wild card so they can be grouped by naming convention.