Beats: Please support AWS IAM Instance profiles!

[davidmco65]

I can't use the Functionbeat on my EC2 server because it doesn't appear to use the default credential provider chain. Meaning that I have a Role assigned to me EC2 instance that gives it permission to access the bucket specified in the configuration, but the deploy function doesn't see use it and therefore has no permissions to do anything with S3.

I'm trying to deploy the functionbeat from our EC2 server and it's not possible given this missing functionality. I've got my entire ELK stack deployment scripted except for this functionbeat.

[exekias]

I think https://github.com/elastic/beats/pull/12727 would fix this if used in Functionbeat? cc @kvch

[kvch]

Yes, I think so. But still, it needs a bit of work in Functionbeat. :)

[kaiyan-sheng]

@davidmco65 Thanks for creating this issue. When you mentioned default credential provider chain, do you mean the aws credential file under ~/.aws/credentials?

[davidmco65]

@kaiyan-sheng No, the default credential provider chain runs through a set of potential credential providers. One of which is the ~/.aws/credentials file, but others include the EC2 Instance Profile - a role that's assigned to the EC2 instance running the beat and would give the beat the permissions it needs to access the resources it needs, i.e., S3 bucket. In our environment, we literally cannot use access key id/secret key, so there is no way to configure the function beat that will allow it to install properly.

Thanks!

[jderose9]

I need to be able to specify a role to assume in a child account via a named profile present in the ~/.aws/config file. This is typically passed as --profile parameter on aws CLI commands and many other tools that interact with AWS.

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

[Thiago-Dantas]

I also need this but in Metricbeat, should I open a new issue?

[kaiyan-sheng]

@Thiago-Dantas I'm planning to use this same issue to track for Metricbeat as well! Thanks!

[jeffvestal]

I have a user who is interested in this feature so they don't have to store creds locally on the ec2 instance.
@kaiyan-sheng ping me if you need more details.
Thanks!

[kvch]

@kaiyan-sheng Are you sure this issue can be closed? The original author requested the feature for Functionbeat. However, your PR only adds support in metrcibeat and filebeat.

[kaiyan-sheng]

@kvch Good point, does Functionbeat use https://github.com/elastic/beats/blob/master/x-pack/libbeat/common/aws/credentials.go#L34 for getting credentials? I will reopen this issue for Functionbeat. Thank you!

[kvch]

No, it does not. Functionbeat depends on the AWS SDK to handle credentials. Thus, it only supports passing those via environment variables. Thank you for reopening.

[prabaharang]

If Functionbeat uses AWS SDK, what is the credential provider chain is being used. Wouldn't AWS SDK allow to use Instance profile credentials by default (in case of Default Credential Provider Chain)?

[kvch]

I guess yes, but we haven't tested it yet. Also, all of the information I was able to found on the subject is for the v1 of aws-sdk-go. However, in Beats we use aws-sdk-go-v2, so there migth be differences between the docs of v1 and reality of v2.

[kaiyan-sheng]

Hi @kvch, since this issue is related to Functionbeat, is it ok if I assign it to you instead?

[kvch]

@kaiyan-sheng I've just assinged myself. Thanks for the ping.

[s-t-ev]

@kvch Any luck with implementing this for functionbeat? Thanks!

[BrunoMCBraga]

Fellas, i come from the future and this problem still persists. Functionbeat seems to still be using AWS env variable credentials. I am currently playing with Elastic stack using one single node on as private network and i cannot afford to put my credentials there, specially when the list of permissions necessary: https://www.elastic.co/guide/en/beats/functionbeat/current/iam-permissions.html#iam-permissions-cloudwatch

Is quite big. Looks like a disaster ready to happen.