Jaeger: Support AWS Elasticsearch IAM authentication by signing requests

[julianvmodesto]

I'd like to use AWS Elasticsearch as my datastore, using AWS credentials to authenticate. This requires signing requests, so we'd probably need to use the AWS SDK here.

[yurishkuro]

Can you provide the link to signing requirements?

[julianvmodesto]

olivere/elastic has helpful examples:

• https://github.com/olivere/elastic/wiki/Using-with-AWS-Elasticsearch-Service

[mabn]

FYI - I'm using https://github.com/abutaha/aws-es-proxy
But it has problems at higher throughputs (didn't investigate it yet) and does not work with elasticsearch-hadoop (used by https://github.com/jaegertracing/spark-dependencies)

[yurishkuro]

I assume the proxy is stateless so can be scaled up with a load balancer.

[mabn]

AWS ES now supports VPC endpoints: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html - It's a much better option than publicly available ES and signing requests.

[yurishkuro]

Indeed. I am going to close this.

[discordianfish]

I think signing requests would make sense even with VPC endpoints. One still might want to protect against tampering with the requests from within the vpc.

[discordianfish]

Actually not even sure you can disable request signing even in a VPC. Anyone doing this?

[mabn]

When using VPC endpoints you assign a security group to ES. You can control access with security group settings. There is no request signing when you place ES in a VPC (I'm not sure if it's possible to have both VPC and request signing).

[discordianfish]

@mabn Ah thanks for clarifying. Didn't know that and will try it.

[wesleyk]

Hey folks. I was wondering if IAM-based authn for ES was still something that is on the table for Jaeger. Reading back on this conversation it seemed to take a different direction and focus on signed vs VPC endpoints.

cc @black-adder

[wesleyk]

@mabn Any chance you had some thoughts on this based on the previous conversation here? Appreciate any help you can provide!

[wesleyk]

@black-adder et al, any chance I could get an update on this subject?

[yurishkuro]

I am pretty sure @black-adder doesn't have time to work on this. Need another volunteer.

[wesleyk]

Sounds good, just wanted to check that this (IAM-based authn for ES) is something y'all would like to see. I can take a stab at it; any pointers on where to start?

Also perhaps a better name for the issue would now be: "Support AWS Elasticsearch IAM authentication". Thanks @yurishkuro!

[yurishkuro]

any pointers on where to start?

something like this: https://github.com/jaegertracing/jaeger/pull/686/files

[wesleyk]

Fantastic, thank you!

[wesleyk]

So I had a misunderstanding above; signing and using IAM based auth are one and the same. Furthermore, this is possible even with VPC-based access to ESS. More details on Amazon's docs here: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html.

The go elastic search client that jaeger uses has some docs on signing requests that make the integration seem fairly reasonable: https://github.com/olivere/elastic/wiki/Using-with-AWS-Elasticsearch-Service. I'm taking a stab at integrating with the AWS go SDKs now.

[ovigio]

@wesleyk
Would it be possible to also have the AWS credentials be retrieved from the ec2 instance IAM role in addition to providing them through configuration like you currently have?

Creating a session can provide credentials. Something like this:
sess, err := session.NewSession(&aws.Config{Region: aws.String(c.AwsConfig.Region)})

Then retrieve the credentials:
credsValue, err := sess.Config.Credentials.Get()

Thanks,

John

[wesleyk]

Ah @jmwaniki good idea; in fact I think that could drastically reduce the surface area of AWS-details that are encoded in the configuration API. We would only need to accept a region flag, and if such flag is set then we would provide IAM auth.

A downside there is that an AWS region flag doesn't scream "this enables IAM". We could make that clear in the docs, or maybe we name the flag something like... .aws.iam_auth_region or .aws.iam_auth.region. Thoughts?

[wesleyk]

@black-adder with that change, do you feel the API surface area is kept minimal enough? I think we need at least one flag for the client to indicate that they actually do want IAM auth enabled; inferring too much based on what's available in the environment seems potentially troublesome and not backwards compatible with existing clients.

[wesleyk]

Ah, but you can actually specify the region in that config file as well! So really, we can just have one flag that specifies that the client wants IAM auth enabled.

So we could have a aws.enable_iam, and then create a session as John described above.

I like that! If that sounds good to y'all I can update the PR with such changes.

cc @black-adder @jmwaniki

[ovigio]

@wesleyk
That sounds good to me, having a flag that specifies client wants IAM auth enabled aws.enable_iam and the other flag for specifying the region .aws.iam_auth_region.

[wesleyk]

Great! I think we don't even need the region because clients can specify that view a local config file.

[black-adder]

I'm good with just exposing aws.enable_iam, can you make the changes so I can review?

[wesleyk]

Fantastic! Working on it.

[wesleyk]

@black-adder PR updated!

[ovigio]

@wesleyk
Just realized the credentials retrieved from the session have a token, that token will expire eventually.

The credential provider determines when the credentials will expire:
https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/

The token however is optional for aws static credentials, omitting the token from the static credentials might solve the expiry issue.

Static credentials provider won't expire:
https://github.com/aws/aws-sdk-go/blob/master/aws/credentials/static_provider.go#L25

Similar to what you had before, with values from the session credentials:
credsValue, err := sess.Config.Credentials.Get()

creds := credentials.NewStaticCredentials( credsValue.AccessKeyID, credsValue.SecretAccessKey, "", )

[ovigio]

@wesleyk
Using session credentials to create static credentials like I commented above fails to connect to elasticsearch. I tried it and the session token is still required to connect to elasticsearch. In any case, wanted to note that session credentials would need to be refreshed before expiring.

[jpkrohling]

Looks like #1139 provides a fix for this.