Elasticsearch: Support for AWS IAM Kubernetes service account permissions

[ErikvanDuren]

Amazon EKS recently supports IAM permissions for Kubernetes service accounts.

I would be nice to have support for service account permissions implemented in (at least) the S3 repository plugin so it is possible to create snapshots to an S3 repository without having to update access keys and tokens on a regular basis.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[DaveCTurner]

I had a quick check and this looks to be implemented by the WebIdentityTokenCredentialsProvider which is in the SDK version we're already using. The tricky bit will be to construct a test fixture that sets up an environment similar enough to EKS to show that it does integrate properly.

[chonton]

What are the security concerns with using system properties, environment variables, or profiles to store credentials? A web search reveals many opinions both for and against each method. Should Elastic be enforcing a particular security policy or should Elastic make deployers aware of the pros/cons of each method?

In Kubernetes, mapped files holding the credentials seems to be one of the recommended practices at this time. (Profile method)

[chonton]

Elasticsearch team please chime in on which methods will be acceptable.

[jim-barber-he]

For this to work this file needs changing: https://github.com/elastic/elasticsearch/blob/master/distribution/src/bin/elasticsearch-env-from-file

When I try to use IAM Roles for Service accounts it currently errors with:

ERROR: File ..data/token (target of symlink /var/run/secrets/eks.amazonaws.com/serviceaccount/token from AWS_WEB_IDENTITY_TOKEN_FILE) must have file permissions 400 or 600, but actually has: 640

The token gets created with the following permissions:

$ ls -l /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token
-rw-r----- 1 root 1001 1001 May 21 23:49 /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token

By default this file has permission 600 and owned by root:root, but when this is set:

podSecurityContext:
  fsGroup: 1001
  runAsUser: 1001

Then the group ownership of the token is changed to match fsGroup and the group read bit set so that the token can be read.
Without that change, the token wouldn't be readable, so the hard check for permission 400 or 600 enforced in the elasticsearch-env-from-file script isn't suitable for this case.

[jim-barber-he]

The distribution/src/bin/elasticsearch-env-from-file file was fixed for version 7.6.1 of ElasticSearch and so that is no longer a blocker.
I've been able to get ElasticSearch to start with IAM Roles for Service Accounts on ES 7.7.0, and the token is mounted correctly along with the environment variables injected correctly, but it still doesn't seem to be working at this point.
I'm seeing errors that contain the following when AWS access is required:

      "caused_by" : {
        "type" : "sdk_client_exception",
        "reason" : "sdk_client_exception: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/"
      }

[hamishforbes]

Has there been any progress on this issue?
I've not been able to get Elasticsearch 7.9.2 to use EKS IAM roles for S3 snaps even though the pod is correctly authenticated.

I'd very much like to not have to inject static credentials as this would be a step backwards from my current non-k8s deployment.
So far elasticsearch seems to be the only application i've found that doesn't correctly support web identities for AWS

What's the reasoning behind not just using the default AWS SDK credential chain here?

[fculpo]

Hi,

ES is also one of the last application in our infrastructure not being able to use EKS IRSA (OIDC).

ES does not tries to read EKS IRSA token mounted in the pods:

ServiceAccount has the following annotation:

Annotations:   eks.amazonaws.com/role-arn: arn:aws:iam::xxx:role/ekssa-es-snapshots

Which results in these env vars to be mounted in the pods:

Environment:                                                                                                                
AWS_ROLE_ARN: arn:aws:iam::xxx:role/ekssa-es-snapshots
AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

But ES keeps trying to use EC2 Metadata:

curl -X PUT -H 'Content-Type: application/json' "localhost:9200/_snapshot/s3_backup_bucket/backup-1" -d '{
    "include_global_state": true,
    "include_aliases": true
  }'

...

{"type":"sdk_client_exception","reason":"The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/"}}},"status":500}
...

Any way to make this working ?