Elasticsearch: Cloud-AWS: S3 Repository Across-Account fails verification.

[JamesBromberger]

Elasticsearch version: 2.2.0

JVM version: 1.8.0

OS version: Amazon Linux 2015.09.02

Description of the problem including expected versus actual behavior: Using Cloud-AWS Plugin to define a repository that is across AWS accounts for escrow backps/snapshots

Steps to reproduce:

1. Create two AWS accounts, A for our primary workload (ElasticSearch), and B for our escrow copy (think: security team)
2. Create a bucket in B (security-bucket), with an S3 Bucket Policy that permits account A to write to a prefix in this bucket, with a condition that the submitter (in A) set the ACL "bucket-owner-full-control'. Also permit account A to list and get objects from this prefix, but not DELETE. (NB: does permit overwrite, but S3Versioning can help here - another topic)
3. In account A, create an IAM Role for EC2 instance that permits access to B's security-bucket, as well as the normal EC2 discovery requirements.
4. Launch an EC2 instance running ElasticSearch in this Role.
5. Configure the ElasticSearch instance to have a repository of B's security-bucket

Provide logs (if relevant):
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node"}],"type":"repository_verification_exception","reason":"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object XXX/Dev/Elasticsearch/tests-DJJ3cNL_ScqiNn6P-LQOAw/master.dat-temp","caused_by":{"type":"amazon_s3_exception","reason":"Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CBD309137D130093)"}}},"status":500}

Describe the feature:
I am using ES 2.2.0 , and setting up a cross account trust where the receiving bucket (in another AWS account0 requires "bucket-owner-full-control", Using AWS CLi confirms I can put objects with the canned ACl into the prefix I have reserved for this; I can list the objects back, and I can read the content back.

However, when trying to set up a backup location in elasticsearch, it fails the verification. My config is:
{ "type": "s3", "settings": { "bucket": "XXX-logs", "region": "ap-southeast-2", "server_side_encryption": "True", "base_path": "XXX/Dev/Elasticsearch/", "buffer_size": "100mb", "canned_acl": "bucket-owner-full-control" } }

However when submitting this I get:
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node"}],"type":"repository_verification_exception","reason":"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object XXX/Dev/Elasticsearch/tests-DJJ3cNL_ScqiNn6P-LQOAw/master.dat-temp","caused_by":{"type":"amazon_s3_exception","reason":"Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CBD309137D130093)"}}},"status":500}

Which from the S3 logs on the target S3 bucket I see a 403 AccessDenied. The only condition that is required by S3 in my case is the bucket-owner-full-control ACL. It appears that the check in index/snapshots/blobstore/BlobStoreIndexShardRepository.java calling org.elasticsearch.repositories.blobstore.BlobStoreRepository.testBlobPrefix() may not have the canned ACL being added, and thus fails to get set up.

This maybe somewhere around startVerification() in repositories/blobstore/BlobStoreRepository.java.

In my use case, the target bucket is to be write once (no deletes; escrow copy of the backup, so the corresponding "ensure moves are supported" is also likely to fail....

[clintongormley]

@dadoonet could you take a look at this please?

[clintongormley]

or @xuzha :)

[xuzha]

np @clintongormley I will verify this asap :-)

[xuzha]

Thanks @JamesBromberger for reporting the issue.

I went through the code, looks like it do have the canned acl. I'm a bit confused here, I saw you have two accounts here, looks like your Elasticsearch is using account A, and it is trying to use a bucket created by account B?

Please correct me if I'm missing something here. Based on the errors msg, ES complains that it doesn't have the permission to write to the bucket. And I think the feature in https://github.com/elastic/elasticsearch/pull/14297 is for specifying a canned in your request when creating a resource. For example, if you set log-delivery-write, it would grant write permission to the Amazon S3 LogDelivery group.

[jlintz]

I'm seeing this on ES 2.4.0, I get an access denied but the repository is still created and I'm able to snapshot to it successfully. Using the IAM role policy listed in the docs

[ppcharli]

I have the same problem on ES 2.4.1, I get an access denied but the repository is still created.
I'm using Minio Server

[jlintz]

The issue for me ended up being that the IAM policy needs delete access to test access to the bucket. I had a blanket policy in place that didn't allow delete's on S3 buckets that was causing this issue. Once I removed that, things worked as expected.

[dadoonet]

No further feedback on this one.
I'm closing but feel free to reopen and/or add comments if any.

[mikerev]

Elevating IAM permissions for delete rights resolved this issue for me.