Aws-parallelcluster: Importing requester-pay S3 buckets with FSx

[JiaweiZhuang]

Environment:

• AWS ParallelCluster 2.2.1
• OS: CentOS7
• Scheduler: Slurm
• Master instance type: c5n.large
• Compute instance type: c5n.18xlarge

Bug description and how to reproduce:
I am trying to import AWS Open Data (https://registry.opendata.aws/) with FSx. It works well for most public buckets (quite amazing!), but fails with buckets in requester-pay mode.

This is the specific dataset I am trying to import, managed by my research group at Harvard:
https://registry.opendata.aws/geoschem-input-data/
Our current contract is limited to requester-pay mode and probably won't change in the near term.

By setting import_path = s3://gcgrid (our data), FSx seems to have trouble importing it:

$ pcluster create fsx-requester-pay-s3
Beginning cluster creation for cluster: fsx-requester-pay-s3
Creating stack named: parallelcluster-fsx-requester-pay-s3
Status: parallelcluster-fsx-requester-pay-s3 - ROLLBACK_IN_PROGRESS
Cluster creation failed.  Failed events:
  - AWS::CloudFormation::Stack EBSCfnStack Resource creation cancelled
  - AWS::IAM::InstanceProfile RootInstanceProfile Resource creation cancelled
  - AWS::EC2::EIPAssociation AssociateEIP Resource creation cancelled
  - AWS::CloudFormation::Stack FSXSubstack Embedded stack arn:aws:cloudformation:us-east-1:753979222379:stack/parallelcluster-fsx-requester-pay-s3-FSXSubstack-1WHT3VJ4298XE/ed68b7e0-3c3f-11e9-a755-0e0d3a451244 was not successfully created: The following resource(s) failed to create: [FileSystem].

The FSx section in the config file is:

[fsx fs]
shared_dir = /fsx
storage_capacity = 14400
imported_file_chunk_size = 1024
# import_path = s3://era5-pds  # this works well
import_path = s3://gcgrid

Is it possible to get around this by tweaking the IAM role? I am currently using the default IAM settings. This is probably an edge case that is only specific to my use cases. I can also copy the data manually for now.

[sean-smith]

Hi @JiaweiZhuang

Can you grab the error from the fsx-substack? Go to the cloudformation console and click on that stack (or recreate it) and paste the error message from there.

The requester-pay bucket should work, so I'm curious if that's the issue.

[darrylsosborne]

@JiaweiZhuang - This issue you're experiencing is related to the bucket not allowing GetBucketAcl. This can be tested by running: aws s3api get-bucket-acl --bucket gcgrid
It returns "An error occurred (AccessDenied) when calling the GetBucketAcl operation: Access Denied" so the Amazon S3 data repository integration fails. We're working with these bucket owners to enable GetBucketAcl on all AWS opendata set buckets.

[JiaweiZhuang]

The FSXSubstack shows:

Logical ID - FileSystem
Status - CREATE_FAILED
Status reason - Access denied for import path. (Service: AmazonFSx; Status Code: 400; Error Code: BadRequest; Request ID: 91bc63c0-4a1c-4375-b835-46b859d17e6b)

[JiaweiZhuang]

@darrylsosborne thanks for pointing this out! We should be able to fix it on our side.

[sean-smith]

I've added a page to document the FSx issues we've seen so far. Checkout https://github.com/aws/aws-parallelcluster/wiki/FSx-Issues

Thanks for all the bug reports @JiaweiZhuang !

[JiaweiZhuang]

@darrylsosborne We modified the bucket policy as shown below, but still get AccessDenied error with aws s3api get-bucket-acl --bucket gcgrid. Would you be able to point out what is missing? Thanks!

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "reqPaysModif",
            "Action": [
                "s3:GetBucketAcl",
                "s3:ListBucket"
            ],
            "Principal": "*",
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::gcgrid"
        },
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::gcgrid/*"
        }
    ]
}

[darrylsosborne]

@JiaweiZhuang We generally recommend that data providers use this policy when opening buckets:

We generally recommend that data providers use this policy when opening buckets:

{
  "Version": "2012-10-17",
  "Id": "BUCKET_NAME-pds-policy",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "s3:List*",
        "s3:Get*"
      ],
      "Resource": [
        "arn:aws:s3:::{BUCKET_NAME}/*",
        "arn:aws:s3:::{BUCKET_NAME}"
      ]
    }
  ]
}

[jflogaya]

Hi @darrylsosborne I work with Jiawei and I'd like to ask for a more specific policy, this one seems a bit too open. It may not be terrible now, but if there are new features added in the future, having that * would allow those by default as these are rarely reviewed and I'm not really comfortable with it.
Thanks for your help!

[jflasher]

Hi @xaxis3 , we don't have any specific guidance here, but I'd recommend viewing https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html and specifically enabling the actions you're comfortable with. GetObject and ListBucket are the big ones, but we've also seen GetBucketLocation be useful and for this particular thread, GetBucketAcl is needed.

[sean-smith]

@xaxis3 and @JiaweiZhuang Does this solve your problem?

[jflogaya]

Hi @jflasher @sean-smith
I tried multiple variations of the list and get commands individually and none would produce the desired output. I ended up trying the suggested policy like so:
{ "Version": "2012-10-17", "Id": "gcgrid-pds-policy", "Statement": [ { "Effect": "Allow", "Principal": "*", "Action": [ "s3:List*", "s3:Get*" ], "Resource": [ "arn:aws:s3:::gcgrid/*", "arn:aws:s3:::gcgrid" ] } ] }
But still we get the same error message.
I'm going to leave this policy for a couple of days in case you can test and maybe run with verbosity to try and see what may be the issue here, we'd appreciate if you could tell us what we are doing wrong or what can we do to fix it. Definitely prefer to have a set of permissions that make more sense, but at this point we'd like to see something working and then we'll try to shrink the permissions to the minimum viable ones.
Thanks!
j

[demartinofra]

Hi,

sorry for the delayed answer. Unfortunately there is a known problem with FSx using requester-pay buckets which does not depend on ParallelCluster integration. There is a fix coming out in the near future but there is nothing we can do on ParallelCluster side before that.
I'm going to resolve di issue but feel free to reopen if necessary.

Thanks for your understanding,
Francesco

[JiaweiZhuang]

Unfortunately there is a known problem with FSx using requester-pay buckets which does not depend on ParallelCluster integration.

I am still interested in resolving this. Where is the right place to track this FSx issue?

[sean-smith]

@JiaweiZhuang I've brought it up with the FSx team