Aws-data-wrangler: How to load parquet file from S3 using aws profile_name?

[nivcoh]

Hi,
How can I use the code below with AWS profile_name in order to read parquet files from S3?

df = wr.s3.read_parquet(filepath, dataset=True)

Thanks

[igorborgest]

Hi @nivcoh!

You have three alternatives:

Creating a new boto3 session

my_session = boto3.Session(profile_name ="MY_PROFILE_NAME")
wr.s3.read_parquet(filepath, dataset=True, boto3_session=my_session)

Configuring the default boto3 session

boto3.setup_default_session(profile_name ="MY_PROFILE_NAME")
wr.s3.read_parquet(filepath, dataset=True)

Configuring the default boto3 session through an environment variable

$export AWS_PROFILE=MY_PROFILE_NAME

wr.s3.read_parquet(filepath, dataset=True)

[nivcoh]

Thank you @igorborgest , I need a general solution so option 1 will be the best fit.
Unfortunately, it's still not working for some reason:

import awswrangler as wr
import pandas as pd
import boto3

new_session = boto3.Session(profile_name='myProfile', region_name='us-east-1')
filepath = 's3://my_bucket/myfile.parquet'
df = wr.s3.read_parquet(filepath, dataset=True, boto3_session=new_session)

I get:
_ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden_

[igorborgest]

Now seems it is just lack of permission. Are you sure you have privileges in this bucket?

You can also test the same credentials through AWS CLI:

aws s3 ls s3://your_bucket/ --profile YOUR_PROFILE

[nivcoh]

This is becoming an interesting case:

• In CLI I get to see the content of the bucket and specific key.
• I created a test parquet and I get a different error:

InvalidArgumentValue: Object s3://my_backet/my_testFile.parquet is not under the root path (s3://my_backet/my_testFile.parquet/).

-I should mention that the original file I'm trying to open (using the example code above), was written from a different remote environment (EC2) into this bucket, and I'm trying to open it from my local env but it seem to be locked or something.
Now, I don't see any reason why it should matter if the file was written

[igorborgest]

Oh this error now is different, if you are trying to read a single file you should pass dataset=False.

or simply:

wr.s3.read_parquet(filepath)

[nivcoh]

@igorborgest 👍
This solves the issue and I can load the parquet file I created.

As for the original file (which is created by a remote process on an EC2) I keep getting this error:
_ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden_

Is it a privileges issue?

[igorborgest]

Is it a privileges issue?

Yes.

• Is this file in the same bucket of the other tested file?
• Do you also can list the original file through AWS CLI?
• Did you checked the IAM Role attached to your user/profile?
• Are this EC2, bucket and your user all belongs the same AWS account? If not, it could be lack of permissions in the file ACL.

[nivcoh]

1. yes, same bucket
2. yes I can
3. yes, it's a common one I use without a problem for reading/writing
4. The EC2 is in a different account. my user and bucket belong to a 2nd account.

[igorborgest]

Ok, so my guess is that it is a ACL issue. So basically the EC2 should add ACL permissions for the bucket owner during the upload.

[nivcoh]

good guessing!
so using cli it will be something like:
aws s3api put-object-acl --bucket awsexamplebucket --key my_file.parquet --acl bucket-owner-full-control

is there a more pythonic way? or just to pass such one?

[igorborgest]

@nivcoh you can also do the same with boto3.

[nivcoh]

Yap, I figured this out 👍
Thank you for the kind help :)