Aws-data-wrangler: Ability to pass through endpoint_urls to AWS services

[martinSpears-ECS]

A reproducible test case or series of steps:

• Spin up standard ml.t3.medium AWS SageMaker notebook
• Pip install awswrangler (using AWS CodeArtifact)
• Import boto3 and awswraggler
• Create an STS client using regional url:
  sts_client = boto3.client('sts', endpoint_url="https://sts.eu-west-1.amazonaws.com)
• Assume role in another account
  sts_response = sts_client.assume_role(RoleArn='arn:aws:iam:0123456789:role/example-role, RoleSessionName='test'
• Create boto3 Session as per 002 - Sessions.ipynb
  
  my_session = boto3.Session(region_name='eu-west-1',aws_access_key_id=sts_response.get('Credentials',{}).get('AccessKeyId'),aws_secret_access_key_id=sts_response.get('Credentials',{}).get('SecretAccessKey'),aws_session_token=sts_response.get('Credentials',{}).get('SessionToken'))
• Run awswraggler athena _read_sql_query_ using boto3 session
  awswraggler.athena.read_sql_query("select * from database.table limit 10", database="database", ctas_approach=False, boto3_session=my_session)

Outcome

The request times out with an error saying it can not connect to _https://sts.amazonaws.com_. This is expected as the global STS endpoint will not resolve due to no internet access and VPC endpoint should be used.

Desired Outcome

Able to run the query against Amazon Athena using VPC endpoint without error.

Version of code

1.9.5

Any modifications you've made relevant to the bug

No

Anything unusual about your environment or deployment

No internet access, STS and Amazon Athena VPC endpoints are up and working. Tested connectivity using VPC endpoints using native boto3 commands

Further information

Looking in to botocore and boto3 sessions, they do not have a configuration item to force the package to use regional endpoints, this can only be set on the client.

Possible change

Have the ability to set the endpoint_url the appropriate clients manually as a first step. A next step could be the ability to tell the package to use regional endpoint_urls which will default to standard AWS VPC endpoints for the service.

[igorborgest]

Thanks @martinSpears-ECS, we will add VPC endpoints support in our next minor release (1.10.0).

[igorborgest]

Hi @martinSpears-ECS

I've just added the related endpoints to our global configs, so you can configure it programmatically:

wr.config.sts_endpoint_url = YOUR_ENDPOINT

OR you can define it through the environment variables:

export WR_STS_ENDPOINT_URL=YOUR_ENDPOINT

All endpoints available are:

Do you mind to install it directly from our repository and check if it is everything fine for your use case before the official release?
The idea is to publish it in the version 1.10.0 this weekend.

pip install git+https://github.com/awslabs/aws-data-wrangler.git --no-use-pep517

P.S. For more details check the commit above and/or our Global Config tutorial.

[igorborgest]

Released on 1.10.0