Smart_open: Missing Support for Assumed IAM Roles (aws_session_token credentials)

Created on 14 Jun 2018 · 5Comments · Source: RaRe-Technologies/smart_open

I was very excited to find this library, as I needed a solution streaming large datasets to S3.
Unfortunately, it looks like it can't support credentials from an assumed IAM role.

From what I can tell, the only way to configure AWS credentials is by embedding the credentials in the S3 URI:

s3://my_aws_key_id:key_secret@my_bucket/lines.txt

But in order to use an assumed IAM role, I need the access_key_id, the secret_access_key and a session_token.

It would make more sense to me if I could pass in a boto3 session _directly_ to smart open. Boto3 allows a number different methods for configuring credentials, so I would rather use an existing sessions, than try to rewrite the wheel in this library.

If you can give me some direction, I can look into opening a PR. Otherwise, my options are to fork, or to roll my own streaming wrapper...

enhancement medium difficult wishlist

Source

eschwartz

👍2

Most helpful comment

So I'm sorry -- I apparently didn't look at existing PRs, before opening this issues.
Looks like we have a couple PRs working on this already: #149 and #130. I'll see if I can help shepherd those through.

eschwartz on 14 Jun 2018

👍2

All 5 comments

A PR will be great, as that functionality sounds useful. We already came across such request several times, so the demand is clearly there and it's well scoped.

There was a related PR recently, #194 -- feel free to use that as the starting point.

piskvorky on 14 Jun 2018

👍1

Great -- and thanks for responding so quickly.

I'm thinking that smart_open() will accept a boto3_session parameter. Does that sound like a good approach to you?

Looking at #194 - we actually _will_ need server side encryption too, so I may have to help get that one though as well. Any idea why that PR was closed by the author?

eschwartz on 14 Jun 2018