Smart_open: Migrate to boto3

[piskvorky]

Check if boto3 is production-ready and migrate all code from boto to boto3.

[asieira]

I would suggest that even if boto3 is supported, existing implementation using boto 2 is kept as well.

There are many things boto3 hasn't implemented yet. To give one example, the lack of explicit proxy support would prevent me from continuing to use smart_open if it went boto3-only. See #35 for supporting evidence. :)

[piskvorky]

Thanks @asieira , that's the kind of practical feedback we need!

I don't have experience with boto3, and certainly wouldn't want to inconvenience users by a hasty switch.

[brunoalano]

+1

[vinayan3]

+1. I'm having issues with Python 3.5 and Boto 2.39. From what I've read Python 3x is better with boto3

[tmylk]

@vinayan3 What kind of issues are you having? Would be great to check they are resolved in boto3.

[tmylk]

Update on Boto3 : Proxy support still not implemented in https://github.com/boto/boto3/issues/338

[tmylk]

Our test dependeny moto has moved to boto3 unfortunately in ver 1.0.0
We still can't migrate though because of proxy support

• The normal @mock_ decorators will no longer work with boto. It is suggested that you upgrade to boto3 or use the standalone-server mode. If you would still like to use boto, you must use the @mock__deprecated decorators which will be removed in a future release.

[combinatorist]

+1

• I came in at boto3 and I just started using this library.
• Currently, I use this library by string interpolating keys into paths
  
  
  
  • e.g. smart_open('s3://mybucket/' + key)
  
  
• but I really wish I could use boto3 objects (particularly for the advertised smart_open.s3_iter_bucket function)

[tim-schier-frontier]

I'm using moto and would like to move to moto>=1, but smart_open is holding me back due to boto3 vs boto support.
Even using @mock_s3_deprecated for my other moto usage is causing issues now with moto 1.0.1. It looks like a moto bug, but only a bug in the deprecated support for boto.

So yeah, boto3 momentum is building. Oh and it looks like proxy support has been added to boto3 now (ticket @tmylk linked to is closed as implemented).

[piskvorky]

@tim-schier-frontier can you help with the migration, open a PR?

[tim-schier-frontier]

@piskvorky great follow up - put money where my mouth is :)
I am best placed to help with testing/trialing PRs, and also probably chipping in for funding the move (from my employer).
Love the product, hope to help out.

[mpenkov]

I think boto3 now supports proxies: https://github.com/boto/boto3/issues/338

The S3 component is already boto3-based. The tests are still boto-based, however. @menshikh-iv What should we do about this?

[menshikh-iv]

@mpenkov probably migrate tests to boto3 & close this issue

[mpenkov]

We should also upgrade moto to the most recent version.

[inksink]

I have read the PR 164 which will migrate tests to boto3.
But there are many codes used boto in smart_open_lib.py. Do we have such a plan to migrate smart_open_lib to boto3?
I wish to reduce unnecessary external dependencies in my project.

[menshikh-iv]

@inksink yes, we want to migrate all to boto3
CC: @mpenkov

[carolingfish]

I noticed that commit on this has been merged. Any estimate on when the new package including the commit will be released?

[menshikh-iv]

@carolingfish I think I'll release it in the current month

[bkapoorUF]

Any update on when this will be released?

[menshikh-iv]

@bkapoorUF already released as smart_open==1.6.0