There are plenty of examples of this in the community with custom functions, but considering how checksum comparisons are handled internally by aws-cli, it seems strange that s3 sync is not available from boto. Would someone mind discussing?
MatthewRalston
All 2 comments
Duplicate of boto/boto3#358
We've made some progress along these lines. Most of the convoluted uploading logic is handled in s3transfer now, which both boto3 and the cli use. Most of the rest of the CLI code is handling things like printing status updates and converting command line arguments into concrete function calls. I would say that the checksum comparisons aren't that big a deal since there are so few circumstances where we actually run them (because multipart uploads mean etag != md5).
s3transfer has been the backend of the cli and boto3 for quite a while now, but there are some changes we want to make to it still that may end up being breaking changes (such as going multiprocess rather than just multithread). Unfortunately I can't give a timeline on when the interface will stabilize, so if you want to use it now I would recommend locking to a minor version.
JordonPhillips
on 26 Apr 2018
Thanks for the info @JordonPhillips. Very helpful.
MatthewRalston
on 1 May 2018
Related issues
stig
路
3Comments
xennygrimmato
路
4Comments
eode
路
5Comments
thehesiod
路
5Comments
FrancescoRizzi
路
5Comments
Most helpful comment
Duplicate of boto/boto3#358
We've made some progress along these lines. Most of the convoluted uploading logic is handled in s3transfer now, which both boto3 and the cli use. Most of the rest of the CLI code is handling things like printing status updates and converting command line arguments into concrete function calls. I would say that the checksum comparisons aren't that big a deal since there are so few circumstances where we actually run them (because multipart uploads mean etag != md5).
s3transfer has been the backend of the cli and boto3 for quite a while now, but there are some changes we want to make to it still that may end up being breaking changes (such as going multiprocess rather than just multithread). Unfortunately I can't give a timeline on when the interface will stabilize, so if you want to use it now I would recommend locking to a minor version.