Aws-cli: aws s3 sync does not synchronize s3 folder structure locally
The aws s3 sync does not fully synchronize the S3 folder structure locally even if I use it with --delete or --recursive arguments:
aws --version
aws-cli/1.4.3 Python/2.7.6 Linux/3.13.0-35-generic
$ aws s3 ls s3://s3.testbucket
$ aws s3 ls s3://s3.testbucket/
$ mkdir s3.testfolder
$ mkdir s3.testfolder/test1
$ aws s3 sync ./s3.testfolder s3://s3.testbucket/
$ aws s3 ls s3://s3.testbucket/
$ touch s3.testfolder/test1/1
$ aws s3 sync ./s3.testfolder/ s3://s3.testbucket/
upload: s3.testfolder/test1/1 to s3://s3.testbucket/test1/1
$ aws s3 sync ./s3.testfolder s3://s3.testbucket/
$ mkdir ./s3.testfolder/test-to-delete
$ aws s3 sync s3://s3.testbucket/ ./s3.testfolder/ --delete --recursive
$ aws s3 sync s3://s3.testbucket/ ./s3.testfolder/ --delete
$ ls -lah ./s3.testfolder/
total 60K
drwxrwxr-x 4 tobi tobi 4,0K szept 12 15:24 .
drwx------ 71 tobi tobi 44K szept 12 15:22 ..
drwxrwxr-x 2 tobi tobi 4,0K szept 12 15:23 test1
drwxrwxr-x 2 tobi tobi 4,0K szept 12 15:24 test-to-delete
$ aws s3 ls s3://s3.testbucket/
PRE test1/
tatobi
All 100 comments
This behavior is known. The reason why the sync command behaves this way is that s3 does not physically use directories. There are only buckets and objects. Objects have prefixes that act like directories, but s3 does not designate a specific physical object to be a directory.
Therefore, when the syncing occurs, only files are transferred to s3 because s3 does not have physical directories. So when you try to sync up empty directories, nothing is uploaded because there are no files in them. Once you put items in the directory, then the file (with the prefix representing the directory) will be uploaded.
kyleknap
on 12 Sep 2014
Thank you Kyle, it is clear. I know how S3 stores files, but sometimes we need the same directory structure in sevaral places even if there are empty ones or remove from if we do not need anymore.
A good example if you have complex directory structure with a lot of contents locally than you synced to S3. After that an automated mechanism sync this structure periodically to several running instances. You keep up-to date (delete) most of the content from S3 then the automatism re-sync to the places where you used before. Unfortunately you will find the original complex directory structure remains forever on sync targets which may cause confusion if you want to check it or your program try to use this empty folders because of you need always the same everywhere. Moreover the people who use it with --delete options maybe used the "rsync" equivalent before on Linux which keeps the folders synced so counts on the same operation.
I think it would be not hard to implement a switch or option for aws tool to detect somehow if an S3 object is a file or folder (list, size, etc..) and create/delete them locally or in an S3 bucket (e.g. list(bucket.list("", "/"))?
tatobi
on 12 Sep 2014
That makes sense. Will look into adding a feature for it.
kyleknap
on 15 Sep 2014
This would be very useful for our situation as well. If it were added as an option (--sync-empty-directories) people could choose to use it when needed.
ururk
on 21 Nov 2014
+1 Need this feature very badly
xbeta
on 15 Jan 2015
+1. Would like to use it.
w32-blaster
on 7 Mar 2015
+1
danielwinter83
on 31 Mar 2015
I also was surprised by this behavior, given that it is called "sync".
I can work around this in my particular use case, but future users could be spared the pain :)
unixmonkey
on 6 May 2015
+1 on being able to sync directory structure! If you delete a folder it only removes the content, but it leaves the folder behind...
athurber
on 6 May 2015
+1. I have the same needs.
zioalex
on 9 Jun 2015
+1 - surprised that hasn't been implemented yet. Sure, in my case it doesn't matter too much, and I can work around it (or just use placeholder files when creating structures), but it would be a benefit to just have it supported by either s3 sync or s3 cp.
benjaminsherwood
on 1 Jul 2015
+1
s3cmd sync does keep the folder structure but therefore it has some issues when granting access while synching so one needs to run another s3cmd setacl --recursive afterwards…
cauboy
on 20 Jul 2015
+1
ntmggr
on 27 Jul 2015
+1
ghost
on 30 Jul 2015
+1
asdesilva
on 13 Aug 2015
Thanks for the feedback everyone. I think the best option I've seen is to add a --sync-empty-directories option. Let's do that.
jamesls
on 13 Aug 2015
@jamesls I'm expecting somewhat like rsync functionalities, but s3 as an object storage is definitely not the same though.
xbeta
on 14 Aug 2015
+1
makeittotop
on 1 Sep 2015
+1
mradochonski
on 5 Oct 2015
Any timeline for this feature?
mradochonski
on 5 Oct 2015
As a temporary workaround I added an empty .s3keep file to the empty directories and it works for me. This is a hack I usually use to trick git to not treat empty directories as empty ones :)
swarupdonepudi
on 15 Oct 2015
Will this also allow to "remove/delete" empty directories on S3 ?
lcasey001
on 29 Oct 2015
+1
arcadas
on 23 Nov 2015
+1
andrefelipe
on 1 Dec 2015
+1
srirachapills
on 18 Jan 2016
+1
mputilin
on 5 Feb 2016
+1
ixtli
on 19 Feb 2016
+1
asyavuz
on 21 Feb 2016
+1
matteomelani
on 31 Mar 2016
+1
Darkyj
on 6 Apr 2016
+1
dijeesh
on 15 Apr 2016
+1
ghost
on 1 May 2016
+1
xam7247
on 13 May 2016
+1
it-marmalade
on 17 May 2016
+1
steven-klein
on 26 May 2016
+1
moradai
on 7 Jun 2016
+1
miggy282
on 7 Jun 2016
+1
_Makes lot of sense during data migrations to s3._
ritxos
on 15 Jun 2016
+1
thoralf-gutierrez
on 23 Jun 2016
+1 Just got smashed by this... Arg....
bdo-eow
on 5 Jul 2016
+1
murraybrad13
on 7 Jul 2016
+10
It's possible to work around this with dummy files but it would be cleaner if there would be an option to force an empty prefix to synchronize.
quater
on 10 Jul 2016
+1. Use case: backing up an svn repository.
More generally:
aws s3 sync thing
aws s3 sync
I expected thing_copy to match thing exactly.
ghost
on 11 Jul 2016
+1
hesu
on 13 Jul 2016
+1
jumping
on 25 Jul 2016
+1
ghost
on 18 Aug 2016
+1 need to delete empty directories
carhensi
on 26 Aug 2016
How's the progress of adding this option --sync-empty-directories?
any feedback from AWS Team?
Thanks.
hierony94
on 2 Sep 2016
+1 would be a very useful feature for a very useful tool
jimbocoder
on 4 Sep 2016
+1
SirR4T
on 14 Sep 2016
+1 (I too wish that this feature was implemented and wish that Github.com had a StackOverflow.com like interface for "voting" on issues/features).
Dulani
on 14 Oct 2016
+1
MatthewVance
on 10 Nov 2016
+1
VaughnLi
on 15 Nov 2016
+1
rrlino
on 2 Dec 2016
+1
realsby
on 16 Dec 2016
+1
amitlalani
on 12 Jan 2017
2+ years later and it still hasn't happened.. ? will it ever? =/
vesper8
on 31 Jan 2017
+1
MathieuMorrissette
on 9 Feb 2017
+1
sptessier
on 24 Feb 2017
+1
bobgon
on 25 Feb 2017
+1
giseongeom
on 6 Mar 2017
+1000
barbarello
on 11 Mar 2017
+1
ekstro
on 23 Mar 2017
I did some digging on how this could be implemented. All s3 commands eventually end up using TransferManager from the s3transfer library. (referenced here)
To support adding a folder with PutObject we can send an empty string in the Body param. I don't know if this is officially supported though. I implemented this here:
https://github.com/svleeuwen/s3transfer/commit/b7d3745a995a75c5262950bb798c8c57e481c2b3
I'd like some feedback on this from a maintainer before continuing.
svleeuwen
on 28 Mar 2017
+1
brooksandrew
on 14 Apr 2017
my solution was to mount my bucket with s3fs and rsync from the s3 mount to a directory in my home directory.
thenetimp
on 14 Apr 2017
+1
mohanen
on 3 May 2017
+1 really need this ...
w0203j
on 20 May 2017
+1
Open since 2014? Really? :unamused:
nethershaw
on 30 Jun 2017
+1
nickgardos
on 3 Jul 2017
+1
harishpendela
on 12 Jul 2017
+1
opn1
on 9 Aug 2017
+1
mickrussom
on 10 Aug 2017
+1
thomassngdata
on 7 Sep 2017
+1
aaronovz1
on 25 Sep 2017
+1
sammerry
on 26 Sep 2017
+1
PD-123
on 27 Sep 2017
+1
amnobis
on 27 Sep 2017
@thenetimp This solution is fine for small buckets. We are using a bucket with more than 15TB. S3FS gets horrible slow with bigger buckets.
AukeTembrink
on 13 Oct 2017
+1
muchaohyy
on 8 Nov 2017
Good Morning!
We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.
This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.
As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.
We’ve imported existing feature requests from GitHub - Search for this issue there!
And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.
GitHub will remain the channel for reporting bugs.
Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface
-The AWS SDKs & Tools Team
This entry can specifically be found on UserVoice at : https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168436-aws-s3-sync-does-not-synchronize-s3-folder-structu
ASayre
on 6 Feb 2018
great job Andre, close an issue and give us a link that isn't related to the issue. Of all the useless posts
thenetimp
on 6 Feb 2018
The generic boilerplate is disappointing. I think the line between feature request and a bug report can be pretty blurry. To save people some searching the UserVoice post for this feature request is available at https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168436-aws-s3-sync-does-not-synchronize-s3-folder-structu
skwashd
on 6 Feb 2018
Based on community feedback, we have decided to return feature requests to GitHub issues.
jamesls
on 7 Apr 2018
+1
mainframe
on 5 Jun 2018
+1
kipkoan
on 17 Jun 2018
+1
thugbeatz
on 20 Jun 2018
+1
tinyheero
on 23 Jul 2018
+1
shankar-moeng
on 26 Jul 2018
+1
Anvesh-Gurudu-Bose
on 29 Aug 2018
+1. Would be a nice feature to add.
mukteshkrmishra
on 7 Sep 2018
+1
khadrawy
on 17 Sep 2018
+1
schmite
on 6 Oct 2018
Same issue
awscli==1.16.74
jqadev
on 14 Dec 2018
+1
foxel
on 12 Apr 2019
-1
jriffel
on 15 May 2019
The aws s3 sync command is already recursive, so there is no need for a recursive option, In addition the sync command only copies things that don't already exist on the destination. If you point to a folder it will recursively sync everything inside that doesn't already exist on your target destination. This is different then the aws s3 cp command. The cp command copies whatever you tell it to, regardless of it it already exists on the target. The cp/ mv/ rb command takes a --recursive option for recursively copying/ moving/ deleting folders/files. Thanks
3ggaurav
on 27 May 2019
@3ggaurav the issue is originally from 2014 when I recall sync had a --recursive option.
Additionally if you're going to quote a stack overflow answer verbatim, it's generally good practice to reference/give credit to it.
k3ypad
on 6 Jun 2019
Still no progress on this ?
aryus96
on 8 Jun 2020
+1
Degraund
on 18 Nov 2020
Related issues
hapx101
·
3Comments
matt-sexton
·
3Comments
ypant
·
3Comments
ronaldpetty
·
3Comments
975204
·
3Comments
Most helpful comment
Based on community feedback, we have decided to return feature requests to GitHub issues.