Aws-cli: s3 sync does not preserve permissions across buckets

[mpchlets]

When I sync across buckets with a command of:

aws s3 sync s3://bucket1 s3://bucket2

I would expect the files in bucket2 to have the same permissions as the files in bucket1 by default as it is a "sync"

Instead they only have the default permissions.

[kyleknap]

Do all of the files have the same permissions? Because for now you could run the sync command with --acl to set permissions. However, currently running sync command with no other optional arguments does not look for permissions and ensure when that the object is transferred, it has the same permissions as the original object. We would have to add an additional argument/feature too handle it.

[mpchlets]

No - if all the files had the same permissions I could use the --acl flag.

This feature is really a must have of a sync mechanism, particularly when you have thousands of files with different permissions and want a true sync across buckets not just a copy of files.

Do you know of another way to get permissions on a per file basis, so I can script this easily enough?

Thanks for helping.

[kyleknap]

Yeah if you run an aws s3api get-object-acl on each of the objects. You could get acl's. Here are the docs for the command.
http://docs.aws.amazon.com/cli/latest/reference/s3api/get-object-acl.html

[mpchlets]

OK - thanks that will work, but this feature would still be greatly appreciated in the sync, if not by default, --preserve-s3-acl flag would be just fine.

[kyleknap]

No problem. Will consider the feature. I believe the reason that the reason it has not been implemented is because the operation itself may be costly. To determine the objects in a s3 bucket a call to ListObjects is made however this does not return the acls for each object. In turn, we would have to go back and run a GetObjectAcl on every object listed, which could take a while if there are thousands of items in the bucket.

[mpchlets]

As it did take some time for me to discover that the permissions were not synced over, perhaps updating the documentations of s3 to s3 sync to mention that only the data not the permissions are synchronized would save someone else time in the future.

Thanks again

[Chili-Man]

any updates on this ?

[nikmolnar]

I added a pull request (#1535) to address this (also other S3-to-S3 transfers, like copy) a few weeks back. Any chance of getting it considered for inclusion?

[ultrahaus]

Hi, I know it's an old thread, but we solved this through one sync by permission type. Bottom line: you can filter your sync by key name with --exclude and --include, also you can specify the ACL of each sync... So, if you have a pattern of keys with you need to set different permissions, run sync command many times you need with different ACLs options. It's far from ideal, but works!

[jufemaiz]

Any thoughts from the AWS team?

[mw44118]

I am pushing my git repositories to S3 just for backups. And then when I sync them back down, all sorts of permissions have changed.

All my executable bin files are no longer executable, and git status squawks about it.

[jayannah]

+1

[srikiraju]

+1

[DCarper]

+1

[jmoratilla]

+1

[felipe-vvoosh]

+1

[Luffles]

+1

[winstonwolff]

+1

[lithiumFlower]

+1

[david-crowdstreet]

+1

[ElliottJPerry]

+1

[JerryArns]

+1, Yes I had my buckets named with a period, had to change to be compatiable for https through cloud front. Synced to new correctly named buckets. Cant access files. Read about permissions. Object level permissions exist. No command to change all object level permissions in a bucket. Wondering how to proceed. The permissions through the aws GUI actaully look the same from both buckets but when looking up the the Access Control List through the cli I see the original bucket had an extra grantee which was not copied over.
{
"Grantee": {
"Type": "Group",
"URI": "http://acs.amazonaws.com/groups/global/AllUsers"
},
"Permission": "READ"
}

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

This entry can specifically be found on UserVoice at: https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168325-s3-sync-does-not-preserve-permissions-across-bucke

[nchammas]

@ASayre - This might be the wrong place to have this discussion, but could you elaborate on why Amazon have awkwardly split the discussions on the AWS CLI across GitHub and UserVoice?

Feature requests seem appropriate here on GitHub. It's relatively easy to filter to just the feature requests, and even sort by the most popular ones:

GitHub also makes it easy to share code snippets or refer to other issues (like #1060, which is related to this one), and a large portion of the AWS CLI user base are already active here on GitHub.

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[bmdelaune]

Anyone come up with a decent solution to get this done? Much needed function.

[david-crowdstreet]

The above answer forces * to be public-read. That does not preserve permissions.

[yeah007]

What's the progress? I also encounter this issue. Now I just use sync --include/exclude to resolve this issue.

[mw44118]

Does anybody want to pay me to work on this?

[marceloagil]

4 years and yet this nightmare continues. I have 10+ millions of files and all with permissions lost during copy. How is possible such a simple thing as keeping permissions happen?

[tsu-shiuan]

+1

[aaronjensen]

For anyone else coming here, I was able to use this to copy one bucket to another and keep ACLs: https://github.com/cobbzilla/s3s3mirror/tree/2.1-stable

It's also significantly faster.

[dougmorin]

For me, i'm using s3 currently to back up my working folders. These include git repo's, and my repo's keep freaking out after all of the permission changes.

For me the solution ended up being to find a way to reset just the permissions per repo. Until AWS fixes their junk, this is the solution for me.

git config --global --add alias.permission-reset '!git diff -p -R --no-color | grep -E "^(diff|(old|new) mode)" --color=never | git apply'

I found the solution here:

https://stackoverflow.com/questions/2517339/how-to-recover-the-file-permissions-to-what-git-thinks-the-file-should-be

[terminator9999]

Hi,

I've encountered the same issue - I had a bucket with tons of objects, while some of them should be publicly accessible. I had to copy entire bucket to another one while preserving the ACLs and of course, setting up the ACLs manually would take me a hell of time.

I've made this simple script in python which copies the objects from one bucket to another and also sets up the ACLs for it.

Feel free to take a look:
https://github.com/terminator9999/aws-s3-bucket-copy/