Aws-cli: s3 sync errors with "file does not exist" for file in exclude path

[simonvanderveldt]

This seems to be similar to #1082

We run s3 sync using the following command:

s3 sync --delete --exclude '*' --include 'someservice@*.service' . s3://<bucket>

This fails with the following error message:

upload: ./[email protected] to s3://<bucket>/[email protected]
Completed 1 part(s) with ... file(s) remaining
upload: ./[email protected] to s3://<bucket>/[email protected]
Completed 2 part(s) with ... file(s) remaining
upload: ./[email protected] to s3://<bucket>/[email protected]
Completed 3 part(s) with ... file(s) remaining
Completed 3 part(s) with ... file(s) remaining
warning: Skipping file /var/jenkins_home/jobs/someservice/workspace/node_modules/mocha/node_modules/.bin/supports-color. File does not exist.

Since we're explicitly excluding everything (which means also the node_modules directory) it seems like s3 sync doesn't actually honor this exclusion in that it still tries to access all files below the current directory even though they are excluded.

[JordonPhillips]

Is the transfer still going as expected? Are any files being uploaded that shouldn't be? I agree that going through excluded files is silly, but if something is getting uploaded improperly then that's a big problem.

[simonvanderveldt]

@JordonPhillips Sorry for the delay, totally missed the notification.

The ignored files are not actually copied over, only the files that we explicitly include.
If I had to guess it seems like s3 sync exits with a non-zero exit code because of the warning it shows, which probably shouldn't be the case.
And that warning again doesn't really make sense because the file that the warning refers to was explicitly ignored by us upfront.

[mcblair]

I am experiencing this as well - warnings for explicitly excluded files.

This is problematic because s3 sync is exiting with a non-zero exit code.

When automating s3 sync like this: Simple S3 Resource for Concourse CI

We exclude everything and only include the directories we want to sync. The result is that the expected files are uploaded but because of the non-zero exit code - the Concourse CI job using the above resource fails.

[geerlingguy]

Even if running with the --quiet option, the exit code is 2, and my CI fails (unless I explicitly allow the fail).

Same problem, seeing a bunch of these when run without --quiet, even if they're all inside directories being explicitly --excluded:

warning: Skipping file /path/to/unreadable/file. File/Directory is not readable.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[simonvanderveldt]

@ASayre Thanks for the heads up

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

Free feedback: That's pretty annoying, another site to register and check to be able to discuss/create feedback/bugs.
We have a nice central place here on GitHub, right next to the sources, where one can discuss/create feedback/bugs on this and many other projects. I only have to register at one place to be able to give feedback/create bugs, access and update/fix the code and get notifications in case something I'm interested in changed. win-win-win.
I'd have to be very involved with a project to go through the hassle of registering an account somewhere else, going there again to check feedback, etc.

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[PengXiaoFabric]

I don't think it is a feature-request. This is a defect. If a folder is excluded, it should be ignored.

[pyrtsa]

I agree this is a defect and also a duplicate of #1588. (I'll share my findings there for the fix.)

[3ggaurav]

Solution(Answer) -- Yes, Same problem was occurring for me also i.e. 'File does not exist' warning.
So aws s3 sync is used to synchronize folders only not for a particular file/files. It's expecting both source and target to be folders (local folders or S3 URIs). If you put your file into a folder and then sync your folder to s3 folder. It work fine. Thank you.

[madhavvishnubhatta]

I see a similar error message but a different situation.

I have an EFS folder with close to 3000 files. 2 can be seen below.

root@0bec216c363d:/home/raw/work# ls | wc -l
2837
root@0bec216c363d:/home/raw/work# ls -lrt test_file*.txt
-rw-r--r--. 1 root root 0 Apr 27 20:26 test_file1.txt
-rw-r--r--. 1 root root 0 Apr 27 20:26 test_file2.txt

Since I want to copy only one of them I use exclude and include options like this aws s3 cp /home/raw/work/ s3://some-bucket/work/ --recursive --exclude "*" --include "test_file1*.txt" (I know that I dont need wildcards here, but this is intended to be an example)

When I use "aws s3 cp" to copy a file to an s3 bucket it takes some time (which is due to some throughput time related to EFS).

During this time , if I rename the second file (which is to be excluded from the copy) then it throws a warning for the second file. This should not be happening as that file is to be excluded anyway.

root@0bec216c363d:/home/raw/work# time aws s3 cp /home/raw/work/ s3://some-bucket/work/ --recursive --exclude "*" --include "test_file1*.txt"

After invoking this command I rename the second file to something else with this command mv test_file2.txt test_file2.txt.bkp

When the original copy command finishes, it throws a warning and returns with a non-zero error code (I just added the time command to show that it takes about 30 seconds, by which I rename the second file).

warning: Skipping file /home/raw/work/test_file2.txt. File does not exist.
upload: ./test_file1.txt to s3://some-bucket/work/test_file1.txt

real    0m31.960s
user    0m2.058s
sys     0m0.420s
root@0bec216c363d:/home/raw/work# echo $?
2
root@0bec216c363d:/home/raw/work#

I believe this should be a bug, but I am not sure if it should be on this bug, or should I open up a new one for this? If it has to be a new one, please let me know and I will create a new one.