gsutil rsync fails if a dangling symlink is encountered

[ggreenway]

If gsutil rsync is pointed to a local directory that contains a dangling symlink, it will fail with output:

Building synchronization state...
[Errno 2] No such file or directory: '/etc/motd'

This occurs even if gsutil rsync is passed -e (ignore symlinks).

[mfschwartz]

Fix will be included in gsutil 4.6.

[ggreenway]

That was a really quick turnaround. Thanks!

On Tue, Sep 2, 2014 at 10:30 AM, Mike Schwartz [email protected]
wrote:

Fix will be included in gsutil 4.6.

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gsutil/issues/222#issuecomment-54187651
.

[mfschwartz]

No problem. If this bug is blocking you let me know - I could build a pre-release (or you can just wait for 4.6 to come out).

[kdkd]

This bug seems to have returned, but with slightly different symptoms.

gsutil version

gsutil version: 4.19

gsutil rsync -C -r /src gs://x/y

Building synchronization state...
Caught non-retryable exception while listing file:///src: [Errno 2] No such file or directory: ‘/src/sbin'
At destination listing 10000...
At destination listing 20000...
At destination listing 30000...
At destination listing 40000...
At destination listing 50000...
At destination listing 60000...
At destination listing 70000...
At destination listing 80000...
At destination listing 90000...
At destination listing 100000...
At destination listing 110000...
At destination listing 120000...
At destination listing 130000...
At destination listing 140000...
At destination listing 150000...
At destination listing 160000...
At destination listing 170000...
At destination listing 180000...
At destination listing 190000...
At destination listing 200000...
At destination listing 210000...
CommandException: Caught non-retryable exception - aborting rsync

If there are any broken symlinks anywhere in the source, it aborts without copying.

[thobrla]

It doesn't look like you're using the -e flag as originally described in this issue. -C (continue on error) only applies to the copying portion of rsync, and not to exceptions encountered while iterating the source/destination. This is documented in gsutil help rsync.

[kdkd]

Maybe I'm misunderstanding, but the original complaint seemed to say that adding -e didn't change the behavior that broken symlinks were causing rsync to fail.

Our specific case is wanting to back up chrooted environments, where there are going to be what appear to be broken symlinks by design.

I'd argue that the current behavior causes more harm than good and that -C should apply to the file iteration stage as well. If the user is already accepting "I want as much as possible backed up even if there were errors" with -C, that probably should cover things like broken symlinks?

Purely for compatibility with other tools, normal rsync or "s3cmd sync" don't abort in this situation.

[thobrla]

You are correct about the original issue - there was a bug where -e still broke even in the case of broken symlinks. That is now fixed and you should be able to use -e to exclude symlinks (broken or valid).

The semantics of -C apply only to the copying phase because applying them during the iteration stage has a number of usability issues:

• In the user backup cases where listing fails due to Unicode filename errors, the user would back up all of their valid filenames, but all of their invalid ones would not be backed up. They'd still get exit code 1, but the steps they would need to resolve the error would be the same regardless, and it would be hard to output a good error message if other non-Unicode failures occurred.
• Depending on the kind of error in listing the the source or destination, using -C in conjunction with the -d flag could wreak havoc and delete a bunch of files that are actually present in the source. For example, imagine the user was using the -C flag to syncing from GCS -> local and lost permission to the bucket, potentially in the middle of the listing operation. In response to the HTTP 403 forbidden we would delete some or all of the destination on the local filesystem. The alternative would be to special-case certain errors as non-continue-able, which seems like an even more confusing semantic.
• If -C did allow for a collection of mixed valid/invalid files, how would we differentiate cases where valid files failed transiently from invalid files that could never succeed? The user would need to parse the output logs to ensure the backup occurred as expected. If they are going to go to the trouble to do that, wouldn't it be better to provide that information up front?

In general, our philosophy is that if you want to refer to a collection of objects/files to copy using the gsutil cp or gsutil rsync commands, then the entire collection that you're referring to needs to be valid. You can use the -e and -x flags in gsutil rsync to filter your collection so that it is valid.

[pmackinney]

I just hit this issue while using the -e flag. The bad symlink pointed to a directory. I've tried several times and it seems to be current in gsutil version 4.20. Same or different issue?

$ mkdir -p /target1/missing
$ mkdir subdir1
$ ln -s -f /target1/missing subdir1/missing
$ rmdir /target1/missing
$ gsutil -m rsync -n -e -r -d subdir1 gs://xxx_redacted_xxx/my_source
Building synchronization state...
Caught non-retryable exception while listing file://subdir1: [Errno 2] No such file or directory: 'subdir1/missing'
CommandException: Caught non-retryable exception - aborting rsync

[thobrla]

I believe this issue was introduced with https://github.com/GoogleCloudPlatform/gsutil/commit/5ecd79b9c54b462c9ee20ae1dd9680e48a0f84fa - the issue is that we try to retrieve the file size in the wildcard_iterator even if it is a broken symlink. Thanks for providing the example - we'll look into getting this fixed for gsutil 4.21.

[jmchapman84]

I am getting the same issue, currently moving our archive backups to nearline and get the following on all broken symlinks:

[13:39 [email protected]:~]$ gsutil -m rsync -C -d -e -r /backup/devserver.ied/current/ gs://xxxx/devserver.ied/
Building synchronization state...
At source listing 10000...
At source listing 20000...
At source listing 30000...
At source listing 40000...
At source listing 50000...
At source listing 60000...
At source listing 70000...
At source listing 80000...
At source listing 90000...
Caught non-retryable exception while listing file:///backup/devserver.ied/current/: [Errno 2] No such file or directory: '/backup/devserver.ied/current/srv/sites/xxxx/www/saml'
CommandException: Caught non-retryable exception - aborting rsync

Any rough idea when 4.21 will be released as this is now holding up our project.

Thanks,

[thobrla]

We're shooting for a gsutil standalone+PyPI release sometime next week (gcloud releases typically follow 1-2 weeks after), but hitting that date is dependent on successful testing.

In the meantime, as a workaround, is it possible for you to delete the broken symlink?

[thobrla]

https://github.com/GoogleCloudPlatform/gsutil/commit/fde548721de1ae8d8da39988b8d3cd1cb09fcf0b should fix for gsutil 4.21

[NeonHorizon]

When does 4.21 get released as this is currently stopping me backing up encryptfs directories?

[thobrla]

We're currently in the process of release testing.

[thobrla]

gsutil 4.21 released standalone and PyPI versions today, and the gcloud-bundled version should be available within the next couple of weeks.

[qhh0205]

Hi, @thobrla
It looks like this bug still exists with gsutil 4.34:
Reason: broken symlink

$ gsutil version
gsutil version: 4.34

$ gsutil -m rsync -d -r ~/.jenkins gs://xxx/jenkins_home
...
CommandException: Caught non-retryable exception - aborting rsync

[houglum]

The fix mentioned above still holds when using gsutil rsync -e. The documentation for gsutil rsync's -C option (which is implied when the top-level -m option is supplied) specifically highlights this behavior; it's working as intended:

-C If an error occurs, continue to attempt to copy the remaining
files. If errors occurred, gsutil's exit status will be
non-zero even if this flag is set. This option is implicitly
set when running "gsutil -m rsync...". Note: -C only applies
to the actual copying operation. If an error occurs while
iterating over the files in the local directory (e.g., invalid
Unicode file name) gsutil will print an error message and
abort.

[qhh0205]

@houglum
I have test with -C, but still not rsync the dir which includes the broken symlink to the gcs bucket. So
how do i rsync dir with broken symlink to gcs bucket without -e args.

$ ls -l test
ls -l test
total 8
-rw-rw-r-- 1 proxy proxy 4 Sep 20 19:14 a
-rw-rw-r-- 1 proxy proxy 4 Sep 20 19:14 b
lrwxrwxrwx 1 proxy proxy 1 Sep 20 19:14 la -> a
lrwxrwxrwx 1 proxy proxy 1 Sep 20 19:14 lc -> c  # this is broken symlink

$ gsutil rsync -C -d -r test gs://backup-repo/test
......
Building synchronization state...
Caught non-retryable exception while listing file://test: [Errno 2] No such file or directory: 'test/lc'

CommandException: Caught non-retryable exception - aborting rsync

[houglum]

We specifically want rsync to fail early if there's a broken symlink (and the -e option was not specified). The comment above from thobrla explains why; specifically,

[...] if you want to refer to a collection of objects/files to copy using the gsutil cp or gsutil rsync commands, then the entire collection that you're referring to needs to be valid. You can use the -e and -x flags in gsutil rsync to filter your collection so that it is valid.

[kdkd]

We stopped using gsutil completely because of this, but just to chime in on why: We _want_ to back up broken symlinks. We were backing up jails (chrooted environments) where the symlink wasn't broken inside the jail, but was broken if looking at it from outside the jail. Those were still important to make backups of, despite appearing broken from gsutil's perspective.

Another use case is backing up filesystems that we don't have full control over. We still need to backup symlinks, and can't have backups stop working because an untrusted user created a broken symlink somewhere in their home directory or in a jail or whatever.

In both cases -e doesn't work (backing up symlinks is important for us, broken or not) and -x also doesn't work (we don't necessarily know where broken symlinks are, even if we could skip them). Our use cases might not be what gsutil was designed for, but because of that we're no longer using google's cloud storage at all.

[houglum]

Ah, I see. I appreciate the insight. If Cloud Storage were to support symlinks somehow, gsutil could build functionality to work with that. But since the service itself doesn't support symbolic links, gsutil can't (trivially) do so either. The workaround it uses when it encounters a symlink is to copy the bytes of the link's target file instead... which doesn't really work for your use case, I agree.

[DanielJoyce]

THis is fairly annoying. Was trying to backup my laptop to my GCP bucket, but apparenly I have dangling symlinks scattered here and there which make it barf. It would be nice if gsutil had a "Ignore broken symlink" option as they are fairly common.

[mbrostami]

This is really annoying, I even tried to exclude the directory contains broken symlinks, but it's always returning the same error.

gsutil -m rsync -dreC -x 'temp/.*$' /var/www/app gs://my-bucket/app 
Building synchronization state...
Caught non-retryable exception while listing file:///var/www/app: [Errno 2] No such file or directory: '/var/www/app/temp/07e7486347d385d1491fe82f3eac9ab0'
CommandException: Caught non-retryable exception - aborting rsync