Syncthing: Add an option to follow symlinks instead of syncing them "as is"

[Albirew]

Let's talk about my case:
I've 2 servers and 2 computers with syncthing on.
Servers sync between them www with symlinks that needs to stay as it (like in current state)
Computer 1 sync with everyone else a folder with symlinks pointing to differents area of my computer (appdata.purple, appdata\thunderbird, userprofile\documents, etc.).
These symlinks need to be treated as if they were real folders. Instead, syncthing currently treat every symlink as a simple file, hence does not backup my data.

Is there a way to add an option in folder sync parameters to treat symlinks inside as links or as folder?
Thanks.

[jpjp]

I think you are asking for the feature that symlinks should be followed. This is currently not supported (see the FAQ https://github.com/syncthing/syncthing/wiki/FAQ).

This issue might be a dupe of https://github.com/syncthing/syncthing/issues/92

[calmh]

Yes, sort of. But this matches the bounty request posted on the forum, so I figure let people have a go at it.

[cdhowie]

I might take a stab at this. Note that as a workaround you can mount --bind instead of using symlinks, but that's swatting a fly with a sledgehammer and I wouldn't want to have to do it that way either.

[cdhowie]

Or, an even easier workaround: have the links go the other way. I use a Git repo for my cross-machine config I want to carry around and in it I have a script that will lay down some symlinks that point into the repo working dir instead of the other way around. That is, ~/.vimrc points to ~/config-files/vimrc instead of vice-versa.

[Albirew]

yeah, that's what I did for now (and working quite well as a workaround).

People wanting to do the same need to be carefull, one of the target apps can have the same behavior as Syncthings and may crash or reset to default.
So if you moved/symlinked an app's data, try the app to see if it's still working.

[calmh]

I implemented this in PR #1983, but didn't merge it. Most of the relevant code is in https://github.com/calmh/syncthing/commit/1068742c0ffcf79a604e573f1512d28e786d4793 and the documentation in http://docs.syncthing.net/draft/followsymlinks.html.

It wasn't merged because I didn't find a graceful way to handle turning it off (essentially suddenly all files that were followed are deleted) and it's too large a footgun. If someone finds a nicer way to handle this they're welcome to pick up the work from where I left it.

[mjmvisser]

A use case for following symlinks (and a workaround using unison).

We have a master repository, /data:

/data
  foo
    assets
  bar
    assets
  baz
    assets

We wish to share a subset of /data with two vendors using symlinks.

/vendor/a
  foo -> /data/foo
  bar -> /data/bar

/vendor/b
  bar -> /data/bar
  baz -> /data/baz

And our workaround:

Create the /vendor trees using symlinks as above, then create an empty directory /share. Now run unison to sync /vendor and /share:

unison -ui text -batch -root /vendor -root /share -follow 'Regex .*'

This will populate /share by following symlinks.

Now use syncthing to share the /share/a folder with vendor a, and /share/b folder with vendor b. Run unison in a cron job to keep /vendor and /share in sync.

[calmh]

Quoting https://forum.syncthing.net/t/600-bounty-for-symlink-handling/2493/29:

unison4 solves this issue by specifying the symlinks explicitly in the config. in syncthing's case it could be done that in the folder settings a new element followSymlink (singular) is introduced, which can appear several times, e.g.

 <folder id="default" path="/Users/jb/Sync/" ro="false" rescanIntervalS="60" ignorePerms="false" autoNormalize="true">
    <!-- ... -->
    <followSymlink>foo/bar</followSymlink>
    <followSymlink>foo/bit/*</followSymlink>
    <!-- ... -->
</folder>

the path names given are relative to the path attribute of the folder element.

so every time a symlink is encountered during the scan, it is matched against the list of followSymlink entries (which could also include ignore pattern style globbing), and if found, the symlink is treated like it's target (be that a symlink, file, folder or whatever).

removing a setting after a sync of the target has been performed, would replace the file/folder on a remote device with a symlink in case symlinks are supported. so files could get lost on remote devices (if they support symlinks), but never on the local device.

This seems like a sensible and proper solution to me. So much so that I'm tagging this as Planned for stuff to do when the current stuff is done.

[profucius]

I just want to thank you for taking this seriously. I depend on Syncthing for syncing files between my desktop and android phone. But some files I need synced (i.e. eBooks) are in a larger folder structure of which I only want cherry picked files synced. My current solution is to have a specific folder dedicated with subfolders for genres, and the cherry picked .pdf and .epub files be symbolic linked into these folders. I have a couple other uses for this system besides eBooks (folders symlinked too), but generally having Syncthing follow symbolic links would be greatly useful to me!

For reference, another backup/sync app that processes files similar to this is called Chronosync (OSX) and another GoodSync (OSX). I was using it like this, but the downside to me is that I have to use another macro system to trigger the sync when my phone is plugged in, whereas Syncthing will automatically sync when it's on my Wifi and charging (love it).

Anyway, you have an avid user user awaiting this feature with baited breath :) Thanks

[cdhowie]

A temporary compromise solution (to get a lot of use cases out of the way) would be to add an option to follow symlinks, but only respect it when the folder is in master mode. (The "follow symlinks" UI option would be disabled if the folder is not in master mode.) This allows the use case of "sync only some parts of this large directory to my mobile device" without enabling a symlink traversal vulnerability.

[AudriusButkevicius]

That is already solved with ignores

[profucius]

I use .stignores in other syncs, but it is simply not feasible with the scope of what I'm doing. First, there are thousands of files/folders in my heirarchy, and I want to cherry pick only certain files. Second, symbolic linking would allow me to change the folder structure; I want desktop storage folder structure to be different than the synced storage folder structure. And no, I'm not making copies of the files, I would need an unnecessary separate hard drive. Third, having a specific folder with symbolic linked folders and files within gives me the power to control my parameters outside of Syncthing, and use other tools with it (change file timestamps, print a synced-only folder/file list, sync symbolic linked files with other apps, etc).

For these reasons, I still believe Follow Symbolic Links is a powerful necessary feature to include in Syncthing.

I should mention I want to use this feature on Mac OSX (10.10).

[timhowes]

For more complex use cases where you want to separate the file syncing from the file organization, I'd suggest looking into git-annex and the git-annex assistant. It stashes all the files in a .git/annex/objects directory and automatically generates symlinks within your directory structure referring back to the appropriate objects. And it lets you selectively decide which objects get stored on which machines. There's a significant learning curve compared to Syncthing, however.

https://git-annex.branchable.com/
https://git-annex.branchable.com/assistant/

[AudriusButkevicius]

@profucius I think you misunderstand how this whole thing will work. You will still have to explicitly specify which symlinks you will want syncthing to follow, essentially resulting in the same thing has as ignores, essentially just allowing you to reorganize a bit, which can already be achieved with hard links to be honest.

[profucius]

@timhowes Thanks, I'll take a look at this to see if it fulfills my needs in this area.

@AudriusButkevicius I already have my files/folders symbolic linked into a folder for syncing. I did this months ago when I was using GoodSync (OSX) for syncing. Maintanence of this will be minimal, versus switching over to an stignore file.

Problem is, I can't use GoodSync for transferring certain files because it doesn't keep file timestamps (camera photos). And I don't want to be using two syncing apps just because Syncthing doesn't follow symlinks, I'd really like to have it all in one app.

Secondly, an stignore file doesn't fulfill my need to have the synced folder be a different hierarchy than the source. Symbolic links allow me to create a different hierarchy for syncing.

Thirdly, hard links only work on the same hard disk. I want the symlinks to be placed elsewhere, I have a main folder of symlink folders centralized on another drive.

"Following Symbolic Links" certainly does have benefits to people with needs beyond the scope of typical goals.

[calmh]

Guys. This is planned. There's an implementation under way. Stop arguing for what's already decided in your favor.

[profucius]

Again thank you, your hard work is appreciated. Just to clear things up, I'm not arguing with anyone, my goal is simply to ensure that I clearly state my goals, in case the differences noted are important in adding the feature. Anyway, looking forward to it!

[calmh]

I'm unplanning this again for the moment as I'm not sure the solution is a practical one even though it's technically possible. I welcome further discussion to address @frispete's comment in #3387.

[calmh]

Deleting offtopicness. Keep focused in the bug tracker please.

[f1egmatik]

Workaround for linux users with running Samba server:

mkdir .links # if you haven't it yet #
ln -s whatever .links/
vim /etc/samba/smb.conf # add .links directory as [links] share #
echo "//127.0.0.1/links /path/to/directory cifs nounix 0 0" >> /etc/fstab # nounix option is very important here # 
mount /path/to/directory 

Then add /path/to/directory to Syncthing.

[wweich]

Why go through so much trouble mounting it through samba, if you can just bind mount?

[f1egmatik]

"mount -t cifs" and "mount --bind" are not equivalent in this case. For example, i have large "Music" directory with hundreds artists on the computer. And i want the same music but only with several (about 20-30) artists on my mobile player. With "--bind" we need to mount every artist (symlink to sub-directory) separately, because if we mount whole .links directory with "--bind" symlinks inside it are still represents as symlinks. And if i want to rotate some artists, i need to unmount and mount directories with root rights every time.

When i mount with "-t cifs -o nounix" samba represents all symlinks as regular files and directories (hooray! -- says @profucius). I need to mount .links with root rights only once. And i can easily remove and create symlinks in .links directory to rotate music (on my player).

Offtopic: i already write bash string to automatically replace one artist per night with crontab:

cd /path/to/Music/.links/ && ln -s `shuf -n1 -e ../*` ./ && rm `shuf -n1 -e *`

[calmh]

This has been cooking for a long time and each attempted implementation has run into unanticipated difficulties. Given that and the theoretical difficulties of ensuring correctness in the presence of (followed) symlinks I've decided that following symlinks will not be implemented. I'm also locking this issue because I don't want followup arguing about in the bug tracker. Anyone is welcome to argue about it on the forum, but unless the argument is backed up by a foolproof implementation it's unlikely to have a positive outcome. Sorry, peeps.