Client: [Windows] Junction links (symlinks) are being synced

[Pathduck]

Since the 2.2.2 update the client has started syncing directories linked by junction points (symlinks on Windows).

I am syncing My Documents folder, on Windows this contains a "My Games" folder which often gets very large. I have created a junction point to another drive to save space on c:.

Earlier versions of the OC client did not sync these, I believe junctions stopped syncing some 1.x release. Now they are suddenly back to working, and I did not find any thing in the release log to explain why they would now work when before they did not. If they now are expected to work I have to rethink my folder setup.

Steps to reproduce

1. Create an NTFS junction point in a synced folder
2. Start client and add folder
3. The files in the linked folder will also be synced
   

   Expected behaviour

I would expect links (junction points) to not be synced

Actual behaviour

The files in the linked folder are synced

Server configuration

Owncloud 9.0
I don't own the server, it's a shared hosting solution.
I assume this is a client issue.

Client configuration

Client version: 2.2.2

Operating system: Windows 10 Pro, x64

OS language: English UK

Installation path of client: d:\bin\ownCloud\

Logs

Will provide if needed.

[ckamm]

Quick tests: Linux symbolic links skipped by 2.2.2, but Windows directory symbolic links and junctions are uploaded.

[ckamm]

@dragotin I've debugged this a bit, and it looks like an unintended consequence of 055c2ef73f84b68ebfbe0296fadef438981cba37 (for #4056). When I remove the checks for SIS and DEDUP, the symbolic link and junction point gets ignored again on windows.

[dragotin]

@ckamm yes, I agree on that. The question is if there is a way to detect the junction without mixing up with the network files.

[ckamm]

@dragotin The problem is that the code assumes the reparse point tag is a combination of flags, when it's just a value. I can fix that, but the patch will need testing for https://github.com/owncloud/enterprise/issues/1225 and https://github.com/owncloud/client/issues/4056 .

[ckamm]

@dragotin Pull request: https://github.com/owncloud/client/pull/5036

Note that we accidentally broke that for 2.2.0 - people may complain if we restore the old behavior again....

[Pathduck]

Thanks for looking into this.

Personally I would have preferred symlinks and junctions to actually work but I understand that it's complicated - at least now it can hopefully be predictable again :)

[mcastroSG]

Hi:

Junction points or symbolic links are not synced.

Steps Executed
1.- On a shared sync folder add a new Junction point as mklink /J Junctionpointname PointedFolder
2.- On a shared sync folder add a new Symlink as mklink /D symlinkname PointedFolder

Result

Junction point or symlink are not synced and a two new entries at activity > not synced tab could be found : Symbolic links are not supported in syncing.

Client OS: W7/8.1/10
Client Version: Version 2.2.3-nightly20160708 (build 6234)

Server OS: Ubuntu 14.04
Server Version: 9.1 RC1

[ghost]

I guess I started using ownCloud 2.2.2 just before this commit and was really appreciating its feature and its similarities to how most clients handle linked folders.

I've now upgraded to ownCloud 2.2.3.6307 and was wondering if this feature to include linked folders could perhaps be optional instead of reverted/removed?

[Marc05]

I'd also love to have symlink support. It seems this was requested about 2 years ago already (#1440). Sucks that it's been put off for so long. I recommend closing this ticket and continuing the discussion on the other one.

[guruz]

I agree!

[guruz]

I'm reopening this issue after some discussion with @dragotin
We're quite unsure if we should really go on with this, as this is really a feature that only a minority of users would use.

Maybe we need to distinguish more between junction points and normal symlinks?

[ghost]

Thank you for reopening this issue (also sorry for not creating a new one).

If I understand correct I'm hoping symbolic links mklink /d to be synced again. I'm not really sure about the exact behavior of junction points or how different software handle these.

[dragotin]

guys, there are a couple of issues already. I _thought_ I summarized a couple of questions that we have to clearify before we get going with symlinks, but can not find it. Let's start here.

1. Do we want to sync the symlink or dereference. That is the Gretchen-Question
2. If "symlink-sync" we need an object "symlink" on the server, and thus server support.
3. A symlink object on the server would be mostly useless on the server if it points out of the sync dir
4. If we de-reference the symlinks, what if the link target goes away? Wipe the file on the server?

Probably there is more to think about. @gladiac do you have input? Thanks!

[Marc05]

Here is a document that distinguishes between the differences fairly well.:
http://www.2brightsparks.com/resources/articles/NTFS-Hard-Links-Junctions-and-Symbolic-Links.pdf

A good way to start would be to answer the question: What should the default behaviour be?

We need to take some things into consideration to answer this:

• Distinguish between Hard, Junction, and Symbolic links.
• If it's a Junction or Symbolic link, the target may be on a different drive (local or external), hence monitoring file changes is difficult, and maybe not practically possible.
• How should links behave on the server?

There are many different situations that come into play here, and having the default behaviour account for all of them would be nice, but perhaps not practical if the feature is to be released sooner than later. The goal for now could be to distinguish between all three different types of links, but treat them the same - that is, only synchronize the link and not the data. In this case, the server would not follow/open links, only synchronize them to clients.

Having even this initial functionality available to users would be beneficial, and can be expanded on afterwards given that the functionality to distinguish between types of links would be available already.

[guruz]

Seems like Windows 10 has improved symlink support now? https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/#Z2PXD4EP0gBrsYmu.97

[9k22]

The way to handle junctions or symlinks seems clear to me: even under Windows these kinds of soft links are now handled transparently for windows applications. The ownCloud client is a windows application, so: handle it transparently! If you are afraid, that users are confused by a changed behaviour, then skip it by default (as it is done in Version 2.2.4), print a warning, but provide a checkbox in the settings to activate the syncing of junctions and symbolic links. If you think, this could be a security issue, then make a statement in the settings dialog, but let the user decide, if he wants to take the "risk".
I understand the requirement to sync the reference itself instead of the object which is referenced (#1440), but this is clearly the less frequent use case and should therefore be ranked with significant lower priority.

Another related issue is, that linked files should be linked again on the server. This is exactly, what dropbox does: if you have 2 symbolic links to a big file (e.g. video clip), the upload of the first link takes a while, but the upload of the second link is done very quickly - obviously client and server exchange checksums (or similar information) to detect duplicates. BTW: this logic of the dropbox sync protocol has nothing to do with client symbolic links, but would also works for "real" copies of big files at client side - the dropbox server simply avoids to store redundant data this way.
My original plan was to replace dropbox by ownCloud, but unfortunately, without these two features this is not possible :-(

[RalfWW]

@9k22: Full support from my side!!!

[SamuAlfageme]

@ckamm could FileSystem::isJunction() from https://github.com/owncloud/client/pull/6349 be also useful in this context?

[ckamm]

@SamuAlfageme No. Detecting symlinks and junctions is something we already do during discovery. The main obstacle to synchronizing these is that one would need to think carefully about the desired behavior and implement that. And that's not been a priority so far.

[9k22]

Exactly. BTW: the "need to think carefully about the desired behavior" is IMHO not that big thing: the strategy of e.g. dropbox is perfect: simple to understand, simple to communicate, totally Independent from the way it is organized on the client side, and behaves exactly as I need it. And this is really not "rocket science". I wonder, why not just copy and implement the dropbox behaviour ? Until this is not done, I cannot move from dropbox to owncloud.

[guruz]

@ckamm didn't you do something related to this topic recently?

[ckamm]

@guruz What I did was make sure recursive deletions didn't delete data behind junctions in #6349.

I agree with @9k22 that treating a symlink as transparent and just syncing the data behind it makes sense. Potential issues:

• Changes behavior. Someone somewhere will depend on symlinks being ignored.
• How does propagating remote renames/moves work (the local symlink could be relative?)
• Do we detect broken symlinks and preserve the server data in that case, or does a broken symlink cause the remote data to be deleted?

[9k22]

I agree with the identified issues. Proposed mitigation:
ad a) let the user choose the preferred symlink behaviour.
ad b) rename: why is it a Problem ? Move: absolute symlink: why is it a Problem ? relative: if the symlink breaks, the file is gone (see ad c))
ad c) a broken link does not refer to an existing file System object. You could not even make use of such a broken link on the local device, so, why would you think, that it should be preserved on the Server? If a user chooses explicit symlink behaviour, then it is his responsibility, that symlinks do not break. Even if they break: worst case is deleting the copy on the Server, but at least not the original local data. BTW: this is exactly the behaviour of Dropbox and it is fine that way (even although I wondered, why parts of my server data had disappeared ;-) )
BTW: Dropbox does not even care, if a duplicate of a file is a physical copy, or a symlink on the local device. It just checks, that it is the same file (checksum) and stores it only once on the server and keeps track of all references. That's simply - perfect.

[aib]

Changes behavior. Someone somewhere will depend on symlinks being ignored.

Make it optional. Also, https://xkcd.com/1172/ :)

How does propagating remote renames/moves work (the local symlink could be relative?)

Rename: Rename the symlink without touching its target
Move: a) Move the symlink without touching its target. If the link was relative and is now broken, a subsequent write will create the file. The case when it cannot be created should be identical to the case with a regular file. This is the "I don't care about symlinks" option. Will create extra copies of the linked file.
b) Move the symlink without touching its target. If the link was relative and is now broken, update it to point to its old target, converting it to an absolute link if necessary. This is the "my user uses symlinks for convenience, and I'm helping them" option. Users who want their symlinks intact should not be syncing them.

Do we detect broken symlinks and preserve the server data in that case, or does a broken symlink cause the remote data to be deleted?

A broken symlink should be treated as nothing, as it barely provides any more data than a file that does not exist. A symlink entering the broken state is the same as a file being deleted (which is what is happening to the targeted file, anyway)

[ogoffart]

I guess this is the same as https://github.com/owncloud/client/issues/1440

[TheOneRing]

Closing as duplicate of #1440