Syncthing: Editable Folder Paths

[jebeld17]

As sourced in #4209, Folder Paths are not editable in Syncthing, and are not editable due to "potential deletion of files," etc..
I personally don't agree with this stance taken by devs, and believe users should be given an option to change folder paths.

_Why?_
Main thing: Users risk major deletion of files by deleting the directory from ST just as much as having an "edit" button. This can be true because an "edit path" implementation has never been made and put in place, thus we have no example this hypothesis is true rather than estimations and guesses made from an "if, then" situation. Even if these estimations are accurate - such as that the entire directory's contents may delete if another device is delayed to speak with the device we are changing the path/directory on until files have already started being moved from directory1 to directory2, simple locks and rules can be set in place in regards to these situations, such as putting all the contents of directory1 into a locked "in use" state, denying pings from other devices until the directory change has been made, copying the directory instead of moving until the user confirms the files are there backed by HASHs from both dir's, or simply just automating the "delete and re-add" process from ST, leading to...

_As far as "deleting and re-adding" the dir:_
Experienced users know how to change a path, yet inexperienced users will often not know the "correct" way of doing this, which is to delete the share from Syncthing on the device and re-add into a new path with the same Folder ID - as seen in #4209.
Yes, inexperienced users (or users who have never faced this situation before) can learn how to tackle this trick, but it consists of extra steps

I hope I didn't make this simple issue into a big ordeal, but I wanted to put my two cents in to help the project as a whole become more accessible to all types of users.

[calmh]

I honestly don't think we should implement this. For one, I don't know what the "correct" result should be. When I change the folder path, should Syncthing:

1. Just update the path in the config and expect that the contents in the new path are correct? (that means any files not present there will be considered deleted and removed from every other device, etc)

2. Update the path and assume that the state is blank, thus downloading everything from the network again? (What if it was not blank?) This orphans any changes made in the old location but not yet synced to the network.

3. Move all the files from the old path to the new? (What the move fails?)

4. Copy all the files form the old path to the new? (What if the copy fails?)

There are good arguments for and against all of these behaviors. The only one we could reasonably implement is number one, but the dangers of that are significant.

[wweich]

I would say 1 but with automatically pausing the folder an telling the user to make sure that at the new location is what should be there (copy, move the files, or let it blank and remove files from the other devices)

[slytomcat]

I think that the more convenient for inexperienced user is option 3. But I agree that there is a big question: what to do in case of failures in folder movement.

I see two possible ways to solve move failures:

1. Pause the folder syncing and report about error - lets user solve the issue and unpause the folder.
2. Move all files back to old folder and report about error.

First option will require some more deep knowledges from inexperienced user - it sounds bad.
Second is created one more place for failure - during the movement files back. And we recursively got in the same problem that we tried to solve.....

I would suggest to try the option 2 and in case of fail - use option 1. The only thing I consider important that the secondary fail is that one that I can't imagine the reason why it could happen.

[calmh]

Users changes path in config

Dialog pops up

Hi! It looks like you're trying to change a folder path! Please specify how you want to do this:

(*) The new path is blank. Syncthing will assume nothing and resync the contents from the
    network. The contents of the old path will be left untouched by Syncthing.

( ) The new path contains the same set of files as the current path. Syncthing will pause
    the folder to give you time to accomplish this, by copying or moving the files. When you
    unpause the folder the contents better be correct, or else.

[No! Wait!]                                                                       [Engage!]

There's an implicit option "the new path contains something else and I'm inviting disaster" behind both options (but primarly the second one).

[slytomcat]

I assumed that new_foldet_path is not exists at the moment of folder path changed.
When new_foldet_path is exist at that moment of change confirmation - this dialogue is OK.

I also assume that the change of folder path is performed within some new dialogue but not directly in-line. Just put the button 'Change' next to the folder path field in the folder config dialogue. This new dialogue (Change folder path) can check the new folder existence and show the progress of file move.

[Ferroin]

I agree with @slytomcat that it should probably be a separate dialog and the folder path should not be directly editable from the normal folder configuration dialog. This isn't something that's going to be done regularly under any sane state of normal operations (the only possible use case I can think of for doing it regularly would be some kind of rolling snapshot setup, but there are better ways to handle that at other levels (pause folder, copy data to snapshot location, unpause folder)), so convenience isn't as important. Making it a separate dialog also makes it a bit easier to ensure that the user knows what they're doing and further avoids the risk from the 'just click yes until it shuts up' mentality that some less computer savy users have.

[slytomcat]

Edit Folder dialog:
...
Folder Path
[current path (not editable) ]{Change-button} --> *

*

Change Folder Path dialog:
[new folder path (editable) ]{Confirm-button}

Move progress (shown only after Confirm-button pressed)
[++++++++---------------------------------------------------------------------------------]

---

In case of move fail:

Change Folder Path: ERROR dialog
Changing of Folder Path failed due to
All changes will be reverted.
{OK-button} --> (backwarded progress bar can be shown)

---

In case of changes reversion can't be performed:

"BIG TROUBLE" dialog:
Sorry, but Syncthing can't revert the changes due to

Folder paused.

Some data still in while rest of data is in the . You'd better fix the the issue (collect all data in one folder) before unpause the folder.
_____________________________________________________{Ok-button}

[Ferroin]

From a practical perspective, I don't think any initial implementation should include the ability to migrate the data for the user. It's at most a convenience function, nothing more, and handling of that is liable to be a point of contention.

If such a feature ends up implemented though, I think the correct way to handle it is to copy all the data, and then delete it from the source folder once everything is copied. This may cause issues for people who are running with nearly full storage devices, but it also ensures that a rollback is possible simply by deleting everything from the destination folder.

[slytomcat]

Copy_then_delete - requires not only double space on the storage, it also requires a long time wait.

That why I consider move as better solution, even if it more risky. As option we could ask user what he/she prefer: copy & delete or just move. It can be done in Change Folder Path dialog.

[ProactiveServices]

How about:
User elects to change folder path. Syncthing opens a prompt informing user that this operation will shut down Syncthing, after which user must move all of the files themselves, then restart Syncthing themselves. OK / Cancel. Perhaps a "Warning" link with more text stating that if the new dest is empty/incomplete, data loss will occur.

The idea of Syncthing doing the moving (across different file systems, permissions problems, transient errors, existing files, Syncthing crashing half-way through, nested sync folders, ignores not being applied due to new path, god knows what else) strike me as bombs waiting to go off. A dialogue box asking if the new dest is empty or ready also makes me wonder if many users would misunderstand, or not read it, and shoot their foot off.

[calmh]

The idea of Syncthing doing the moving (across different file systems, permissions problems, transient errors, existing files, Syncthing crashing half-way through, nested sync folders, ignores not being applied due to new path,

:scream:

Yeah we're not doing that.

[Ferroin]

@slytomcat A move operation isn't atomic (that is, it can fail half-way through), and it becomes a copy-then-delete if the source and destination aren't on the same filesystem. The point of that suggestion was to avoid the need for error handling in the rollback if the operation fails. Even then, it's still almost always going to be faster than a resync from the network.

@ProactiveServices That would work, but has issues if Syncthing is started by systemd (systemd _will_ restart it when it exits) or another process supervisor. Pausing the folder and then requiring the user to unpause it avoids that issue, and doesn't disrupt syncing of other folders on the same device as much.

[slytomcat]

@Ferroin you are absolutely right that move operation isn't atomic in common case.

But sometimes move is just rename of the folder name. And only RO rights of user on folder (either source or destination) can fail this operation (and this fail is not require any roll-back actions). I see no reason to use copy-then-delete in such case.

So, the move operation has at least two cases:

• Rename the folder (within the same storage)
• Movement data to another storage
  And we have to consider both cases when we talk about the move.

[calmh]

There can be no moving of the folder given the existence of overlapping folders as mentioned above.

[Ferroin]

@slytomcat Actually, other things can cause a rename(2) call to fail (even non-permissions related things on some filesystems). It is admittedly still supposed to be atomic (at least, on UNIX systems, not sure about Windows) even if it fails, but things get complicated in a few cases (say you're moving up or down the directory hierarchy, for example, going from /sync/folder/path to /sync/folder), and it's easier to just avoid all the logic to detect things like that and copy-then-delete instead. At a minimum, you would have to pre-create the directory structure and then rename each individual file to avoid nesting issues.

[slytomcat]

... and even more - /sync/folder/path and /sync/folder could be different storages...

[slytomcat]

At a minimum, you would have to pre-create the directory structure and then rename each individual file to avoid nesting issues.

As I know most Unix|Linux FS don't do the move in such manner. Movement of folder within the same storage/file system is done by changing of metadata only for moved folder. I.e. movement from /sync/folder/path to /sync/folder is just movement the 'folder' catalog record from catalog /sync/folder/ to catalog /sync/.
That's why I got some missed files when I tried to watch for file system changes via i-notify.
Inotify creates only pair of related events: MOVE_IN /sync/folder and MOVE_OUT /sync/folder/path and nothing more events for anything that is inside the folder. For this particular case - it is not a problem. But if I move folder from /some/other/folder to /synchronized/folder (that is under my inotify watcher) then I gom only one event MOVE_IN /synchronized/folder and after that I loose all control over the folder containment if I don't try to rescan the folder.

[jebeld17]

@calmh but what if anything fails?
The user should be warned of the potential risk and should be backing up his or her data anyways, regardless if he or she uses Syncthing. That's just standard precautionary practice.

[calmh]

I'm tagging this issue so that it may live on and some day be resolved. But as it stands it is not actionable as it's not clear what the desired outcome is.

[jebeld17]

I think it ultimately should ask the user what he or she would like to do, whether it's to merge the current files and folder structure in the desired path with the syncthing folder, format the desired path and sync as usual thereon out, choose a different path, etc.. Any options a standard 'move' operation would do in a standard file explorer.

[RenaKunisaki]

Should be able to at least rename the folder on the filesystem.

[atomdmac]

Renaming would be especially nice. If I want to back something up from my phone to my PC, it seems that the directory created on my PC's filesystem is dictated by what it's named on my phone. If I'm looking at my PC's filesystem in the CLI or in a file explorer, this can be pretty confusing.

[calmh]

If you accept the automatic suggestion it’s based on the folder label, yes. But you can specify whatever you like at that point. This is about changing the choice after the fact.