Syncthing: Ignore deletes

[compusolve-rsa]

Syncthing rocks. Okay, next usability issue...

Scenario is as follows: Android takes videos, and syncs to PC. After sync I delete the files on android.

It would be fantastic to have a way that even with "ignore deletes", the interface tracks deletes in .stfolder both ends and still shows proper syncing for this type of usage case. Currently it shows out of sync when ignore deletes is on.

I love syncthing.

[imsodin]

Actually, what stops us from instead of just dropping deletions marking them as invalid/ignored instead? That would then get rid of the out-of-sync status.

[calmh]

You mean if I have a file, I'm configured to ignore deletes, and I receive a delete update ... I should announce the delete with the invalid bit set, even though I still have the file?

This is about looking out of sync from the remote side right, not locally. We don't look out of sync locally when ignoring deletes afaik.

[n1trux]

Actually it shows "out of sync" on both sides. IMHO if "ignore delete" is activated, the peer should still pretend that it has deleted the file.

Imagine lots of users and someone not wanting others to delete their files, be it on accident or on purpose.

[calmh]

Syncthing doesn't currently know how to lie. In order for it to tell other devices it has deleted a file that needs to be the status for that file in the index. Since it's not the truth it would then discover the file as not-deleted on the next scan. Lying in index updates would be a whole new layer of complexity.

[n1trux]

Interesting viewpoint, I haven't considered that!

Another idea would be to have a "partial one-way sync" from the phone to the server.

• file is uploaded from Phone→server
• file is deleted on phone, server doesn't delete
• deleted file is ignored on phone
• when server changes file, it gets un-ignored

I'm not sure if that'd solve @compusolve-rsa's issue though. Currently they seem to want to use Syncthing as a one-way sync without recreating stuff on the phone?

EDIT: I guess lying would be the more intuitive way, now that I think of it. I don't know how the DB is currently organised, but can't you store a "deletion bit" which tells it to not re-discover the file?

[AudriusButkevicius]

Syncthing is bidirectional sync, partial one way sync is not something that syncthing is designed for. Yes, it's annoying when you use it in a way it was not designed for.

[n1trux]

Which tool would you suggest in that case?

Can we, as the users, get a write-up on the main page about the design of syncthing since this issue has come up several times now? There's for example #863 which talks about a similar issue.

And why is it that even though Syncthing is not designed for partial one way sync, you can set "Send only" or "receive only" folders, combine multiple peers and have a "ignore deletes" option? Somehow I think the "ignore deletes" option should be removed then as it's against Syncthing's scope.

[AudriusButkevicius]

I don't have a recommendation for what you are looking for. What you are looking for is more like rsync rather than syncthing.

The issue you linked is in no way related to this one. It talks about database never being cleaned up of items that no longer exist.

I agree, ignore deletes should be removed, as I don't think it fits with syncthing's purpose, but that ship has sailed, and there are tons of people that rely on it existing, so I guess it will have to stay.

However, I don't think we should make any design changes or additional feature work to support a crutch we don't want to have in the first place.

[n1trux]

IMHO you should either remove/deprecate "ignore deletes" (maybe in 2.0.0?) or support it. But "partial support" or leaving in a crutch which goes against the scope of the project seems contraproductive and prone to further confusion by the users, don't you agree?

What you are looking for is more like rsync rather than syncthing.

As far as I know there's no "distributed rsync" which allows multiple targets over a spotty connection with a similar protocol to Syncthing, which is what OP and I seem to want.

[AudriusButkevicius]

It's not an easily accessible feature and people already have to go into scary looking menus, which clearly indicates that it's not a mainline feature.

I think what people use this feature for is to backup their android photos and delete them on the device, while keeping them on their servers, this is not really "distributed", and a simple retry loop handles the spotty connection side of things.

[berglh]

Hello. I've searched a lot about this functionality and really feel like I'm just hitting the same problem many Android one way media-sync people are having. The "Send Only" option is still useful in the mobile context, currently I'm using ignoreDeletes on my "camera" folder from Android, which is as expected with the current software deisgn, always "Out of sync" and there is the threat of "Override changes" which blows always the global state of all the other nodes in the share, even if "ignoreDelete" is true.

In my context, I have three devices with two shared folders. For a general purpose "Dropbox" style folder, everything works just fine, as is the main design philosophy of Syncthing.

A.) Hosted Server: Send & Receive
B.) Home Server: Send & Receive
C.) Android Phone: Send & Receive

Even if this was not the intended use of Syncthing, I still see the use of a blind "Send only" folder type as useful. At the moment, my "camera" folder has the following configuration on each device:

A.) Hosted Server: Send & Receive (I want deletes from B, not C)
B.) Home Server: Send & Receive + IgnoreDeletes (To prevent total deletion of folder from accidental hit of "Override Changes")
C.) Android Device: Send Only (Only send what is in this folder, ignore rest of Global State)

While I'm currently using it as a hack, ideally I want to have the "True Global State" to be shared between Hosted Server and the Home Server, and I actually want to the Android Device to not care at all about achieving a true global state locally.

I believe the confusion users have initially with "Send Only" is that this feature is actually "Send Only as Primary". If the Global State does not match the "Send Only" device, in this case the Android device, then the "Override Changes" button is presented. In the context of a "Send Only Primary", this makes absolute sense and is a great indicator that something is conflicting and action should be taken with this being a good remedy for inconsistency.

In the "Send Only Secondary" context that I am using, and I seem to find ample other Syncthing users trying to replicate, it is problematic in my configuration where I want any delete on my Home Server transmitted to the Hosted Server. In this configuration though, if "Override Changes" is pressed, it is ignored on the Home Sever (keeps the files) but then deletes all the files from the "Hosted Server", essentially voiding my offsite copy usefulness.

With a new folder type on the Android Client, "Send Only Secondary", it should only need to to look at the Global State to figure out what files are not present in the state and transmit them to the Send & Receive nodes. Once the files exist in the Global State, then the Send Only Secondary should be "satisfied" that it's state is synced into the Global State. I don't see that there is any lying here, it just does not need to satisfy the Global State locally.

The problem that I can foresee is that unless the Home Sever and Hosted Server know that the Android device was "Send Only Secondary", then it would consider it "Out of Sync". From that perspective, there would need to be a shared "Node" classification in the share itself in order for the Full State clients to know that "Android Device" is a "Send Only Secondary". Is it this part of the functionality that is contentious to the devs because it would be adding an additional layer node role mappings in the share state?

[calmh]

I think the right way to do it would be to have “ignore deletes” be a sending-side option like you suggest. When set, and when we detect a deleted file, we set a local flag on that file in the database and announce it as invalid. From other devices point of view this simply means we don’t have the file any more and make no claims about it.

(And we don’t use master/slave terminology. Especially not right now.)

[berglh]

I think the right way to do it would be to have “ignore deletes” be a sending-side option like you suggest. When set, and when we detect a deleted file, we set a local flag on that file in the database and announce it as invalid. From other devices point of view this simply means we don’t have the file any more and make no claims about it.

(And we don’t use master/slave terminology. Especially not right now.)

Ah, you are right that is definitely use of a charged word - as innocent as it was in this computing context. I openly apologise to anyone I may have offended with my comment. I've updated my comment to use Primary and Secondary in it's place, and will endeavour to be more conscious of this in the future.

[calmh]

Not clearly actionable, I rewrote a bit into #6870