Syncthing: Resync of offline machine stuck in "Out of Sync" state after dirs+files were added and then removed elsewhere

[javmorin]

Current Behavior

I am seeing syncthing get into a blocked "Out of Sync" state every time I switch branches in a code repo I have synchronized. This is a repo that I have been syncing for 6+ months without issue, but is now (with 0.14.16) failing. I cannot say for certain if 0.14.16 is the cause or if it was 0.14.14 or 15 as I haven't done a checkout of this particular branch for about a week prior to today.

I currently see 27 items (a mix of files and directories) "unable to sync" due to a missing parent directory. Here is a sample screenshot from the UI with filenames scrambled.

Looking at the paths the directories are all sub-directories of a (missing) parent directory, and the files are contained in those sub-directories, so that might be an indication of where the issue is. Also of note is that all of the indicated files and directories are not currently present on either machine. I had switched to a branch, then switched back while the failing machine was offline. The syncing machine state _should_ be a no-op actually because it's state matches that of the remote. I should note that I experienced the exact same thing on another machine and "fixed" it by switching branches directly on that machine.

Expected Behavior

• Syncthing completes sync of all non-existent directories once offline machine is reconnected

I will try to get a reproduction script together, but it may take a little bit of time to pull off. Happy to run any suggested diagnostics on the failing machine. I seem to be running into all the syncthing bugs lately after 6+ months of worry-free use :-/

Version Information

Syncthing Version: v0.14.16
OS Version: Arch Linux (x64)
Browser Version: N/A

[AudriusButkevicius]

Use forum for support.

[javmorin]

@AudriusButkevicius I would be happy to do that, but I believe I am reporting a regression. This is functionality that was working fine up until recently. I have been able to formulate a script to reproduce the error.

Start with two machines in sync (hereon referred to as SideA and SideB)

• run setup script in root of synced directory (see below)
• allow SideA and SideB to sync fully
• run Purge script (see below)
• allow SideA and SideB to sync fully
• stop/shut-down SideB
• run setup script again on SideA to recreate purged files
• trigger rescan on SideA and allow to complete
• run purge again on SIdeA
• trigger rescan on SideA and allow to complete
• start SideB and it will get stuck with 16 failed items.

Setup / File Creation script (tested on Linux with Bash)

for P in sample/edir1/edir2/edir3a/edir4/edir5 sample/edir1/edir2/edir3b/edir4/edir5/edir6; do
  mkdir -p $P
  for D in $P/ndir1/ndir2{a,b}; do
    mkdir -p $D
    for F in nfile_{one,two,three}; do
    echo $D/$F > $D/$F
    done
  done
done

Purge Script

find sample -type f -name 'n*' -delete 
rmdir $(find sample -type d -name 'n*' | sort -r)

[AudriusButkevicius]

Thanks, I'll try that.

[AudriusButkevicius]

So I repro-ed the issue, thanks.

[javmorin]

Mine fills up with Folder "**redacted**" isn't making progress. Pausing puller for 1m0s. but as you said, no error is logged.

[AudriusButkevicius]

So I understand the issue, yet not sure how to fix it.

[javmorin]

Understood. Honestly, I am just glad you were able to reproduce it, so I have confirmation it's not just a quirk on my hardware. Happy to run any tests you and the team come up with. Will advise if I come up with something on my end.

[jebeld17]

I have been having this exact same issue on my end, too.

[XXXXX] 15:32:06 INFO: Device YYYYYYY client is "syncthing v0.14.16" named "localhost"
[XXXXX] 15:32:40 INFO: Folder "xxxxx-xxxxx" isn't making progress. Pausing puller for 1m0s.
[XXXXX] 15:33:40 INFO: Folder "xxxxx-xxxxx" isn't making progress. Pausing puller for 1m0s.

I am sharing this folder between three Windows 10 PCs, all of the same architecture and on the same Windows 10 version.

[ProactiveServices]

@AudriusButkevicius Should people with the progress problem, but those without "parent path is not a directory", start a new thread? Or is it all related?

[AudriusButkevicius]

Its all related I suspect. We can open new ones once we fix the one we are aware off.

[jebeld17]

Its all related I suspect. We can open new ones once we fix the one we are aware off.

It is all related. See the image below grabbed from the same PC:

Syncthing Version: v0.14.17
OS Version: Windows 10 (x64) Pro and Home, version 1607 Anniversary
Browser Version: Synctrayzor 1.1.13, GUI for Syncthing

[koppor]

Same issue here. For two files: I renamed the parent folder. For the third file: I deleted the parent folder.

Is it possible to add a button saying "Remove" which removes the file out of the synchronization queue?

• The remote PC does not show this file
• Locally, these files do not exist

So I assume this is "just" an issue of the local database.

Unfortunately, a full rescan of the local directory does not help. Thus, the suggestion of "Remove"

[jebeld17]

@koppor I do not think this is "just" a local database issue. I get this error message on all PCs connected to the Syncthing folder.

[jebeld17]

The issue is #1521

[AudriusButkevicius]

No, this issue is global, for all versions of syncthing participating that had the vulnerability fixed.

[javmorin]

@AudriusButkevicius Just checking in on the status of this issue? It looks like the #3840 pull request stalled, but is there anything I can help test at this point?

[AudriusButkevicius]

We still haven't come up with a fix that does not introduce a vulnerability.

[jebeld17]

Why should this be closed?
This is obviously an ongoing issue that has yet to be fixed.

[jebeld17]

By the way, I was able to fix the problem on all my devices by:

1) Turning on all devices, connecting to the internet, starting Syncthing services on each - the usual.
2) Making a carbon copy backup of the share do folder, pasting elsewhere on your PC.
2a) Removing any redundant data and temporary files from that copy.
3) Copy the Folder ID from the original sync.
4) Delete the entire original shared folder (DO NOT delete the contents within)
5) Reboot Syncthing and run a re-scan of all directories. Let all of your devices talk to eachother to share status info for a few minutes.
6) Remove the shared folder from the Syncthing UI itself.
7) Make a new share, changing out the Folder ID generated for the old Folder ID we copied earlier. Save.
8) Restart Syncthing services again. Re-scan all directories just to double check.
9) Repeat these actions in this exact order on all other devices.
It is very important to keep all devices on and connected the entire time this manual rebuild process is being done.

[AudriusButkevicius]

It was closed because a xommit in the latest release addresses it.