Syncthing: Selective Sync

[Nutomic]

It would be nice to have repositories that are not stored on the local device, but where files can be accessed on demand.

This would be useful for Android (and other devices with little storage space). Android has the Storage Access Framework that allows this to work seamlessly (remote files can be accessed just like local files, and only metadata and maybe cached data is stored locally).

This is something that we might want at some point in the future, not immediately. Just to keep it in mind and plan for it. I will also have a closer look at the API and what exact functionality we need.

Edit: I should probably mention that this should be a per repo setting (not per node).

[calmh]

Sure. However do keep in mind that syncthing targets Mac, Linux and Windows primarily (with a few "tier 2" platforms like Solaris and FreeBSD _kinda_ supported) so new functionality that doesn't work or make sense on those platforms will have a hard time getting in.

[guziks]

Hello everyone, let me put my two cents in. I think this kind of functionality is actually very useful for "tier1" platforms (especially on laptops).

Use cases

Installs folder

One has "Install" folder (300Gb), wants to sync it between work and home desktops. He also has a laptop with small SSD, and wants to have access on demand to currently needed installs

Project with huge files

Team works on a project and there are huge files needed only by few people

Projects catalog

Team has one sync folder for all projects:

"Projects"
-"Project 1"
-"Project 2"
...
-"Project n"

Members are able to select (with checkbox) which projects they want to sync to their particular device.

Solution

You may already see my approach to solution:

1. Should be able to "Pause Sync" (disable file sync, but keep metadata sync). This is kind of Enable/disable a repo #183
2. Should be able to select subfolders to sync

Subfolders selection implementation i see like this:
We have "Partial Sync" button for particular sync. When it pressed the root of sync folder is paused (as in point 1), but every root subfolder continue to sync files. After that we can use checkboxes for this subfolders to pause them (unckecked == paused). And so on down the folders tree.

[calmh]

I think this might be two different things? Selecting sub-parts of a repository to synchronize, vs saving files in a magic framework thing like the one mentioned above?

[guziks]

I agree that saving files to such a framework is different thing. But there are commons with getting files. Syncthing android UI can be a document provider (in terms of SAF), the goal here is convenient interaction with other android apps. The way android UI actualy requests individual files from core app and the way core app fetches particular files, I think similar to selecting subfolders to sync.

[calmh]

Yep

[calmh]

@Nutomic Can we retitle this one "selective sync" or so, since that seems to be the part of it that is in syncthing's "domain"?

[Nutomic]

Sure ;)

[velolala]

One question here:

This issue is labeled far-future -- however I believe it is desirable to have something similar sooner. A selective sync may be achievable by simply allowing negative patterns with the ! modifier (git-style) in .stignore.
E.g. in the directory containing foo bar and baz, your .stignore could look like this:

  *
  !foo
  !baz
  *.swp 

which would lead to the selective syncing of only fooand baz (and even there no .swp files).

What do you think?

[calmh]

.stignore is in the wrong direction, i.e. it prevents file from being published from the source node that has the file.

[calmh]

To clarify why this is far-future:

• To be nice, it requires a nice UI, which is a fair bit of work. A workaround for that could be to have an .stexclude file or something, after all there is no UI for .stignore.
• There are a bunch of corner cases to handle:
  
  
  
  • What do we do with files that have already been synced when they get excluded? (including files that were not yet synced, but where in a directory that is now excluded)
  
  
  • What do we do with files that are present when the exclusion is removed? (for each of the cases is-new, is-changed, is-older)
  
  
  • probably other stuff I haven't though of
  
  

[velolala]

Okay, sorry I misunderstood then.

What I want is being able to produce different 'views' of a huge folder; therefore I'd like to select only some of the content (by negative pattern) for sync.

However, it will probably not be sufficient, since there can only be one .stignore per folder (== it would be necessary to allow for several, which leads to more complexity)...

I guess I need to open a different issue to ask for that functionality :)

[calmh]

There's a bunch of different use cases and scenarios in this ticket. I'm going to go ahead and close it with the exclude patterns now possible in .stignore (https://discourse.syncthing.net/t/excluding-files-from-synchronization-ignoring/80/5) and let people reopen tickets with clear use cases on the stuff not covered by that functionality.

[robsonsobral]

Hi! May I give you a little more to think about?

We could have two more types of folders, beside _master_: _selective_ and _on demand_.

On both we should be able to navigate through the folder right on UI in an accordion menu style. Every item on the _selective_ mode would have + signs and checkboxes. The _on deman_ mode would be simpler: + and a downwards arrow which allow the users to download the file or folder to their own computer; so, no corner cases, no sync, just download.

_on demand_ could help a lot of people who wants to download their songs or books or presentations for work or references for design, etc...

Both types would cover my use case. I have about five thousands files, on different languages, and the users just need to download some of them.

Thanks for the attention and for the useful software.

[ghost]

+1

[jonada]

This is not solved with .stignore, which is permanent.

Rather this is issue is the same/related to syncthing/syncthing-android#81.

Support must be added in both syncthing and the anroid app.

[fnkr]

+1.

.stignore does not solve the problem. Please reopen.

[AudriusButkevicius]

There is a new master ticket for this issue.

[lkwg82]

@AudriusButkevicius Which one?

[AudriusButkevicius]

https://github.com/syncthing/syncthing/issues/2491