Syncthing: Next Gen Ignores: Requirements

[calmh]

This issue replaces the ones that relate to fundamental issues with how our ignores work. The following is the up to date list of requirements:

Certain Requirements

• Must be understandable by someone who understands the interface in use by DropBox, OneDrive, etc.
• Must be a simple tree view with checkboxes
  
  
  
  • Root of tree (=folder root) checked by default
  
  
  • Can uncheck root and then check just the branches that we want synced
  
  
• Must be able to black list extensions
• Must live in config (not .stignore files)
• Must "live update". That is, if I ignore a file that was going to be pulled, we should not pull it. If I unignore a file that we are missing, it should be pulled shortly thereafter.
• Must be able to specify if ignored files should be preserved or removed on directory delete.
  
  
  
  • Per what though? File extension? Directory?
  
  
• Power user access to flexible, underlying patterns
• Be able to to remove files for folders that are not picked, as it remove things that are now ignored.
• Have inclusion and exclusion mode, ignore everything, include X, versus include everything, ignore X.

Maybe?

• Per directory file extension black lists ("never sync *.tmp")
• Per directory file extension while lists ("only sync *.doc")
• Should sync between devices
• Ability to ignore a directory by the presence of a special file in it (cachedir.tag).

Comments and discussion below. A project maintainer will keep the list above in sync as the requirements change or are clarified. Note that this is is not a list of problems with the current system, but requirements for a _new_ system.

[crccheck]

Why not also keep .stignore, but make it sync between devices as .gitignore and .hgignore do? That way, there's a _developer friendly_ way to keep files out. As someone who had to delete 100k node_modules files yesterday (not to mention .DS_Store, packages, build, .tox, etc), that would have save me 3 gigs of bandwidth and time.

There's two distinct user stories:

1. As a user to Syncthing who also knows Dropbox, I want a familiar interface for selectively syncing files so I only sync what I care about
2. As a developer using Syncthing, I want a familiar interface for selectively syncing files globally so I only sync what I care about without additional set up every new install.

[calmh]

A .fooignore file might come naturally to developers, but that's really the only demographic that expects a hidden file with a bizarre extension (think Windows). I think it's a developer accident that it exists at all in it's current form today - it's really a piece of folder configuration as anything else. Not having it in the folder itself also works better with read only folders - perhaps you want to share the contents of a DVD-ROM... (This also requires handling the non-existence of .stfolder, but there's a ticket for that.)

Whether it syncs or not is really somewhat beside the point of the existence of the file, but I do think there should be at least an option to sync the patterns, yes.

(You can do that today with a little #include hackery as well.)

---

To bring it more in line with the actual ticket; what's the actual requirement that having an .stignore on disk fulfills?

[cdhowie]

I don't care where the ignore data is stored, but patterns allow me to express ignores in a concise manner and with more flexibility than a tree option. Please at least retain this functionality, if not in a file called .stignore then somewhere else. None of the options provided in this ticket summary will meet my needs.

If the current pattern-based ignore option is removed I'm afraid I may have to stop upgrading Syncthing, as versions without pattern-based ignore wouldn't meet my needs.

[calmh]

@cdhowie What are the needs, that are not covered by the tree options and the extension white/black lists?

[cdhowie]

There is no accommodation for prefixes in the current proposal. I can't be the only one with a ._* ignore.

Neither is there a way to express a pattern along the lines of "ignore all objects named bin under directory foo" whereas we can do that today with

/foo/bin
/foo/**/bin

I'm not asking you not to implement a nicer interface for new users, I'm asking you not to sacrifice the power users on the altar of the masses.

[calmh]

Well, currently the masses are being lead to the slaughter _en masse_ as they try to work the ignore system, so some sort of balance there is required. :D

The prefix thing is a good point, and should probably be supported somehow. I need to think about the last one...

[AudriusButkevicius]

So the two could coexist, but power users will have to grasp their concepts around about the order in which it gets executed, etc.

[calmh]

Well, yes. At some point whatever the user is configuring, graphically or text based or however, is going to to get compiled into some list of match objects of some kind. Today that's a list of regexps, unfortunately that's a bit inefficient to evaluate. But something like that; whatever it ends up being could of course be exposed in some way so that they can be input directly into the config or something.

[cdhowie]

whatever it ends up being could of course be exposed in some way so that they can be input directly into the config or something.

I would absolutely be happy with that resolution, if there is a (somewhat) easy way to edit this in the GUI. Like an "advanced" button at the bottom of the simple UI that opens the list of compiled patterns, letting me fiddle with them.

(I'd be willing to contribute dev time to making that happen, as this is an important feature for me.)

[calmh]

Possibly. The way I could envision it in that case is that you'll probably have a simple default layout with a tree view and a list of extensions or something. Switch to advanced mode and you get the compiled patterns or something and can edit them at will. But once you've done that, you probably won't be able to switch back, as what you created in the advanced mode might not be expressible in the simple mode.

[cdhowie]

But once you've done that, you probably won't be able to switch back, as what you created in the advanced mode might not be expressible in the simple mode.

Same thing I was thinking. Or, you can switch back, but you'll get a warning that your manually-modified patterns will get destroyed if you do.

[AudriusButkevicius]

I think we can keep what we have today and keep it the same way it is today, just add a thing before ignores which is the dir selector.

[calmh]

Okay I added that to the requirements

[cdhowie]

Thanks @calmh. If you are in need of dev power to implement this feature let me know. I don't have a lot of time but I'm willing to spare some for ST.

[mkroehnert]

@crccheck keep in mind that storing .stignore files inside the directories to be synced leaks metadata if you copy the directory to a USB stick for example.
You might not want other people to know which program you are using for synchronization and which files are ignored.

[Schroedingers-Cat]

I think it is important to differentiate between ignoring as part of setting up a new repository and ignoring as part of setting up a new node.
When you create a new repository and want to exclude f.e. all binaries prefix_*.extension from the subdirectory /foo/bin, you don't want to see these files on any new node you sync this folder to. That's why exclusions expressed at this state as part of the folder setup should be considered an essential information and thus be synced automatically to any new node. What is excluded at this stage should never get any attention by syncthing (for that repository).
In contrast to that is ignoring as part of setting up a new node. Users likely have a device to which they don't want to sync the entire repository but only a specific subfolder. These information are device specific and should only be cared about on that node.
To summarize, ignoring at the repository level is some global filter that every device applies while ignoring at the node level is a filter that is specific to that device only.

Ignoring at the node level is probably what most users want as this is what Dropbox/Onedrive etc. offer. You have a tree list and uncheck a folder/file you don't want to download to that node. A simple interface with a folder tree, maybe even a file list, with checkboxes should suffice.
Ignoring at the repository level is what advanced users do. They care about not syncing Thumbs.db or some config.dat files of a subdirectory containing device-specific information like it's screen resolution. Even something simple like excluding files based on their extension means that they activated the option "show file extensions" in the explorer (which is what most users don't do). I don't think that these users need a more user friendly way to deal with exclusions because if they can express their needs in a form like "filter out all docx-files" they should be able to put it like *.docx. So instead of switching between simple mode and editable compiled patterns (you called it "advanced mode"), start with the simple folder tree list that configures at the node level and add a button called something like global ignore patterns or advanced mode where power users can input ignore patterns (like we have now) for the repository level.
This way, power users have fine grained control about what sort of data a syncthing repository contains and both most users and power users have a simple method to control what portion of that syncthing repository a certain node downloads/uploads to other nodes.

That is why I like @AudriusButkevicius suggestion:

I think we can keep what we have today and keep it the same way it is today, just add a thing before ignores which is the dir selector.

This is good for all users. Plus it doesn't involve rewriting the entire system, just adding the "simple node-level" sync system and it's GUI. (Not that the system would be simple to implement, but simple to use).

Also, repository-level ignore patterns don't need to reside in a .stignore file, they could also be saved within the syncthing configuration folder and synced between nodes as metadata.

[AudriusButkevicius]

Also, another thing people asked for is synced ignores and global ignores (applied to all folders)

[sneak]

If set to a non-random sync priority (e.g. newestFirst) , the puller getting hung up on a directory delete with ignored files inside it (e.g. .DS_Store) means the folder stops syncing until the logs are checked for why (and the directory manually deleted). The default of 'random' seems to mitigate the impact of this, if I'm understanding it correctly.

[katrinleinweber]

I'd like to chip in a pretty good example of visualising exclusions from a folder tree: Crashplan. The greyed-out & red-marked files are excluded via filters:

What Syncthing could improve over this design is the ability to right-click a file or folder & quickly access the menu to change the rules. E.g. the context menu could have the option in-/exclude files/folders like this one, depending on whether an already excluded or included file is right-clicked. This would open an edit filters panel on the side, in which changes trigger a preview in the folder tree. The preview could highlight & explain what would happen to the file/folder set (in colorblind-safe manner).

Including a specific file or sub-folder that is excluded by a rule should be as simple as clicking the greyed-out check-box.

[capi]

What would also be good would be a policy per directory (as advanced setting), if new sub-directories are to be added automatically or not. For some directories it's desirable to automatically add new children, for others, it's not.

[AudriusButkevicius]

Not sure what you mean by policy.

[capi]

Sorry, bad wording, policy == setting in this case, since it governs how new folders are treated.

[AudriusButkevicius]

I stilldon't understand what you are asking for, or how it relates to ignores.

[cdhowie]

He is asking for a per-folder setting on how Syncthing reacts to a new subfolder becoming available on a peer node, when using the simplified folder selector UI. Some use cases may want new subfolders to synchronize, others may not want that (they would want new subfolders ignored by default).

[AudriusButkevicius]

Ah, ok now I get it.

[capi]

Many of the above proposals for ignores are basically a selective sync. You select in a tree which directories should be synced. What is missing from this concept in my opinion, how _new_ directories are treated. Should they automatically be selected or should they be ignored by default. And I think this should be configurable per directory.

Jungledisk used to have a setting like that (in older versions at least).

@cdhowie Yes, thanks, thats basically it.

[Djip007]

For me the most important is while liste.
I want to define that I only nead to sync for exemple *.jpg...

[Djip007]

May be create 2 "filters":

• One for things that can be send to other device...
• One for things that can be receive from other device....

[Djip007]

Implementation suggestion (not IHM... )
It is sometime difficult to chose from black/or/white Liste. So implement with a list of successive rule:
for each rule define the pattern (for example all file like *.jpg) and define the action (include, exclude)

for the "pattern" it may be on name and property (file/dir)...

for rule it can be more advance (simple to implement more difficult tu use...)

• first create "categories"
• pattern act on a category+name+properties+...
• and the action send element on a "category".

At the end we just need to define if element on a category is include or exclude...

[martin-rueegg]

a remark to begin: I just skimmed over the issue, so i might have missed some points. please apologise.
ah, and an other one: it grew quite huge this post. :-D

how about saving the ignores in config.xml and (partly) sync it? following my idea:

---

config.xml as the root for ignores

there are four different types of ignore lists:

1. GLOBAL
   this list is considered for evaluation of any folder and any device
   https://github.com/syncthing/syncthing/issues/2491#issuecomment-158389511, https://github.com/syncthing/syncthing/issues/2491#issuecomment-158387548
2. DEVICE
   this list is considered for evaluation of any folder shared with the given device
3. FOLDER
   this list is considered for evaluation of the given folder shared with any device
   https://github.com/syncthing/syncthing/issues/2491#issuecomment-158387548
4. REPLICA
   this list is considered for evaluation of the given folder shared with the given device

<configuration version="x">
    <folder id="default" path="/Users/jb/Sync/" ro="false" rescanIntervalS="60" ignorePerms="false" autoNormalize="true">
        <device id="3LT2GA5-CQI4XJM-WTZ264P-MLOGMHL-MCRLDNT-MZV4RD3-KA745CL-OGAERQZ">
            <ignoreList>
                <!-- REPLICA IGNORE LIST
                items that match this list, will (not) be sent/received to/from this device
                -->
            </ignoreList>
        </device>
        <!-- ... -->
        <ignoreList>
            <!-- FOLDER IGNORE LIST
            items that match this list, will (not) be sent/received to/from any device
            -->
        </ignoreList>
    </folder>
    <device id="3LT2GA5-CQI4XJM-WTZ264P-MLOGMHL-MCRLDNT-MZV4RD3-KA745CL-OGAERQZ" name="syno" compression="metadata" introducer="false">
        <address>dynamic</address>
        <ignoreList>
            <!-- DEVICE IGNORE LIST
            items that match this list, will (not) be sent/received to/from any folder
            -->
        </ignoreList>
    </device>
    <!-- ... -->
    <options>
        <!-- ... -->
        <ignoreList>
            <!-- GLOBAL IGNORE LIST
            items that match this list, will (not) be sent/received to/from this host on any folder
            -->
        </ignoreList>
    </options>
</configuration>

The <ignoreList> element

MAY have the following elements:

• ignore
• ignoreNot
• ignoreInclude

the <ignore> and <ignoreNot> elements

mandatory attributes

MUST have the following attributes:

• match
  the part of the path that is matched, may be
  
  
  
  • path
  
  
  • dir
  
  
  • filename
  
  
  • basename
  
  
  • extension
  
  
  • (if the filename starts with a dot [e.g. .htaccess] then the fist dot would be ignored when identifying the extension.)
  
  
  • (would well-known doubble extensions [e.g. .tar.gz] match as a whole?) yes, they would, as it is only converted into a regex and would include the second dot in the pattern.
  
  
• pattern: similar to current patterns, except for the leading ! (use ignoreNot instead), unless the ~ modifier is set

optional attributes

MAY have the following additional attributes:

• size
  in case of a file, the size thereof. in case of a folder, it should only sum up the sizes of non-ignored files within the given folder. any valid size information
  (<|=|>)[0-9]*[1-9][TtGgMmKkBb]?
  
  
  
  • prefixed with <, = (default), or >
  
  
  • optionally suffixed with [TtGgMmKkBb] (defaults to M[egabytes])
  
  
• time
  any valid ISO date[-time] information #1917 prefixed with
  
  
  
  • m|c for mtime (default) or ctime respectivly
  
  
  • <, =(default), or >
(m|c)?(<|=|>)YYYY(-MM(-DD( HH(:NN(:SS)?)?)?)?)? (with timezones?)
  
  
• modifier
  can be any combination of
  
  
  
  • i case-insensitive match
  
  
  • d only applies if item is a directory
  
  
  • f only applies if item is a file
  
  
  • s only applies if item is a system file (windows)
  
  
  • h hidden file (windows)
  
  
  • l (small L) only applies if item is a symlink
  
  
  • n only applies to new items created locally ( @capi )
  
  
  • r only applies to new items created remotely
  
  
  • [DELETE] will delete the ignored files/folders upon request for the deletion of a parent folder from a remote
  
  
  • ~ pattern is a regex
  
  
• exists
  an other pattern (either without globbing, or standard globbing, or also as regex, if ~ modifier is set), that looks for a file to exist within the given directory (implies the d modifier) see #1114.
  
  
  
  • if a relative path is given, it is always relative to the path currently checked, e.g exists="CACHEDIR.TAG" would look for the existence of that file within every directory matched by all the other criteria (except for size, which would only be evaluated after exists for performance reasons).
  
  
  • if an absolute path is given, it is _relative to the folder root_ and yields true, if any match of exists falls within the directory in question.
  
  
• remote
  decides, if the rule is sent to remote devices https://github.com/syncthing/syncthing/issues/2491#issuecomment-158161786. can be any of
  
  
  
  • yes|true|1 (default). the rule is replicated to remote devices, so they can ignore sending updates
  
  
  • ignore incoming updates are silently disregarded (to avoid a transfer a "copied locally" reply could be sent)
  
  
  • conflict incoming updates will result in a local conflict, named
    {basename}.syncting-ignore.{extension} which is used as local replacement for {basename}{extension}
  
  

pattern evaluation

• the order of the rules is irrelevant
• instead, the rules are evaluated on the bases of the directory tree.
• ignoreNot take precedence over ignore
  thus, if a directory is ignored, all the items below are ignored as well, even if a ignoreNot would match
• so for any given path if no rule matches or at least one single ignoreNot, then the item is synchronized, unless it's parent is ignored.

The <ignoreInclude> element

.stignore files, that are not specifically referenced to, are ignored. this solves the problem, if different users on the same device want to share the same folder but with different ignore sets.

mandatory attributes

MUST have the following attributes:

• path
  contains the path to the file to be included.
  
  
  
  • if the path is a relative path, it is evaluated relative to the
  
  
  • folder root, if it is in config.xml
  
  
  • the files own directory, if it is an included file
  
  
  • can be an absolute path as well
  
  
  • (maybe: when starting with "folder://" then the path could be relative to an other folder definition
    [e.g. folder://default/ingore.xml])
  
  

optional attributes

MAY have the following additional attributes:

• prefix
  If given, it's value is prepended to every rule
• modifier (requires prefix)
  can be any combination of
  
  
  
  • i prefix matches case-insensitive
  
  
  • ~ prefix is a regex
  
  

file formats

if the content of the file starts with < (after BOM), then it is considered to be in XML format, and the root element MUST be <ignoreList>. otherwise it is treated like a current .stignore file.

building the ignore list

as the order of the rules does not matter according the pattern evaluation outlined above, evaluation starts at config.xml and crawls down to all ignoreIncludeed files. while doing so, i suggest that two tables with the following information is built:

ignoreFiles

• id
  autonumber
• source
  
  
  
  • [NULL] for config.xml
  
  
  • full path of included source file
  
  
• parent
  [id] of file that included this file
• skipped
  
  
  
  • 0 for the first entry of one given source
  
  
  • 1 for any subsequent entry to an already existing source
  
  

ignoreRules

• id
  autonumber
• source
  ignoreFiles' [id]
• folder
  
  
  
  
  • [NULL] for GLOBAL and DEVICE ignore list
  
  
  • FolderID for FOLDER and REPLICA ignore list
  
  
• device
  
  
  
  
  • [NULL] for GLOBAL and FOLDER ignore list
  
  
  • DeviceID for DEVICE and REPLICA ignore list
  
  
• pattern
  pattern converted to a regex and prepended with prefix
• modifier
• size
• sizecomparison sum of
  
  
  
  • 0 ignore
  
  
  • 1 smaller than (<pattern)
  
  
  • 2 equal as (=pattern)
  
  
  • 3 bigger than (>pattern)
  
  
• time
• timecomparison sum of
  
  
  
  • 0 ignore
  
  
  • 1 older than (<pattern)
  
  
  • 2 equal as (=pattern)
  
  
  • 3 younger than (>pattern)
  
  
  • 10 mtime (default)
  
  
  • 20 ctime
  
  
• type
  showing if ignore or ignoreNot (or later other rules, see blow)
  
  
  
  • 0 ignore
  
  
  • 1 ignoreNot
  
  
• exists
  exists pattern converted to a regex
• remote
  representing the remote value of <ignore> and <ignoreNot>
  
  
  
  • 0 replicate
  
  
  • 1 ignore
  
  
  • 2 conflict
  
  

| id | src | fldr | dev | pat | mod | s | sc | t | tc | n | exists | r |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | | | | .+\.pdf$ | | | 0 | | 0 | 0 | | 0 |
| 2 | | | | (^|/)hallo\.txt$ | | | 0 | | 0 | 0 | | 0 |
| 3 | | | | (^|/)foo/bar$ | d | | 0 | | 0 | 0 | | 0 |
| 4 | | | xx | .* | | 20*2^20 | 3 | | 0 | 0 | | 0 |
| 5 | | ff | xx | /foo/bar/.+$ | | | 0 | | 0 | 1 | | 0 |
| 6 | | f2 | z | .* | f | | 0 | | 0 | 0 | | 0 |
| 7 | | f2 | z | .+\.jpg$/i | if | | 0 | | 0 | 1 | | 0 |
| 8 | | f2 | z | .* | d | 0 | 2 | | 0 | 0 | | 0 |
| 9 | | f2 | z | ^/stuff/movies/
this_one_movie_
I_want_to_watch
\.avi$ | f | | 0 | | 0 | 1 | | 0 |
| 10 | | f2 | z | .* | d | | 0 | | 0 | 1 | ^/stuff/movies/
this_one_movie_
I_want_to_watch
\.avi$ | 0 |

1. global ignore for pdf files
   GLOBAL: <ignore match="extension" pattern="pdf" />
2. global ignore list for files named "hallo.txt"
   FOLDER: <ignore match="name" pattern="hallo.txt" />
3. folder ignore list for content of directory "foo/bar" (at any level)
   FOLDER: <ignore match="name" pattern="hallo.txt" modifier="d" />
4. don't send any file bigger than 20MB to device xx
   DEVICE: <ignore match="path" pattern="**" size=">20M" />
5. however, do send files from folder "ff" in subfolder /foo/bar to xx even if bigger than 20MB
   REPLICA: <ignoreNot match="path" pattern="/foo/bar" />

@Djip007's whitelist cold be implemented like the following for his device z for folder f2:

• ignore all files
  FOLDER[f2]: <ignore match="path" pattern="**" modifier="f" /> (row 6)
• do _not_ ignore all .jpg-files (case insensitive)
  FOLDER[f2]: <ignoreNot match="extension" pattern="jpg" modifier="fi" /> (row 7)
• ignore all folders, that do not contain any jpg-files, and as such have size=0
  FOLDER[f2]: <ignoreNot match="path" pattern="**" modifier="d" size="0" /> (row 8)

also @AudriusButkevicius' https://github.com/syncthing/syncthing/issues/1185 could be solved:

• ignore all folders, that do not contain any non-ignored files (and as such have size=0)
  FOLDER[f2]: <ignoreNot match="path" pattern="**" modifier="d" size="0" /> (row 8)
• do _not_ ignore /stuff/movies/something/this_one_movie_I_want_to_watch.avi
  FOLDER[f2]: <ignoreNot match="path" pattern="/stuff/movies/this_one/movie_I_want_to_watch.avi" modifier="f" /> (row 9)

an other and better approach would be, of course, to use the exists attribute:

• ignore all folders, that do not contain any exists files
  FOLDER[f2]: <ignoreNot match="path" pattern="**" modifier="d" exists="/stuff/movies/this_one/movie_I_want_to_watch.avi" /> (row 8/10)

building the ignoreRules

before including a file, it's entry in ignoreFiles is created in any case. if, however, there is already an other entry with that path, the file is not processed again, instead the flag skipped is set to 0.

now all the source files should be regularly scanned or a file system watch should be established, in order to detect changes. when a file is changed, all it's rules are re-evaluated. this means, that

• the corresponding entry in ignoreFiles and
• all it's "children" should recursively be deleted. then,
• all the entries in ignoreRules that have no longer a corresponding entry in ignoreFiles should also be deleted.
• now the file is re-parsed
• after completion, all source that have no [skipped]=0 entry, parse the given file and set one entry to [skipped]=0. this would be the case, if an inclusion has been removed during modification.
• after successful re-build of the ignoreRules, this is populated to the active devices, where the device and shared folders and the remote flag are taken into consideration.
• information retrieved from remote can be added to the same table, with the DeviceID as source. as such, all ignore information is in one place for evaluation.

further considerations

sync status

populating the ignore list to the remote devices could help to show accurate sync status as the remote site can also evaluate the ignore list and see if any update needs to be sent.

built-in file list

there could be a (documented) built-in list of patterns [see https://github.com/syncthing/syncthing/issues/1631#issuecomment-107966413 and https://github.com/syncthing/syncthing/issues/1631#issuecomment-107966480] which could still be overwritten with ignoreNot.

filter instead of ignore

if the <ignoreList> would rather be a <filterList>, then other rules could later be incorporated. E.g. something like a

• <prefer> element, that is basically the same as an <ignore> element, with an additional mandatory property target which would point to which side a conflict would be resolved.
• or similarly a <force> element, that would always favour for one side, similar to the "Master Folder", but not necessarily for the entire folder. (see also unison)
• also a <follow> element to define if a symlink shall be transferred as a symlink or it's target
• or also <version> to define File Versioning

---

thank you for taking the time to read. hope this can become a useful contribution.

[AudriusButkevicius]

One of the requirements is this being user friendly, and the ideas do not seem to be user-friendly as you have to learn a new meta-language as well as it is close to impossible to have a noob friendly UI for this.

No other software in the same activity space implements such a mad feature set, hence nor do I think that we should try really hard either.

I think it's better to get at least the basic thing off the ground to start with. By the end of the day, there is only a few guys around working on this, and not a team of 20, so the more complicated we make this (raising expectations), the less likely it will happen.

[martin-rueegg]

thank you @AudriusButkevicius, for reading or at least skimming over my post.

No other software in the same activity space implements such a mad feature set, hence nor do I think that we should try really hard either.

ok, that is your opinion. [unison](http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html, which is in that activity space since, well v2.9.1 around 2002-04, has implemented a slightly less mad feature set (ignore Name, ignore Path and ignore Regex) as of v1.169. it also does all the other mad stuff like

• ignore xxx add a pattern to the ignore list
• ignorenot xxx add a pattern to the ignorenot list
• nocreation xxx prevent file creations on one replica
• nodeletion xxx prevent file deletions on one replica
• noupdate xxx prevent file updates and deletions on one replica
• backup xxx add a pattern to the backup list
• backupcurr xxx add a pattern to the backupcurr list
• backupcurrnot xxx add a pattern to the backupcurrnot list
• backupdir xxx directory for storing centralized backups
• backuploc xxx where backups are stored ('local' or 'central')
• backupnot xxx add a pattern to the backupnot list
• backupprefix xxx prefix for the names of backup files
• backups keep backup copies of all files (see also 'backup')
• backupsuffix xxx a suffix to be added to names of backup files
• copyonconflict keep copies of conflicting files
• follow xxx add a pattern to the follow list
• force xxx force changes from this replica to the other
• forcepartial xxx add a pattern to the forcepartial list
• immutable xxx add a pattern to the immutable list
• immutablenot xxx add a pattern to the immutablenot list
• maxbackups n number of backed up versions of a file
• mountpoint xxx abort if this path does not exist
• nocreationpartial xxx add a pattern to the nocreationpartial list
• nodeletionpartial xxx add a pattern to the nodeletionpartial list
• noupdatepartial xxx add a pattern to the noupdatepartial list
• prefer xxx choose this replica's version for conflicting changes
• preferpartial xxx add a pattern to the preferpartial list

an other app Synchronize Ultimate for android supports the following options (include/exclude, both case sensitive and insensitve):

1. File extension equals
2. File extension starts with
3. File extension ends with
4. File extension starts and ends with
5. File extension regex
6. File name contains
7. File name starts with
8. File name ends with
9. File name starts and ends with
10. File name regex
11. File size larger than
12. File size smaller than
13. File size between
14. File created before
15. File created after
16. File created between
17. File edited before
18. File edited after
19. File edited between
20. File created age older than
21. File created age newer than
22. File created age between
23. File edited age older than
24. File edited age newer than
25. File edited age between
26. File hidden
27. File not hidden
28. File writeable
29. File not writeable
30. File readable
31. File not readable
32. Folder name contains
33. Folder name equals
34. Folder name starts with
35. Folder name ends with
36. Folder name starts and ends with
37. Folder name regex
38. Folder created before
39. Folder created after
40. Folder created between
41. Folder edited before
42. Folder edited after
43. Folder edited between
44. Folder created age older than
45. Folder created age newer than
46. Folder created age between
47. Folder edited age older than
48. Folder edited age newer than
49. Folder edited age between
50. Folder hidden
51. Folder not hidden
52. Folder writeable
53. Folder not writeable
54. Folder readable
55. Folder not readable
56. Path contains
57. Path equals
58. Path starts with
59. Path ends with
60. Path starts and ends with
61. Path regex

in addition, it adds several conditions that apply to any given rule.

One of the requirements is this being user friendly, and the ideas do not seem to be user-friendly as you have to learn a new meta-language as well as it is close to impossible to have a noob friendly UI for this.

obviously you are able to modify xml, as you do in config.xml. creating a basic interface e.g. for the directory tree selection cannot be that difficult.

and as far as the "new meta-language" is concerned, globbing is already supported (and has to be learned) and RegEx can be considered to be standard, i guess. so is xml, which i probably wouldn't have chosen for a config file (or at least adhered to the xml specification, which requires it to start with the <xml ... tag ...), but xml was already given.

and yes, the basic things is always good. however having a vision as well. it might be implemented in the way proposed, without implementing everything at once. having an extensible infrastructure might help avoiding programming effort on the long run.

but sure, i'm not one of your team and my proposal seems to be trashed as mad. even though it might address several open issues at once. on a personal level, a response like yours, @AudriusButkevicius does not encourage the community to contribute. whatever ...

[AudriusButkevicius]

The proposal is perfectly valid, and I am not trashing it. Mad means insane or bad shit crazy as in complex, so I am just being realistic of what is the most important thing that we need right now.

I understand where you are coming from, and the features you are describing, but I feel we should aim for something my mother could understand, and she definately doesn't dig globbing, regexp or xml, so let's get the most important thing in place first, so that people who don't want to learn the meta language could use the thing, as it's already very cryptic.

Also, you have to understand that unison is a one off fire and forget thing which has no state, nor it cares about any state.
Once you involve a decentralised, stateful application, these things become much more complicated, and I am somewhat not fond of trying to debug someone's 90 different rule set, which comes combined from per-folder, per-device, global lists, yet I see why you would want it, as I assume advanced users want syncthing to be the swiss army knife, with more knobs than we as developers can later meaningfully support.

[martin-rueegg]

ok, thanks for clarification.

just a not on unison: in fact, it does have a state (unless you tell it to ignore that `-ignorearchives') and it does also support decentralised updates, but only on a peer-to-peer basis. if you set it up with fsmonitor, it is also supporting real-time, multidestination updates. so it is, IMHO, quite close. with some drawbacks and some advantages of course ...

regarding your argumentation for the ease of use: i think there are two different things that need to be regarded:

1. user interface (and usability): i agree, should be usable without RTFM
2. option storage

my proposal was regarding the later part. and i was merging requests that

• have been gathered from other issues
• from my long term experience in synch'ing issues

so it could be an approach, that in the first step, out of my proposal, only the following subset would be implemented:

• only ignore and ignoreNot elements are supported, with only the following features set
  
  
  
  • match="path" (as is now)
  
  
  • pattern current globbing is supported as-is (also for backwards compatibility)
  
  
  • modifier only i (case-insensitive) is supported
  
  
• only FOLDER (and maybe GLOBAL) filter lists are supported
• only <ignoreInclude path=".stignore"> is supported and enabled by default (for backwards compatibility; or, the rules are imported during upgrade and then the ignoreInclude can be trashed - which however would break backwards compatibility.)

from the GUI, only the following would be supported:

• enable/disable <ignoreInclude path=".stignore">
• add non-globbing file/folder from existing path, with the option to ignore=yes|no and case-sensitive=yes|no resulting in a entry like:
  <ignore[Not] match="path" pattern="/folder/filename.ext" [modifier="m"] />
• only exact these cases would be also displayed in the UI; and manual added or existing globbing would be presented in a textbox, instead of the tree view.

however, internally the evaluation of patterns could be done the way i proposed, specially regarding the propagation of the exclude rules to the peers. yes, i'm aware that there is the issue of disclosure of file-tree structure information. however, i think it's still ok to do so as:

• the information will not be saved on the remote device, only kept in memory
• you also _do_ trust the other device up to a certain degree, otherwise you would probably not share files in the first place.

now, all the other mad/cool/crazy stuff could be added later, if still wanted. the thing would only be to start an approach, that might be extensible later on ...

[Zorgodon]

Yeah, If Syncthing started using some sort of system using placeholders or virtual structures for selective sync such as in BitTorrent Sync, it could totally replace it. But currently, with the clumsiness of the ignore system, I, as a normal home user of Syncthing, cannot use it to selectively sync music, movies and documents to my Android from over 100GB folders without manually editing .stignore every time I want a new file!!!

[jcrben]

Perhaps we should split out some of the features into separate issues or threads on the forum to gauge interest? Should feature requests generally be submitted on Github or the forum?

Personally, I want "Per directory file extension while lists ("only sync *.doc")" and also find .stignore files inside directories to be rather helpful. I know Git and I prefer something similar as well...

[Losiara]

Let me express my vision.
I think that such feature must be very flexible:

• White list should be possible as black list.
• Each dir may contain own .stign file. If there is no one, than existable .stign file in one of parents dir relevant than higher level one. Also may be .nost file which repeals all higher level sync rules.
• low tree level .stign file is more relevant than higher level one. There should be boolean algebraic expression which apply contradictable rules from lower directory if specified option "Apply rules from higher level directories".
• .stign file may be syncable may be not. So every machine has own rules.
• For convenience in gui there should be a form with dir tree and text field with glob rules appendable for every directory.

[LubosKolouch]

Maybe for beginning it could start with selecting/deselecting subfolders for sync? Like owncloud client does?

[madumlao]

Just want to drop my 2 cents on this.

---

tree-view as a UI for .stignore

The following requirements _appear to be related_ :

• Must be understandable by someone who understands the interface in use by DropBox, OneDrive, etc.
• Must be a simple tree view with checkboxes
• Must live in config (not .stignore files)

in that they appear to be aiming for user simplicity. It appears as if the main rationale for wanting to ditch .stignore is that the pattern creation / matching in .stignore is too complex for the average end-user.

But I don't think that .stignore _necessarily_ adds complexity for the end-user. If the goal is to make a tree-view ignore / include list, the tree-view could simply be a parser / writer of stignore rules. It doesn't necessarily have to do away with .stignores. Some git clients, for example, allow you to right click on a folder or file and "add to .gitignore" which does what it says on the can and just adds a .gitignore rule.

After all, most end-users don't actually write their syncthing config, they generate it from the syncthing UI prompts. .stignore files can be generated and parsed the same way.

A little UX massaging would allow you to add a menu with options like:

• ignore this file
• ignore files like this.doc... (menu)
  
  
  
  • named exactly this.doc
  
  
  • also ending in .doc
  
  
  • containing __ in the name
  
  
  • custom rule (prepopulated with filename)
  
  

All of which just add the relevant .stignore rules which are relatively easy to generate. So the power users still get their behind-the-scenes thing (can edit custom stignore rules by hand) whereas average users can teach their moms to do it.

As for the visualizer, an earlier suggestion gave a filesystem tree which the end-user could browse. Files hit by an ignore rule would be colored differently, and you could right-click to "unignore" them or otherwise customize the rule.

The above idea is really orthogonal to other features requested here. Like .stignore files could do with more features if you want, just demonstrating that you don't have to do away with .stignore just to cater to a great user interface.

---

Multiple .stignore levels

In my opinion, there is value in putting ignore functionality in one or all of (1) the config file, (2) a shared .stignore-like file/config, (3) a node-local .stignore file (what we currently have). To my mind they work as levels of priority, with the most specific target (node-local .stignore winning).

• in the config: can apply to the node as a whole. e.g. global excludes. "This node never downloads videos because it's my phone and they dun fit here"
• in the share itself (synced .stignore-like file): "none of the nodes in this share should sync their these videos because we're a video production company and we're not going to sync all the intermediate outputs of the rendering process as it's too expensive"
• in the node-local .stignore file: this "this particular folder on this node only _does_ download videos because even if it's my phone, these are small videos"

You can always write rule generators / readers for all of the above to display / edit them pretty. I think allowing power users to edit the rules directly does not detract from that.

---

.stignore should be inside the share directory by default

I also think that keeping the node-local .stignore in the folder itself is generally a good thing. This guarantees that copying a syncthing folder (something a naive end-user might do) is an atomic operation that includes ignore rules. If a user, e.g. copies a syncthing folder by USB to another node, then on that node uses the copied directory as a syncthing share, you probably _want_ the .stignore to be on the node as well. I shudder to think what might happen to 3 nodes with the same share:

• Alice ignores B
• Bob does not ignore B
• Charles loads a copy of Alice
  
  
  
  • Charles's B is empty (because Alice ignored it)
  
  
  • Charles's B is _more recent than Bob's_
  
  
  • Charles's B is added to the share
  
  

Would loading Charles cause Bob to delete B? I hope as heck not. If the .stignore rules are always copied alongside the directory copy, you would ensure that doesnt happen.

Furthermore, if your syncthing interface parses .stignore rules that come with the folder, then when the user opens' Charles' interface and navigates to wherever the ignore rules should be, they would see the ignore rules from Alice. I'd argue that completely matches user expectations as they copied the folder from Alice. If they're reliant on the UI for generating the rules, they wouldn't know that they would also have to copy some other snippet of config to get it to work as expected.

[tomachinz]

on Mac OS it's vital that .DS_Store files are ignored application wide. I can't think of any reason why you'd want to sync those files yeah? The Mac app should support ignoring them by default, and also, cleaning up the mess it created: one of my repos has 2,850 of them....

-rw-r--r-- 1 tom staff 12292 25 Dec 21:35 .sync-conflict-20161228-025941.DS_Store
-rw-r--r-- 1 tom staff 28676 1 Mar 23:11 .sync-conflict-20170309-001227.DS_Store
-rw-r--r-- 1 tom staff 26628 22 Mar 12:53 .sync-conflict-20170322-125753.DS_Store
-rw-r--r-- 1 tom staff 28676 28 Mar 13:42 .sync-conflict-20170330-214609.DS_Store
... 2,800 files later...
-rw-r--r-- 1 tom staff 6148 23 Dec 21:32 .sync-conflict-20161224-024618.DS_Store
-rw-r--r-- 1 tom staff 6148 24 Dec 02:27 .sync-conflict-20161224-030453.DS_Store
-rw-r--r-- 1 tom staff 6148 16 Oct 2016 .sync-conflict-20161223-223242.DS_Store

sync-conflict.txt

[tengwar]

Regarding what @tomachinz said... I think another requirement would be to have sane defaults that ignore .DS_Store, thumbs.db, files ending with ~ and other kinds of garbage that is only relevant locally.

IMHO both casual and power user would appreciate it. And if you have a reason to sync these kinds of files, you know enough about computers to go to settings and disable this default rule.

[cdhowie]

I'd agree with @tengwar and would also suggest that the defaults be across the board and not platform-specific. I'm not sure if this is the correct place to discuss that list of defaults, but vim swapfiles (*.sw?) might be a good addition, too.

[AudriusButkevicius]

There was an extensive list of these on the forum.

[AudriusButkevicius]

Updated with a few items from #4397

[madumlao]

Regarding #4397

Ignore lists support inversion, so you can ignore everything apart from the content you listed. Furthermore you can have centralised ignores by using include directive and just including a normal file that is synced. That means ignores will be the same across on all devices.

Wanna remind you that this is bad UX, because you have to manually set up the include on each new share and you're certain to forget it on _that one share_.

A more sane UX would be to have one of the levels of the ignore system should be private to the node (the current .stignore) and another should be global to the share (e.g. .stignore_synced). You can approximate this behavior today by always putting #include .stignore_synced in every share, but there is going to be a share where you forget this. Add to that the race conditions of adding the ignore _after_ you add the share.

[infectormp]

+1 for CACHEDIR.TAG support

[ibizaman]

@madumlao I agree completely with your whole comment.

About parsing the file, I would go one step further. When you go in the ignore patterns menu, you would see two tabs. One Basic and one Advanced. The Advanced would be the one we have now: you edit the .stignore file directly. The Basic one would be the tree view, _and you could only switch to the basic view if the .stignore has rules parseable by the tree view_.

The idea is to copy the behavior of some tools I use regularly, like Jira for example. It has a JQL language allowing you to query and filter the tickets in complex ways. You can either write the JQL directly or use the dropdown buttons to generate a simple JQL behind the scenes. And you can only use the dropdown buttons if the JQL is simple enough, otherwise the basic mode is unaccessible.

I think it's a good idea to proceed like this because that way we can start with a simple set of rules for the basic view (the tree view), while not restraining advanced users and while allowing us to add features later on, when users had a try.

Anyway, there's no reason we should implement everything at once. What about a small but functional proof of concept? I would be happy to give it a go. What do you think of this first requirement list for the PoC:

• Basic / Advanced tab
• Advanced tab is what we have now
• Basic tab is the tree view
• Tree view lists the whole _global_ folder and file tree, it won't show the .stignore file as it's never synced anyway
  
  
  
  • next to each file/folder, there's a checkbox
  
  
  • if the checkbox is checked, the file/folder is synced
  
  
  • if checkbox is unchecked, the file/folder is ignored
  
  
  • by default all checkbox are checked, generated .stignore is empty in this case (make it compatible with current behavior)
  
  
  • if a checkbox get unchecked, all checkboxes inside that folder get unchecked too (1)
  
  
  • when an unchecked checkbox get checked:
  
  
  • if its parent folder was unchecked, check it (do that recursively)
  
  
  • check all the children folders
  
  
• When clicking on save in the Basic tab, a new .stignore is generated and replaced
• Basic tab can't be opened (greyed out) if .stignore has too complicated rules. If it has, the user is an advanced user anyway.
• The generation/parsing of the .stignore file would simply work like this: for all unchecked file/folder, the .stignore would have a rule with the absolute path of the file/folder. With the special case that it won't generate any rule to exclude children of an excluded folder, because it's not needed thanks to (1).
  That's all the parser would support, if there's another rule not matching the absolute path of a file or folder in the global folder and file tree, the Basic tab is greyed out.

This PoC is not that small, but it's the smallest set of requirements I could come up with which are consistent and would IMO make for a good user experience.

(1) This is because of the following rule, quoting the doc (empahsis mine):

A pattern beginning with a ! prefix negates the pattern: matching files are included (that is, not ignored). This can be used to override more general patterns that follow. Note that files in ignored directories can not be re-included this way. This is due to the fact that Syncthing stops scanning when it reaches an ignored folder, so doesn’t know what files it might contain.

What do you think?

[AudriusButkevicius]

This needs more than that, as this assumes a default behaviour of excluding. It essence, any new directory added would automatically be included, which might not be what users want, so we need a way to invert the whole selection process.

I think this needs to be split into two separate matching engines, one for tree view, second one for patterns.

[madumlao]

This needs more than that, as this assumes a default behaviour of excluding. It essence, any new directory added would automatically be included, which might not be what users want, so we need a way to invert the whole selection process.

Isn't this already accomplished by ** followed by more specific !foo !bar rules?

[ibizaman]

Taking into account it would be a PoC for a tree view version of a simple .stignore file, I don't think we should tackle the inverse parser here. Not saying it wouldn't be a good feature to have, but I don't see any reason why the PoC requirements should include that.

Also what @madumlao said. Which wouldn't work in the Basic tab of the PoC as it would be deemed too complicated for the parser.

That being said, I don't understand what you mean by "this" assumes a default behavior of excluding. Is "this" the PoC requirements? The PoC would _only_ be a parser of .stignore. When a new folder or file is created, the current behavior will kick in: because it's not added in the .stignore file, it will show up checked in the tree view. So I would've said the opposite: the PoC assumes the default behavior of including.

Edit: what's more, we could easily do both. The Advanced tab would just stay as-is while the Basic tab would have an additional checkbox at the bottom "include new directories by default". Depending if it's checked, we would generate a .stignore like I proposed or like @madumlao proposed. Again I wouldn't do this in the PoC, but the cool thing with this IMO is that we let the user choose what he wants, while not changing other parts of syncthing since both behaviors are already possible right now.

[tengwar]

The basic tab + advanced tab concept seems nice. Later more widgets
could be added to the basic tab to support more rule types. For example
rules that ignore certain file extensions could be shown as a list of
ignored filetypes. And patterns that use just a wildcard (and not
advanced regexes) could be shown with a nice inline graphical
representation of the wildcard (I believe macOS terminal search does
such a thing).

Default rules that would be shipped with Syncthing (ignore Thumbs.db,
ignore Vim swapfiles, etc.) can be complicated so GUI could show them as
a list where items wouldn't be patterns, but descriptions of what these
patterns do. Each item would have a checkbox to enable/disable it, but
to edit the underlying patterns you would need to go to the "advanced" tab.

[Hoeze]

What is the current state of this?

[AudriusButkevicius]

None, nobody is working on this.

[erdnaxeli]

Isn't this already accomplished by ** followed by more specific !foo !bar rules?

Actually it's the reverse, the first rule that matches is used. To only sync folders foo and bar:

!/foor
!/bar
**

This is the way of selecting files to sync as of today.

[ibizaman]

Actually, I am, very very slowly though. I have the UI working with the tree view coming from syncthing's global state. What's left on my side is the correct parsing/formatting of the rules.

I'm actually so slow on this that I have some merge conflicts. No real blockers, just need to find a big enough chunk of time to power through this.

[paulwalko]

I was looking around at things a found the 'selective' branch and although it's over 3 years old I figured it might be somewhat useful since nobody's mentioned it yet.

[grahamPegNetwork]

@ibizaman
Awesome to hear, this is a high value feature for me. I tend to run file sync software on desktop, laptop, server and mobile. One highly wanted feature is to have repositories for music, audio books, ebooks etc, and be able to easily "check out" a folder from the directory tree (a particular album/book) on mobile. Currently I'm stuck with streaming from plex which is less than ideal. I previously used Seafile for this and believe Owncloud/Nextcloud had similar functionality.

I'd be glad to offer a modest tip via ETH/PayPal if this feature makes it to my phone. Ping me if this gets added. :-)

[ibizaman]

@smartlaw-jenkins that's very appreciated. Actually my incentive is to make my wife use syncthing. 😅 For her to use something it must be pretty - I can't blame her - and this a missing feature that is needed for the adoption of syncthing.

[ibizaman]

I fixed the merge conflicts, code is nearly good for an alpha PR.

There's one thing I'm struggling with which is testing the parser that would read the rules in the textarea and tick the boxes in the tree.

By struggling for testing I mean manual tests are super slow I don't know how to add automated tests on the js code.

So, how can I test the js code? I don't need any browser or user interaction, just some input/output testing on several js functions.

If easier I can post the PR as it is now.

[AudriusButkevicius]

Sadly we have literally nothing for JS testing.
An early PR can point out flaws before you sink more time into a potentially suboptimal implementation.

[Avamander]

I followed the chain of issues to here and it seems that one of the original ideas has gotten lost. Namely, on-demand file download. This would mean everything is in a blacklist until the user selects the file to be downloaded and displayed. The use-case of this would be basically serving files to clients, be it music, pictures, movies or something else, clients that lack the space and bandwidth to sync entire media libraries to.

A few more specific examples:

• I would love it if I could instruct the Android app to download certain song from my music library with just one tap, I don't want to manually pick songs I want on my device especially if symlinks aren't supported, the solution at the moment is a waste of disk space, and my time and effort
• My laptop to sync one folder of the project I'm currently working on instead my entire collection of projects that takes hundreds of gigabytes

[ibizaman]

@Avamander With the latest idea, you would need to:

1. add a rule with * to ignore all
2. tick the checkboxes in the tree for the songs you want to download

Could you explain the following more? I would say these contradict themselves.

instruct the Android app to download certain song from my music library with just one tap, I don't want to manually pick songs I want on my device

Also, from what I understood, you can already edit the .stignore file to do what you want, although it's something manual and very tedious at the moment.

EDIT: by new ideas I'm talking about https://github.com/syncthing/syncthing/pull/5132

[Avamander]

@ibizaman

instruct the Android app to download certain song from my music library with just one tap, I don't want to manually pick songs I want on my device

The most important part here is the with just one tap, the 1. add a rule with * to ignore all makes syncthing ignore everything downloaded (right?) and that would mean unchecking a box only has the effect of not syncing changes of the file - it doesn't delete it from the device. The closest example of the functionality I have in mind is GDrive's "Available offline" button and if I have understood everything correctly the latest idea doesn't behave exactly like that.

[LubosKolouch]

MS's OneDrive does the same thing with W10... "Save space and download things as needed", or some similar name.

[AudriusButkevicius]

@Avamander there is syncthing-lite for android that probably suits better what you want, I don't think we'll have that part of main syncthing. Also, W10 one-drive placeholder functionality is out of the question as it depends on some undocumented win32 apis.

[Hoeze]

I'd love a solution which shows all the files but only keeps a portion of them locally.

What about some FUSE solution?
I thought FUSE has already been ported to Windows.
Android should not be that of a problem, I'm pretty sure it already has appropriate API's.

[AudriusButkevicius]

There is syncthing-fuse already a thing, yet I am not sure how much it works, and what the performance is like, anyways, both of these are definately not canidates to become part of syncthing.

[ibizaman]

One could always use https://rclone.org/cache/ in the meantime. I didn't try it myself though.

[rptb1]

I reached here from #193, which I find by searching for "selective sync". I'll just state what I personally want: Resilio style selective sync as on Android. I don't want to have to edit text files with patterns, or even have such files. I want to (for example) point Syncthing at my entire music collection on my NAS, then be able to pull down tracks (files) or albums (folders) by picking them from a searchable tree. Once synced, they stay in sync if changed. If I delete them on the Android end, they stay deleted until I ask for them again. In this way I can use any number of source machines as libraries of things I might need quickly, without fuss, without having to remember where they are, on any number of other devices. Thanks!

[grahamPegNetwork]

Further discussion regarding implementation has moved to #5132

[nekr0z]

Not that my opinion should really matter, but I'm all for "ability to ignore a directory by the presence of a special file in it", provided that it can be user-tuned, too. What I mean is not only cachedir.tag triggering this behaviour, but rather anything explicitly defined by user in .stignore (or whatever new model for expressing user's wishes on this subject).

Let's say a user can put some condition in .stignore that would tell Syncthing to ignore any directory containing some file named, say: ".stdearsyncthingdontevertouchanythinginthisdirectoryprettyplease" - that would be an ideal solution for the issue of ignoring git-enabled projects that seems to pop up in forums now and again.

[fdietze]

In my head it makes sense to have two different concepts, supported at the same time:

1. One per directory-ignore file that is synced (like .gitignore, which is also committed). -- Maybe even a setting to activate .gitignore support.
2. One per client configuration (file/syncthing config) which is not synced and specifies which files will be downloaded

Trying to mix these two concepts is not possible in my opinion and seems to create strange tradeoffs.

[poisonborz]

As mentioned in #4045, it should be possible to add an ignore config to not only a synced folder, but to the device/client (for effect over all files and folders).

This would give a fair boost in usability: no need to separately copy/maintain different ignore files, or different clients could only listen to specific files from the same source (as in music-only, image-only clients).

[amra]

To boost this task I supported this issue with $50 on bountysource.com: https://www.bountysource.com/issues/28398667-next-gen-ignores-requirements

[totti4ever]

To boost this task I supported this issue with $50 on bountysource.com: https://www.bountysource.com/issues/28398667-next-gen-ignores-requirements

thanks for mentioning the bountysource, which let me top up a bit as well! And I saw another bounty coming in today :-)

[shish]

Is there a way that I can donate just towards the "Maybe: Should sync between devices" part? I don't care about any of the other features being discussed in this thread, but I would pay $200 for that one feature without a second thought :)

(I know there are workarounds, like I can "just remember to manually type #include .stglobalignore every time you set up a new folder or a new host" - but I've been forgetting to "just" do that one simple non-obvious step every time I set up a folder for coming up to 5 years now, and I don't think I'm going to start remembering it any time soon :P )

[jkbecker]

soooo.... is any of this realistically ever going to materalize? I had to switch to Resilio Sync because selective sync is a hard requirement, and although none of the solutions discussed above are on Resilio's level, I would definitely switch back if only a basic selective model was available...

[amra]

Is there a way that I can donate _just_ towards the "Maybe: Should sync between devices" part? I don't care about any of the other features being discussed in this thread, but I would pay $200 for that one feature without a second thought :)

(I know there are workarounds, like I can "just remember to manually type #include .stglobalignore every time you set up a new folder or a new host" - but I've been forgetting to "just" do that one simple _non-obvious_ step every time I set up a folder for coming up to 5 years now, and I don't think I'm going to start remembering it any time soon :P )

I am also interested in one sub-feature. I do not see any other way how to boost the development but support all ignore features.

[imsodin]

As the sync part is just a maybe point in this issue, and there may (sic!) be a good enough solution to achieve it within the current system, lets discuss about that in a separate place (and open an issue if it turns out to be doable/actionable): https://forum.syncthing.net/t/syncing-ignore-patterns/14272

I already put up some initial thoughts there as an incentive to actually move the discussion there ;)

[gabriel-fallen]

What do you think about ignoring everything mentioned in .gitignore/.hgignore/etc. found in the directories subject to synchronization, relative to the ignore file? The same way recent utilities like SIlver Searcher, Ripgrep and fd-find behave by default?

Obviously analogous to the said programs we need a way to switch off this default, but as long as this feature is developers-aimed we can just place special un-ignore files in appropriate directories.

[tengwar]

@gabriel-fallen IIRC the utility-independent file name that most tools like ripgrep support is .ignore.

And in the first post there is a requirement that says:

Must live in config (not .stignore files)

So I don't think a file-based solution is going to be accepted.

[gabriel-fallen]

@tengwar first, Ripgrep supports exactly .gitignore: https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#automatic-filtering

Second, the point is to be tool-dependent: support .gitignore, .hgignore and some other existing popular ignore files from revision control system and such.

The thing is I'm syncing a directory with some Git repos inside it, so the repos already have .gitignore files containing things I'm not going to sync. So I had to go through all these ignore files and copy-paste patterns into Syncthing, while Syncthing can handle that completely automatically and transparently.

[amra]

Second, the point is to be _tool-dependent_: support .gitignore, .hgignore and some other existing popular ignore files from revision control system and such.

This is fine. Settings in file is simple and strait forward. But .gitignore/.hgignore only set files to ignore. I would like to configure files/folders to download (opposite to ignore) and ignore the rest. That is not supported in .gitignore.

An example:
I share library (music/video/images) between PC and smartphone. Smartphone allows just a subset of the library (to save space) and ignores all other. Now the PC adds a new music folder, which will be immediately downloaded on the smartphone because the new folder not mentioned in ignore settings. I just would like to set some folders in the library to download.

The configuration must enable patterns for ignore as well as for allow.

[gabriel-fallen]

@amra I'm not saying syncthing should adopt .gitignore/.hgignore as the only ignore/allow config. I'm saying it would be nice from syncthing to respect these ignore configs _when and if_ it encounters them while walking a directory tree.

Yeah, whitelist is a very nice thing to have, especially on receiving side as in your example. Totally agree.

[ProactiveServices]

@gabriel-fallen the problem with accepting third-party software ignore lists is several-fold: they are likely to have differing syntax today and tomorrow may be different still; the ignores are for different purposes: I may not want $privatedata to be pushed to a remote git repo, but do want it synced to my other machines; and which ignore pattern system is to take precedence?

[gabriel-fallen]

@ProactiveServices I can't say about all third-party software, but all of Git, Mercurial, Subversion and Syncthing use globs for ignore patterns. And I _really_ doubt these applications will throw backwards compatibility out of the window anytime soon. :wink:

Supporting _all_ third-party software out there is clearly unrealistic and unnecessary. But support for the most widespread ones would be nice.

Regarding ignoring different things I totally agree. In my first comment I said there should be a way to "ignore ignore" or a whitelist. As long as this feature is aimed at advanced users (programmers) files-based solution is acceptable in my view.

[calmh]

We're not going to support other programs ignore patterns. .gitignore looks quite similar to our ignore patterns, but works completely differently (last match wins, cannot unignore children of ignored parents, for example). It is not simply a matter of "it's also globs" -- different semantics, even if the syntax is similar.

[gabriel-fallen]

@calmh OK I see. Thanks for the clear decision.

[finsterwalder]

I'm a new Syncthing user.
(Thanks so much for this really great tool! And in particular for the QNAP version!)
So I just had my first fight with Syncthing patterns...

I think that the current patterns are not bad, but have some problems, that make the ideas posted in this thread a little difficult. I would suggest to modify the ignore pattern handling slightly and then add a UI on top of that, which generates ignore patterns under the hood.
And, as discussed above, an "advanced" mode should be added to allow for more tricky patterns for powerusers.

So what are the problems with the current ignore patterns and how should they change?
(Keep in mind, that my understanding of the current system may be wrong, so please correct me, when I am)

1. sub-sub-folder (un-)ignore is cumbersome, because I have to take care of all the intermediate levels. This makes a simple GUI, where I check a folder three levels deep more complicated.
   With path /a/b/c/d/... when I want to ignore /a but not /a/b/c/..., I just want to write down:
   !/a/b/c
   /a
   And not
   !/a/b/c
   /a/b/
   !/a/b
   /a

2. Patterns ending in with a slash behave unexpectedly. As a first time user I thought, that a trailing slash would clearly state that it only matches a directory. But instead it means only match the directories content. So currently a/ and a/** are actually the same and that's unexpected and misleading. I don't really see a usecase for that.

3. There is no way to clearly differentiate between directories and files. (Follows from point two above).
   Now I can't write a pattern that ignores files, but not directories that match the same pattern or vice versa.
   (e.g. ignore sub-directories starting with a dot but include files starting with a dot, that reside in the same directory)
   I would change the behaviour of a trailing slash to mean directory only. So /a*/ would mean any directory starting with a, but not files starting with a.
   /.*/ -- ignore directories starting with a dot, not just their content
   !/.* -- do not ignore files starting with a dot

4. ** matching slashes is strange. ** should only match any number of sub-directories. But patterns like ab**cd behave strangely and are really complicated to understand and predict. I would restrict the usage of ** to full path matches only. So only allow ** inbetween two slashes, at the start before a slash or at the end after a slash. ab**cd should be forbidden. ab/**/cd would be allowed, as would **/ab and ab/**. ab*/**/*cd would do the same as ab**cd does now, but would be more explicit and easier to understand.

The simple treeview GUI really just needs patterns to include and exclude specific files and directories. That's doable, in particular when point 1 above is fixed.

So a GUI could translate all the checkboxes into specific include and exclude patterns and vice versa. An additional textbox could be provided for advanced patterns.
I think it should actually be possible to combine the two. So when I have whatever patterns in my list, I should be able to filter all the ones, that specify a specific file or directory (no wildcards) and translate them into a treeview and vice versa without losing anything.
So I could even show an advanced mode to edit all patterns and then switch back to treeview. Best of both worlds! :-)

[xxxserxxx]

Is the design to have the ignore file synced or not? It appears so, from the current ticket description. Is so, this seems an odd decision for the purpose. For most use cases, wouldn't you want per device settings for selective synchronization, and isn't that defeated by syncing the ignore file(s)?

[pagdot]

You could probably ignore the ignore file

[imsodin]

The issue tracker is not the right place to ask questions, please use https://forum.syncthing.net for that.

[shish]

wouldn't you want per device settings for selective synchronization

Now that you mention it, I wonder if the reason that this is taking so long to go anywhere is because we're treating "selective sync" (typically a per-device setting looking at folders) and "ignore *.tmp files" (typically a global setting looking at globs) as if they are a single feature? If we instead treated them as two separate features which interact with each other, would one (or both) have a chance of getting implemented?

[xxxserxxx]

@guziks, in #193 mentioned some great use cases. That ticket was closed in favor of this ticket, so I assume someone decided this design covers those use cases. Most of those use cases specifically talk about situations where some machines sync some parts of a share while ignoring other: the set of shared/ignored files differ between machines.

The proposal should be explicit about whether the excludes file will be synced or local only, and it'd be nice to figure out how (if it's shared) all of the use cases as proposed by @guziks will be addressed. Right now, the only live ticket about this feature specifies implementation requirements without describing the problems (situations, use cases) the requirements address (and the ones they don't).

@imsodin: from the ticket description:

Comments and discussion below. A project maintainer will keep the list above in sync as the requirements change or are clarified.

I'm asking for clarification of the ticket, so I think it's appropriate to comment here.

[hlovdal]

I have read through all the comments here and one takeaway is that syncthing ought to have something probably named Folder ignore profiles.

• A shared folder can have zero or more folder ignore profiles associated.
  
  
  
  • Zero means behaviour exactly like today.
  
  
  • Examples: desktop, laptop and tablet.
  
  
• When attaching a new device, the device with the folder content sends a list of all the currently associated folder ignore profiles to the blank device.
• In addition to filling in the device id, the user must select which folder ignore profile to use where "None" is an additional choice.
• The selected folder ignore profile will be used to create an initial .stignore file on the new blank device.
• The .stignore file along with any files it includes (recursively) are the first files that are synchronized. Synchronization of all other files are delayed until this is done.
• After this initial transfer, the .stignore file (and included files) is never automatically updated again, even if the profile is modified.
• Folder ignore profiles are configured in folder settings in a new tab next to "Ignore Patterns".
• Folder ignore profiles are not stored within the shared folder.
• Folder ignore profiles should probably be synchronized between all the devices (and then needs conflict handling).

The above can be implemented today without any dependency to any new ignore behaviour, and it will fully address the forget and race condition issues that @madumlao raised.

The above should represent a complete MVP. Some smaller additional features that could be added later:

• Force propagation of folder ignore profile updates. If you update say the "mobile" profile on your desktop computer, you can chose that this update should be sent to all devices. If a device receives a profile update that matches the profile that initially was used, then it can be applied automatically if configured to do so or have it as a pending question for the user to confirm before doing the update. This means that profiles need a unique id so that renaming from "mobile" to "mobile phone" does not break things.
• Allow for re-selection of profile.
• A folder ignore profile could either be folder specific or global (possible to share between multiple shared folders).

The force propagation ting might not be that important since it effectively already can be achieved with a simple #include .stignore.mobile.