Spksrc: Syncthing: set fs.inotify.max_user_watches?

[Jamesits]

Setup

_Package Name:_ Syncthing
_Package Version:_ 0.14.49-15

_NAS Model:_ DS1817+
_NAS Architecture:_ x64
_DSM version:_ DSM 6.2.1-23824

Syncthing reports

failed to setup inotify handler. Please increase inotify limits, see https://docs.syncthing.net/users/faq.html#inotify-limits

Is it possible to set fs.inotify.max_user_watches to a larger value everytime syncthing package starts?

[BenjV]

I would suggest not to use syncting anymore.
Their advice is to set it as follow
fs.inotify.max_user_watches=204800

This would mean you lose 220 Mbyte on kernel memory for a 64bit Nas and 110 Mbyte on a 32bit Nas.
That is an obsceen amount of memory lost for just one application and on an normal Nas would be left with not enough memory to run normal operations.
So this will never be set in the standard package, because it will cripple all x64 Nasses with less memory.

If you want to do that to your own system, login with putty and give the following command.

echo "fs.inotify.max_user_watches=204800" | sudo tee -a /etc/sysctl.conf

It is possible that this must be repeated after a DSM update,
You better use the standard cloudsync functionality for file syncing, that is a much better solution.

[mxdpeep]

cloudsync is not an option for VPS backup syncing
and Synology sync soutions mostly create CaseConflict files and folders

[luke-cbcd]

You better use the standard cloudsync functionality for file syncing, that is a much better solution.

I have tried both and find two major problems with cloudsync vs syncthing:

• For large folders (>200k files), even with a quad-core Atom CPU, Synology themselves do not recommend using cloudsync (https://global.download.synology.com/download/Document/WhitePaper/Synology_Cloud_Station_White_Paper.pdf). My experience is that the database it uses is very slow with folders exceeding this size.

• If both ends of the transfer are behind NAT, cloudsync can be very slow at transferring files (even ignoring the aforementioned database bottleneck) - in the 100s of KB/s. Syncthing, otoh, manages ~100 MB/s, on the same devices, with the same files, on the same network.

Your point about memory usage is a good one, but for some of us, where sync is the main purpose of the NAS, I can live with losing a few 100MB of RAM for the benefits that syncthing confers.

[BenjV]

200.000 files in a single folder is a very bad idea for any operation system.
It will make every operation on such folder very very slow and not just a sync operation.
Don't blame tools for wrong designed systems.

And syncing very large amounts of data is a bad design anyway with whatever tool you are using.

Speed and NAT have nothing to do with eachother, that is surly the wrong conclusion if you have speed problems.
If using Nat is the problem then you have an router( a device that is performing the Nat function) that does not perform or is defect, or you have a network problem.

And least but not last you must remember that Cloudsync is designed as a background service which means that all other operation get more priority.
Syncthing is eating up all resources on a Nas, leaving none for the normal Nas operation which normal users don't like very much.
And don't forget, kernel memory is lost forever and cannot be used anymore even if Syncthing does not use it.
And I predict that that the need for a higher "fs.inotify.max_user_watches" stems from the obseen amount of files in a folder and that even the recommended amount shall not be enough.

[luke-cbcd]

200.000 files in a single folder is a very bad idea for any operation system.

Not 200k in a single folder, 200k in the sync database (total number of files within a shared folder, if counted recursively).

Speed and NAT have nothing to do with eachother

When behind a NAT router, the only way for Cloudsync to operate is via Synology's Quickconnect relay service. This is what causes slow transfers. In my experience:

• Cloudsync, no NAT/LAN: 100+MB/s | Cloudsync, NAT/broadband: 100KB/s
• Syncthing, no NAT/LAN: 100+ MB/s | Syncthing, NAT/broadband: 70MB/s (saturates broadband link)

Syncthing is eating up all resources on a Nas, leaving none for the normal Nas operation

Speed of sync obviously scales with resources. If Cloudsync is using fewer resources, it will take longer to sync - which was my complaint.

IME syncing an 11TB shared folder (~250k folders, ~5m files) on a DS918+ with 4GB of RAM there is a massive performance hit as the out-of-sync hashes are swapped from RAM to disk. With 12GB of RAM on an otherwise identical DS918+ it isn't an issue - syncthing tops out at about 5GB RAM usage with plenty of speed left for serving files. And that's just for the initial sync - as blocks move into sync across the cluster the RAM usage drops, and the performance hit goes away.

[Jamesits]

Everyone has their needs. It's not about which solution works for everyone, but is there any possibility we make this an option for the people who need it?

As for me, 2GiB memory consumption is certainly better than scanning multiple terabytes of media files on an HDD array and calculating hashes out of them every 10 hours. NAT traversal is important for me too.

[BenjV]

This scanning is only done initial, after that just when you add, delete of remove a file the database is updated so this is just the wrong conclusion.
The point is that syncthing uses kernel memory for this and that is a stupid idea.

If NAT is a problem with QuickConnect, then the same problem would occur with syncthing and QuickConnect.
But I don't see why QuickConnect would slow things down?
If your convinced it is QuickConnect, then you should try portforwarding and use DDNS.

But al this should be asked in a Synology forum and not in this SynoCommunity forum.
The question was that you wanted the package to be changed so that fs.inotify.max_user_watches was increased.
I explained that such thing will not happen, because it will cripple the Nas for normal use.
I gave you a way to do it for your own Nas, but it will not be added to the package.

[luke-cbcd]

I agree we're getting off-topic but just for the sake of anyone else who finds this thread in future:

If your convinced it is QuickConnect, then you should try portforwarding and use DDNS.

This is of course the correct course of action, but in the real world it is not always feasible. For example, it is impossible to convince a hospital to open a port so that a NAS can sync data; NAT traversal/relay is the only way in this situation. For those with this sort of use case, syncthing is superior to cloudsync (albeit with the provisos in terms of whole-system performance we've noted in this thread).

[BenjV]

Syncthing also need a portforwarding or the use of QuickConnect, so I don't see the difference.

But I think your testing is somehow wrong, because in the past QuickConnect was using the Synology servers for the datatastream to go through, but now a day they only use that for the initial connection and after that the Synology Servers will step out of the link.

So I don't see why QuicKConnect would slow down things and if so they will slowdown Syncthing equally.

[luke-cbcd]

But I think your testing is somehow wrong
I don't see why QuicKConnect would slow down things

I don't know what to say to you. Same hardware, same data, same networks. Syncthing is an order of magnitude faster over NAT, so it must be doing something different with it's relays to quickconnect.

in the past QuickConnect was using the Synology servers for the datatastream to go through, but now a day they only use that for the initial connection

I tested it three weeks ago.

[BenjV]

And no portforwarding for Syncthing?
Why should it be different then?

But if you are happy with the result it is fine with me.
I am not going to install syncthing again to test it.
I used it in the past and will never use it again, it was slow and totaly unreliable.
Maybe it was then an unmature product but then they should have labeled it a Beta and not a production version.
And this item only strenghten my idea about syncthing, demanding so much kernel memory it a stupid solution.
The problem most likely stem from the use of GO.

[luke-cbcd]

And no portforwarding for Syncthing? Why should it be different then?

No, no port forwarding on either. I can only assume that quickconnect is the bottleneck, rather than the nature of NAT itself (which should - as you say - not limit throughput once a connection is established).

[Jamesits]

Syncthing:

• itself can utilize UPnP and automatic NAT traversal
• might work behind double NAT
• You can set up your own discovery/relay service
• works on more OSes

Synology solution:

• will TRY to connect directly but no solid NAT traversal, no double NAT
• UPnP is not supported very well (router model limitation, weird check that I never pass)
• official relay which is slow in some part of the world
• require static port forwarding on router

[BenjV]

Using Upnp in a router is considered to be a security hazard.

If you want to use two Synology Nasses and no port forwarding it is also for Synthing impossible to create a connection without the help of something like QuickConnect.
No application is capable of doing double NAT without the help of Upn in the routers or double port forwarding on the routers.
QuickConnect does not do data stream relay and is only used for setting up the initial connection.

But this the last thing I will post on this issue, I have no intention to go into a marketing battle.
Use whatever you want and pay the pricze.

[raako71]

cloudsync blows. no simple versioning, cannot change client/host IP, no fine grain information on whats even going on. Logs dont even show up.
EDIT: New Syology Drive Client has much nicer client side versioning!