Desktop: Sync fails: "too many open files"

[gcala]

Hi, sync seems broken on my laptop. It fails with a series of "Too many open files" messages in the client. It stops for a moment and then restarts the whole process from beginning.

I don't know if this error is related to desktop client or machine where it runs.

Any idea? Thanks

Client configuration

Client version: 2.5.1 git (rev. b37cbe)

Operating system: Arch Linux

OS language: Italian

Qt version used by client package (Linux only, see also Settings dialog): 5.12.0

Client package (From Nextcloud or distro) (Linux only): community/nextcloud-client 2.5.1-2

Server configuration

Operating system: Arch Linux

Web server: apache 2.4.37

Database: mariadb 10.1.37

PHP version: 7.3.0

Nextcloud version: 15.0.2

Logs

1. Client logfile: https://cuteworks.it/nextcloud/s/KzWbigWWwNjRk2n

2. Web server error log: no errors

3. Server logfile: no errors

[bleidingGOE]

I have the same issue on my Linux system and it seems to be only triggered when I process larger numbers of very small files, e.g., small pictures or git-repositories. Client 2.3.3 works, so it has to be a problem that was introduced with version 2.5.x. I also tried the dev-branch for the nextcloud-client -> same issue.

Client configuration

Client version: 2.5.1
Operating system: Xubuntu 18.04 LTS
OS language: English

Server configuration
Managed Nextcloud (Owncube.com) - Version: 15.0.0

Logs
Web server error log: to my knowledge -> no errors
Server logfile: to my knowledge -> no errors
Client logfile:
20190109_1148_owncloud.log.0.gz
20190109_1200_owncloud.log.0.gz
20190109_1200_owncloud.log.1.gz
20190109_1212_owncloud.log.0.gz
20190109_1213_owncloud.log.0.gz

Let me know whether you need more info for debugging purposes!

[GieltjE]

Could it be this crashes on Windows machines? When we try to sync large numbers of new files (2k+) it just crashes after a while.

[bleidingGOE]

Yes, I was also able to reproduce the same results on a virtual machine with Win 7 and the 2.5.1 client for Windows. Unfortunately, I cannot provide logs at the moment. But the Linux logs should already help the dev team once they take care of this issue.

@GieltjE Could you maybe also upload logfiles to make it easier for the dev team to fix our issue?

[GieltjE]

The client log just logs sync complete, a friend of mine will hopefully post the windows logs soon.

We also noticed that it happens on lots of small files, when the files get bigger it doesn't crash every time.

@bleidingGOE, the windows logbook should provide the most information

[dennmuel]

Confirming the "too many files" error on Linux Mint. However the client does not restart but instead either dies without notification or causes the operating system to freeze.
As a workaround one can disable sync for the causing folders, so that at least all others get synced.

[bleidingGOE]

It has been 2 month since @gcala opened the issue - the dev team did not even respond despite us providing several logs and precise descriptions. Is nextcloud still actively developed?

[GieltjE]

According to github there are enough commits in the server code, but the desktop client just seems to be getting updated translations....
We can't even force the clients to update to new versions resulting in major parts of the userbase running outdated crap....

[camilasan]

Hi All,

Note that Github isn't a support portal. We provide support to customers only. Github is our developer platform. If you're not a customer you can ask the community for help on help.nextcloud.com.

We of course try to deal with bugs as quick as we can, but it can take time as we have lots of other things on our plate. You are welcome to contribute to this free and open source product, to submit patches that fix bugs for example. We try to respond quickly to pull request from contributors.

Have a great day!
Camila

[lbdroid]

I ran into this issue today after a distro upgrade. The version I was running prior to the upgrade was https://github.com/nextcloud/desktop/commit/57bc7918d7b0650c116f3512787f7677d4e5ab17 . I am running that version again, and it is not affected by this REGRESSION.

If anybody cares enough about this problem to work on it, I'd suggest bisecting with that commit as a starting point.

@camilasan : your response comes off as hostile and rude. Not a particularly good way to encourage people to help in this project. I mean after all, how dare people want to ASSIST the project by reporting and discussing bugs in lieu of supporting it financially.

[jmjos]

Hello!

I also encountered the problem. It's not a bug within nextcloud, rather a system feature limiting the number of open files. E.g. see https://easyengine.io/tutorials/linux/increase-open-files-limit/ for a fix.

The issue can be closed.

[GieltjE]

It's not a Linux only problem, we only use Windows clients.

[bleidingGOE]

I already tried this on my ubuntu 18.04 LTS - did not work. Moreover, the problem also occurs on Windows.

On 27 June 2019 10:00:14 CEST, JM Joseph [e-mail] wrote:

Hello!

I also encountered the problem. It's not a bug within nextcloud, rather
a system feature limiting the number of open files. E.g. see
https://easyengine.io/tutorials/linux/increase-open-files-limit/
for a fix.

The issue can be closed.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/nextcloud/desktop/issues/1035#issuecomment-506237981

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[wioxjk]

Experiencing the same problem with 2.6.1 on the following distros:

Ubuntu 18.04, 19.10
Ubuntu Budgie 18.04, 19.10
Trisquel 8, 9
Debian 9, 10
Parabola
Windows 8, 8.1, 10
Linux Mint 18, 19
OpenSuSE Tumbleweed

sadly, changing ulimit does not work.

The problem does not occur on 2.3.1

Photo and videofolder is 900 GB

[jeroenpraat]

+1 on Ubuntu 19.10.

I'm trying to sync a 25GB photo folder. 'Too many open files', 100% CPU and application freeze. I tried it with or without hidden files.

Seems like a huge issue and I'm perplexed that this is not taken seriously and still not fixed.

[wioxjk]

Seems like a huge issue and I'm perplexed that this is not taken seriously and still not fixed.

I agree, I've asked in the official Nextcloud channels but I have not even gotten a reply there.

[wioxjk]

Still no attention from the developers?
I am strongly considering stop using Nextcloud both for private use and corporate, this issue is starting to become a plague for both usecases.

[claell]

I added some appropriate labels. Hopefully some developer can take a look at this, soon.

Just to get the impact correctly: This is completely preventing the concerning files to be ever synced and does not slowly disappear after more and more files got synced, correct?

[bleidingGOE]

Thanks for adding the labels!
Regarding your question: No, it does not slowly disappear - I left it running for several days without any progress.

[claell]

Thanks for clarifying @bleidingGOE. That make the issue rather severe in cases where it appears.

[SolarDesalination]

It's a bit embarrassing that Nextcloud is more focused on developing sub-standard apps that there already exist viable open source solutions for instead of making sure that foundational pieces such as folder synchronization is working properly. This bug has been open for over a year. I won't be recommending this software in the future to corporate clients.

[lbdroid]

@mpass3 : in all fairness (and I have no ties to nextcloud from a development perspective, just as a user), desktop synchronization really isn't a primary feature of nextcloud, and in many/most cases, the concept of synchronization is too confusing to actually use safely/reliably. I.e., change a file in both local and remote filesystem... Which version wins? The fact that there is no INTUITIVE answer to this means that the concept is flawed. In fact, this very problem is the reason why tools like git exist, and you would never hand that tool to a corporate paper pusher.

For corporate use, what you really want is to mount the nextcloud data into the local filesystem. You can use WebDAV for that. Let the end user decide on what files to overwrite.

[bleidingGOE]

@lbdroid : I absolutely disagree. Even for large corporations, a client synchronization feature is essential. The issue of "which file should I keep" has been solved by Dropbox, GDrive, Syncthing, and so many other projects.

Moreover, based on your suggestion I tried to use my Nextcloud on Ubuntu 18.04 LTS with different WebDAV approaches -> They are horribly slow. No way to use this productively. Once you start googling the "slow WebDAV" issue they recommend using the client, which leads us back to this issue here.

[claell]

Can we please keep the discussion here on topic? Feel free to continue in the forum, you may post a link to a that here, so the others know where to continue. The topic you are talking about is definately worth disputing. But this is the issue tracker of nextcloud and this is a specific issue, so your comments are rather unrelated.

[bleidingGOE]

Again, I disagree. The relevance of an issue is determined by how many users it affects and whether it can easily be solved by using a different approach. Not clarifying the gravity of the issue is why it has not been addressed in more than a year.

[claell]

Users affected may either use the emojis on the top to vote or if relevant add additional information in the comments. Also workarounds can of course be mentioned.

However, the last comments were discussing the relevance of having a working client. This is a meta discussion and could be applied to just any bug that is open here. So it is not specific for this issue.

Not clarifying the gravity of the issue is why it has not been addressed in more than a year.

I guess lacking development power for the client is the main reason. If you have the skills and would like to contribute that would be very much appreciated.

[nerdling]

Workaround for me (both changes required):

/etc/systemd/user.conf
DefaultLimitNOFILE=65536

/etc/security/limits.conf
*       soft    nofile  65536
*       hard    nofile  65536

And reboot.

Seems like the sync client isn't closing files as it's done with them. Normally fine, but after 1024 items it's this error. Is some code closing the FDs outside of a loop rather than inside?

[GieltjE]

/etc/systemd/user.conf
DefaultLimitNOFILE=65536
/etc/security/limits.conf

• soft nofile 65536
• hard nofile 65536

Awesome for *nix variants but Windows doesn't have these settings.

[bleidingGOE]

Already tried that on Ubuntu 18.04 LTS - did not work for me.

On 23 April 2020 16:40:07 CEST, Michiel Hazelhof [e-mail] wrote:

/etc/systemd/user.conf
DefaultLimitNOFILE=65536
/etc/security/limits.conf

• soft nofile 65536
• hard nofile 65536

Awesome for *nix variants but Windows doesn't have these settings.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/nextcloud/desktop/issues/1035#issuecomment-618433696

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[wioxjk]

Workaround for me (both changes required):

/etc/systemd/user.conf
DefaultLimitNOFILE=65536

/etc/security/limits.conf
*       soft    nofile  65536
*       hard    nofile  65536

And reboot.

Seems like the sync client isn't closing files as it's done with them. Normally fine, but after 1024 items it's this error. Is some code closing the FDs outside of a loop rather than inside?

Does not work as I wrote here: https://github.com/nextcloud/desktop/issues/1035#issuecomment-546673468

And I agree with the previous poster - bleidingGOE - that the sync feature is a essential part of Nextcloud and has always been. Defending the developers in this case does not make any sense at all, it is a bug and should be treated as such.

[warnerbryce]

I got this bugs too,
I get back to v.2.3.3 and no more issue.
@camilasan don’t want to open her eyes and see that since v.2.5 things went wrong with the sync engine on Linux / Windows and OS X.
The v.2.6.4 get better results in many cases i use it on my linux but not on my windows. Impossible to get logs because the software hangs and is irresponsible.

One year waiting bug

[claell]

Impossible to get logs because the software hangs and is irresponsible.

You should be able to write a log directly to a file, that will probably circumvent this problem.

[claell]

This might be the same as #1769. It should have been fixed in #1899. Please read through #1769 whether it is the same issue and there is also a link to a daily AppImage build where that issue seems to be fixed. Maybe you also want to test that out, whether it fixes your problems as well.

[nerdling]

Removing the increased fd limits and then using the AppImage to fetch a fresh Nextcloud mount succeeds. The problem appears to be fixed in the AppImage.

[claell]

Nice, thanks for the feedback! I will leave this open until it lands in a stable release.

[danjpgriffin]

PR #1899 to fix this issue only made it into the 2.6-stable branch. I downloaded v3.0.1 yesterday and unfortunately the issue has reappeared. I'm just checking if reapplying the patch to 3.0.1 works, with then reopen PR against master