Server: Folder with 1,2 million files not possible to sync

[chwba]

Description already very clear..

Steps to reproduce

1. Install Nextcloud 17.0.1.1 on ubuntu 18.04.3 LTS inside HyperV VM
2. Assign 20 GB memory and 4 CPU cores
3. Manually add 1,2 million files to any users files folder via scp
4. Run occ files:scan and wait 8 hours for it to finish
5. Login to the user using the Windows client, check the folder containing the 1,2 million files, wait.

Expected behaviour

Client dowloads all the files, server does not become instable.

Actual behaviour

Client says "waiting" until, after a very long time, it shows:
An error occured while opening a folder Operation canceled
In the server when I look at htop, I can see that php-fpm: pool Nextcloud is consuming all the 20GB memory of the server and after some time finally kills the processes, all cores at 100% load.

Server configuration

Operating system:
Windows 10

Web server:
Hyper V VM , 4 vCores, 20GB memory

Database:
Postgres 10.10

PHP version:
7.2

Nextcloud version: (see Nextcloud admin page)
17.0.1

Updated from an older Nextcloud/ownCloud or fresh install:
No

Where did you install Nextcloud from:
https://shop.hanssonit.se/product/nextcloud-vm-microsoft-hyper-v-500gb/

Signing status:

No errors have been found.

List of activated apps:

Enabled:

• accessibility: 1.3.0
• activity: 2.10.1
• admin_audit: 1.7.0
• calendar: 1.7.1
• cloud_federation_api: 1.0.0
• comments: 1.7.0
• contacts: 3.1.6
• dav: 1.13.0
• extract: 1.2.2
• federatedfilesharing: 1.7.0
• federation: 1.7.0
• files: 1.12.0
• files_pdfviewer: 1.6.0
• files_rightclick: 0.15.1
• files_sharing: 1.9.0
• files_trashbin: 1.7.0
• files_versions: 1.10.0
• files_videoplayer: 1.6.0
• firstrunwizard: 2.6.0
• gallery: 18.4.0
• logreader: 2.2.0
• lookup_server_connector: 1.5.0
• nextcloud_announcements: 1.6.0
• notifications: 2.5.0
• oauth2: 1.5.0
• password_policy: 1.7.0
• privacy: 1.1.0
• provisioning_api: 1.7.0
• recommendations: 0.5.0
• serverinfo: 1.7.0
• sharebymail: 1.7.0
• support: 1.0.1
• survey_client: 1.5.0
• systemtags: 1.7.0
• text: 1.1.1
• theming: 1.8.0
• twofactor_backupcodes: 1.6.0
• updatenotification: 1.7.0
• viewer: 1.2.0
• workflowengine: 1.7.0
  Disabled:
• encryption
• files_external
• mail
• user_ldap
  

Nextcloud configuration:

{
"system": {
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"localhost",
"SENSITIVE VALUE",
"SENSITIVE VALUE",
"SENSITIVE VALUE"
],
"datadirectory": "REMOVED SENSITIVE VALUE",
"dbtype": "pgsql",
"version": "17.0.1.1",
"overwrite.cli.url": "https:\/\/SENSITIVE VALUE\/",
"dbname": "REMOVED SENSITIVE VALUE",
"dbhost": "REMOVED SENSITIVE VALUE",
"dbport": "",
"dbtableprefix": "oc_",
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"installed": true,
"instanceid": "REMOVED SENSITIVE VALUE",
"log_type": "file",
"logfile": "\/var\/log\/nextcloud\/nextcloud.log",
"loglevel": "0",
"mail_smtpmode": "smtp",
"remember_login_cookie_lifetime": "1800",
"log_rotate_size": "10485760",
"trashbin_retention_obligation": "auto, 180",
"versions_retention_obligation": "auto, 365",
"simpleSignUpLink.shown": "false",
"memcache.local": "\OC\Memcache\APCu",
"filelocking.enabled": true,
"memcache.distributed": "\OC\Memcache\Redis",
"memcache.locking": "\OC\Memcache\Redis",
"redis": {
"host": "REMOVED SENSITIVE VALUE",
"port": 0,
"timeout": 0.5,
"dbindex": 0,
"password": "REMOVED SENSITIVE VALUE"
},
"logtimezone": "Europe\/Berlin",
"htaccess.RewriteBase": "\/",
"maintenance": false
}
}

Are you using external storage, if yes which one: No

Are you using encryption: No

Are you using an external user-backend, if yes which one: No

Client configuration

Client
Windows 10 client 2.6.1sstable-Win64 (build 20191105)

Browser:
Not applicable

Operating system:
Windows 10

Logs

Web server error log

apache error.log

Nextcloud log (data/nextcloud.log)

Log

Browser log

Not applicable, synced with Windows client

Extra

Tries to increase memory limit for nextcloud

Tried with:
/etc/php/7.2/apache/ : memory_limit = -1
/etc/php/7.2/fpm/ : memory_limit = -1
/etc/php/7.2/cli/ : memory_limit = -1

And with:
/etc/php/7.2/apache/ : memory_limit = 512M
/etc/php/7.2/fpm/ : memory_limit = 256M
/etc/php/7.2/cli/ : memory_limit = -1

Same result

[kesselb]

Allowed memory size of 536870912 bytes exhausted (tried to allocate 33554440 bytes)

Probably php needs more memory

Redis server \/var\/run\/redis\/redis-server.sock:0 went away

Please ensure that redis is available

Note: This is the issue tracker of Nextcloud, please do NOT use this to get answers to your questions or get help for fixing your installation. This is a place to report bugs to developers, after your server has been debugged. You can find help debugging your system on our home user forums: https://help.nextcloud.com or, if you use Nextcloud in a large organization, ask our engineers on https://portal.nextcloud.com. See also https://nextcloud.com/support for support options.

[chwba]

Probably php needs more memory

Edited initial post:
Tries to increase memory limit for nextcloud

Tried with:
/etc/php/7.2/apache/ : memory_limit = -1
/etc/php/7.2/fpm/ : memory_limit = -1
/etc/php/7.2/cli/ : memory_limit = -1

And with:
/etc/php/7.2/apache/ : memory_limit = 512M
/etc/php/7.2/fpm/ : memory_limit = 256M
/etc/php/7.2/cli/ : memory_limit = -1

Same result and as already said the server will consume ALL the memory until 20GB / 20GB are used and then ultimately fail.

[go2sh]

Redis might have died due to the memory consumption with oom. I would guess the sync client does not paginate and the server is building (not streaming) a response for the 1,2m files, which will kill it.

[chwba]

Well, that does not sound very healthy..

[jospoortvliet]

whoah, syncing 1.2 mln files. That's quite a thing. @go2sh might very well be right, that the current design wasn't really put together with that in mind. Which probably means it needs quite some changes on server and client, though there might be some ways to decrease the server load if you profile the memory usage of this. Then again, you can then simply re-create the problem by going go 2 or 3 million files :roller_coaster:

Which makes me wonder - why you want to do that? I can not really think of a scenario... Seems a bit of the 'poke-finger-in-eye' thing :roll_eyes: where the doctor tells you: "don't" :laughing:

[kesselb]

If you are interested in digging: https://github.com/nextcloud/server/issues/8962 There are some recommendations for indexes to improve the query performance. They're only work for pqsql unfortunately.

[ghost]

I'd run a more recent PHP and use redis 5 with Unix sockets instead of TCP.

Still it doesn't sound like a good thing with so many files. How is the file structure? A flat dir or subfolders?

What is eating the memory? Sql, redis, apache, fpm?

Do you use redis for local cache or only locking?

[dimm0]

1M files is not a terribly large collection for some machine learning scientists (it can be even quite modest in data volume, since files are usually small), and it's too bad nextcloud is not thinking about working with those. It could've been a nice tool for scientists to sync their collections for processing.

[magikmw]

Not looking to hijack, but I think I have a similar problem on a much smaller scale.

Directory with ~6800 files, total 7GB.
For me Nextcloud client sits on "Checking for changes in remote [dirname]", and then fails with "Operation canceled". Client logs show:
[OCC::LsColJob::finished LSCOL of QUrl("https://_snip_/remote.php/dav/files/_snip_") FINISHED WITH STATUS "OperationCanceledError Connection timed out"

Doesn't look like resources problem though, with RAM sitting steady. The folder also doesn't open in web GUI, but upload from a mobile app apparently does work for it. No other folder manifests this problem, but this error kills the entire sync so other folders aren't being synced.
I can reach the folder over the filesystem and access the files with no issue.

[fabionaspolini]

I'm with the same problem, folder with a lot of small files (2k files) don't syncronize in the desktop app and not show in web page.