Core: PUT requests very SLOW

[loripino21]

Steps to reproduce
Just upload files with Windows Client 1.6.3 Or even 1.7.0 alpha

Expected behaviour
Upload many files (at least 100, if smaller than 100bytes?) within the same minute
Actual behaviour
Only few files uploaded in a minute, Apache logs report that a PUT request takes 13-15 SECONDS to be served

Server configuration
Operating system: XUbuntu 14.04 in a HyperV machine, Windows Server 2012 R2 as Host, fixed vhdx HD, 3000 MB of ram expandable up to 8GB, Vm with 4 cores, Vm never uses more than 10% cpu.

Web server: Apache 2.4.7
Database: MySQL 5.5
PHP version: 5.5.9
ownCloud version: 7.0.2.1
List of activated apps: standard
The content of config/config.php:
'trusted_domains' =>
array (
0 => 'cloud.mydomain.it',
1 => '192.168.0.20',
),
'overwritehost' => 'cloud.mydomain.it',
'datadirectory' => '/mnt/owncloud',
'dbtype' => 'mysql',
'version' => '7.0.2.1',
'dbname' => 'owncloud',
'dbhost' => '127.0.0.1',
'dbtableprefix' => 'oc_',

Are you using external storage, if yes which one: \mnt\owncloud is a cifs mount that is a shared directory of windows server on the virtual internal network. mysql saves db in vhd.

Are you using encryption: no
Client configuration
Browser: irrelevant, same effect on all tested browsers and WebDAV clients.
Operating system: irrelevant, same effect on all tested browsers and WebDAV clients.

Please let me know if you need any further information.

Logs
!!! No reuse of etag for 'files/Altro/Software/Sviluppo/adt-bundle-windows/cygwin/usr/share/zoneinfo/posix/Indian' !!! cache: Array ( [fileid] => 181563 [storage] => home::lorenzo [path] => files/Altro/Software/Sviluppo/adt-bundle-windows/cygwin/usr/share/zoneinfo/posix/Indian [parent] => 180844 [name] => Indian [mimetype] => httpd/unix-directory [mimepart] => httpd [size] => 1011 [mtime] => 1413323929 [storage_mtime] => 1413323929 [encrypted] => [unencrypted_size] => 0 [etag] => 543d9c9cb7ba9 [permissions] => 31 ) data: Array ( [mimetype] => httpd/unix-directory [mtime] => 1413323933 [size] => -1 [etag] => 543d9c9d37ca4 [storage_mtime] => 1413323933 [permissions] => 31 [parent] => 180844 )

From Apache log:
192.168.0.10 - lorenzo [15/Oct/2014:00:08:19 +0200] "PUT /remote.php/webdav/Altro/Software/Sviluppo/adt-bundle-windows/cygwin/usr/share/zoneinfo/right/Africa/Dar_es_Salaam HTTP/1.1" 201 869 "-" "Mozilla/5.0 (Windows) mirall/1.6.3" 14416017
(last number is duration in microseconds - 10^-6)

some from mysql slow queries (>1 sec):
Time: 141014 18:10:32
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.090075 Lock_time: 0.000054 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413303032;
UPDATE oc_preferences SET configvalue = '1413303031' WHERE userid = 'lorenzo' AND appid = 'login' AND configkey = 'lastLogin';
Time: 141014 18:17:09
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.119346 Lock_time: 0.000089 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413303429;
UPDATE oc_preferences SET configvalue = '1413303428' WHERE userid = 'lorenzo' AND appid = 'login' AND configkey = 'lastLogin';
[....]
Time: 141014 18:18:47
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.113130 Lock_time: 0.000165 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413303527;
UPDATE oc_filecache SET etag='543d4ce67feb3' WHERE fileid = '168663';
Time: 141014 18:23:20
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.676195 Lock_time: 0.000099 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413303800;
UPDATE oc_filecache SET mtime = '1413303794', etag='543d4df6734c9' WHERE fileid = '160433';
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.480039 Lock_time: 0.000073 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413303800;
UPDATE oc_filecache SET mtime = '1413303794', etag='543d4df722423' WHERE fileid = '981';
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.472505 Lock_time: 0.000032 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413303800;
UPDATE oc_preferences SET configvalue = '1413303799' WHERE userid = 'luciano' AND appid = 'login' AND configkey = 'lastLogin';
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.332344 Lock_time: 0.000065 Rows_sent: 0 Rows_examined: 0
SET timestamp=1413303800;
INSERT INTO oc_jobs(class, argument, last_run) VALUES('OC\BackgroundJob\Legacy\QueuedJob', '{\"app\":\"search_lucene\",\"klass\":\"OCA\Search_Lucene\Hooks\",\"method\":\"doIndexFile\",\"parameters\":\"{\"path\":\"\\/Altro\\/Software\\/Sviluppo\\/adt-bundle-windows\\/cygwin\\/usr\\/share\\/terminfo\\/6f\\/otek4114\",\"user\":\"lorenzo\"}\"}', 0);
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.386712 Lock_time: 0.000029 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413303800;
UPDATE oc_preferences SET configvalue = '1413303799' WHERE userid = 'lorenzo' AND appid = 'login' AND configkey = 'lastLogin';

[karlitschek]

@icewind1991

[icewind1991]

How long do request take made in the browser?

[loripino21]

Some values reported by mysql:

Created tmp disk tables 61
Handler read rnd 47
Handler read rnd next 37.8 k !!!!
Innodb buffer pool pages dirty 116
Innodb buffer pool reads 2.4 k

[loripino21]

@icewind1991 also ulpoads from browser take 15 seconds after the file has been fully uploaded. instead, get request are not bad. Do I need to create a test account for you to check?

[icewind1991]

If I read the query log correctly each (most?) UPDATE queries take over a seccond.

Can you try doing one of those queries manually, i.e.

UPDATE oc_preferences SET configvalue = '1413303428' WHERE userid = 'lorenzo' AND appid = 'login' AND configkey = 'lastLogin';

which is harmless and see if that also takes a long time.

[loripino21]

I tried to tweak many sql settings, like increasing innodb buffer pool size to 1Gb, so now the slow queries are only in the oc_filecache table.
BTW, execution of the query you posted now is 0.1726 sec.

Some from slow query since last restart:
Time: 141015 16:52:34
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.109603 Lock_time: 0.000086 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413384754;
UPDATE oc_filecache SET mtime = '1379248569', etag='543e8a313f991' WHERE fileid = '183170';
Time: 141015 16:54:25
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.048329 Lock_time: 0.000056 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413384865;
UPDATE oc_filecache SET size='964' WHERE fileid = '183186';
Time: 141015 16:55:31
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.080360 Lock_time: 0.000088 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413384931;
UPDATE oc_filecache SET size='3311' WHERE fileid = '183202';
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.003738 Lock_time: 0.000136 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413384931;
UPDATE oc_filecache SET mtime = '1413384927', etag='543e8ae2bd0c5' WHERE fileid = '183199';
Time: 141015 16:56:11
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.091758 Lock_time: 0.000096 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413384971;
UPDATE oc_filecache SET mtime = '1413384964', etag='543e8b0a0a6b0' WHERE fileid = '182796';
Time: 141015 16:57:13
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.140562 Lock_time: 0.000097 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385033;
UPDATE oc_filecache SET mtime = '1413385025', etag='543e8b481a182' WHERE fileid = '15';
Time: 141015 17:01:56
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.305876 Lock_time: 0.000082 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385316;
UPDATE oc_filecache SET size='2228146' WHERE fileid = '182806';
Time: 141015 17:05:59
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.154735 Lock_time: 0.000123 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385559;
UPDATE oc_filecache SET mtime = '1379937027', etag='543e8d566112f' WHERE fileid = '183281';
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.072256 Lock_time: 0.000090 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385559;
UPDATE oc_filecache SET mtime = '1379249229', etag='543e8d5675912' WHERE fileid = '183287';
Time: 141015 17:06:51
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.733775 Lock_time: 0.000088 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385611;
UPDATE oc_filecache SET mtime = '1379248569', etag='543e8d8a18d89' WHERE fileid = '1070';
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.415515 Lock_time: 0.000052 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385611;
UPDATE oc_filecache SET size='1986' WHERE fileid = '183356';
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.032300 Lock_time: 0.000107 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385611;
UPDATE oc_filecache SET mtime = '1413385607', etag='543e8d8ae019f' WHERE fileid = '183287';
Time: 141015 17:08:35
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.065102 Lock_time: 0.000061 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385715;
UPDATE oc_filecache SET mtime = '1379415717', etag='543e8df24eae1' WHERE fileid = '183374';
Time: 141015 17:10:19
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.032109 Lock_time: 0.000064 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385819;
UPDATE oc_filecache SET size='-1' WHERE fileid = '981';
Time: 141015 17:10:21
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.042956 Lock_time: 0.000052 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385821;
UPDATE oc_filecache SET size='26572558101' WHERE fileid = '1070';
Time: 141015 17:10:32
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.342643 Lock_time: 0.000050 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385832;
UPDATE oc_filecache SET mtime = '1413385826', etag='543e8e6772f2e' WHERE fileid = '182759';
Time: 141015 17:11:25
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.036147 Lock_time: 0.000074 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385885;
UPDATE oc_filecache SET mtime = '1379248571', etag='543e8e9c4b004' WHERE fileid = '183408';
Time: 141015 17:11:53
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.123684 Lock_time: 0.000063 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413385913;
UPDATE oc_filecache SET size='3502' WHERE fileid = '183407';
Time: 141015 17:13:22
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.001694 Lock_time: 0.000096 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386002;
UPDATE oc_filecache SET mtime = '1379322872', etag='543e8f115bb92' WHERE fileid = '183438';
Time: 141015 17:15:21
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.370584 Lock_time: 0.000090 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386121;
UPDATE oc_filecache SET mtime = '1413386113', etag='543e8f8838c10' WHERE fileid = '182759';
Time: 141015 17:16:55
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.089566 Lock_time: 0.000082 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386215;
UPDATE oc_filecache SET mtime = '1413386210', etag='543e8fe67f280' WHERE fileid = '183438';
Time: 141015 17:17:08
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.144115 Lock_time: 0.000085 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386228;
UPDATE oc_filecache SET etag='543e8ff3746fc' WHERE fileid = '183491';
Time: 141015 17:19:08
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.043011 Lock_time: 0.000105 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386348;
UPDATE oc_filecache SET mtime = '1413386344', etag='543e906bb21ae' WHERE fileid = '182759';
Time: 141015 17:19:57
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.055886 Lock_time: 0.000074 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386397;
UPDATE oc_filecache SET size='30874313502' WHERE fileid = '981';
Time: 141015 17:22:31
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.029045 Lock_time: 0.000060 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386551;
UPDATE oc_filecache SET size='8272309' WHERE fileid = '182759';
Time: 141015 17:22:55
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.095709 Lock_time: 0.000161 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386575;
UPDATE oc_filecache SET mtime = '1413386573', size = '-1', etag = '543e914e142b8', storage_mtime='1413386573' WHERE fileid = '183578';
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.073830 Lock_time: 0.001219 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386575;
UPDATE oc_filecache SET mtime = '1413386574', size = '-1', etag = '543e914e1cf89', storage_mtime='1413386574' WHERE fileid = '183578';
Time: 141015 17:25:55
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.451834 Lock_time: 0.000046 Rows_sent: 0 Rows_examined: 0
SET timestamp=1413386755;
INSERT INTO oc_jobs(class, argument, last_run) VALUES('OC\BackgroundJob\Legacy\QueuedJob', '{\"app\":\"search_lucene\",\"klass\":\"OCA\Search_Lucene\Hooks\",\"method\":\"doIndexFile\",\"parameters\":\"{\"path\":\"\\/Altro\\/Software\\/Sviluppo\\/adt-bundle-windows\\/eclipse\\/configuration\\/org.eclipse.osgi\\/bundles\\/537\\/1\\/.cp\\/jars\\/guice-plexus-metadata-2.3.0.jar\",\"user\":\"lorenzo\"}\"}', 0);
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.448099 Lock_time: 0.000093 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386755;
UPDATE oc_filecache SET mtime = '1413386748', etag='543e920236910' WHERE fileid = '161097';
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.189885 Lock_time: 0.000058 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386755;
UPDATE oc_filecache SET size='2464680' WHERE fileid = '183597';
Time: 141015 17:27:56
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 2.482361 Lock_time: 0.000087 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386876;
UPDATE oc_filecache SET etag='543e9279abe96' WHERE fileid = '183645';
Time: 141015 17:28:45
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.130183 Lock_time: 0.000054 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386925;
UPDATE oc_filecache SET size='30882697776' WHERE fileid = '981';
Time: 141015 17:29:50
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.378321 Lock_time: 0.000042 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413386990;
UPDATE oc_filecache SET mtime = '1409590567', etag='543e92ecc1854' WHERE fileid = '183670';
Time: 141015 17:30:58
User@Host: oc_admin[oc_admin] @ localhost [127.0.0.1]
Query_time: 1.788361 Lock_time: 0.000054 Rows_sent: 0 Rows_examined: 1
SET timestamp=1413387058;
UPDATE oc_filecache SET mtime = '1413387049', etag='543e933108106' WHERE fileid = '182796';

[loripino21]

I think it's important to say that owncloud_log now is a 180MB text file repeating errors like:

!!! No reuse of etag for 'files/Altro/Software/Sviluppo/adt-bundle-windows/eclipse/configuration/org.eclipse.osgi/bundles/581/1/.cp/libs/jfreechart-1.0.9.jar' !!! cache: Array ( [fileid] => 183838 [storage] => home::lorenzo [path] => files/Altro/Software/Sviluppo/adt-bundle-windows/eclipse/configuration/org.eclipse.osgi/bundles/581/1/.cp/libs/jfreechart-1.0.9.jar [parent] => 183833 [name] => jfreechart-1.0.9.jar [mimetype] => application/octet-stream [mimepart] => application [size] => 1291498 [mtime] => 1413387785 [storage_mtime] => 1413387785 [encrypted] => [unencrypted_size] => 0 [etag] => 543e961310c8c [permissions] => 27 ) data: Array ( [mimetype] => application/octet-stream [mtime] => 1409575892 [size] => 1291498 [etag] => 543e9613707c0 [storage_mtime] => 1409575892 [permissions] => 27 [parent] => 183833 )

[loripino21]

I was getting angry, so I installed mysql on Windows directly, moved the owncloud database and pointed owncloud to the Windows mysql instance. now performance has improved (the test query you posted runs in 0.10sec and a put request takes 4-5 secs) but it's still unacceptable.
Any hints??

[loripino21]

Some other info.
I tought that the problem was the virtuale machine, so I created a completely new vm , installed linux xubuntu, set up apache2, php only and copied the old owncloud folder (data was already on a shared folder in windows). Pointed owncloud to mysql running on physical server. Only a slight performance improvement, but still 6 seconds for some PUTs.
Don't know what to try now!

[loripino21]

If someone is still wondering where the problem could be, here is a hint. I ran "ps auxw | grep apache2 | awk '{print"-p " $2}' | xargs strace -o /strace/strace.log -ff -s4096 -r" on linux VM, and then "for i in ls /strace/*; do echo $i; cat $i | cut -c7-87 | sort -rn | head; done". The result is the duration of procedures ran by apache processes, ordered by duration descending:
https://www.dropbox.com/s/97r34tonu3133ek/log.txt?dl=0

[msx80]

any news on this issue? I'm seeing similar problems. The program uploads a few small files and then stops for 4-5 seconds, then some other files and other 5 seconds stop. And so on. When the files are very slow, the time spent idle is many times bigger than that spent transmitting.

[loripino21]

Unfortunately no. I am still in this situation waiting. @msx80 can you please check under admin settings if your cron jobs are running normally? I suspect it to be the cause

[msx80]

yes it shows a green circle with last cron execution timestamp. Why do you suspect that's the cause? AFAIK the cron task is only executed every 15 minutes to catch up with the disk changes, or am i wrong?

[loripino21]

Maybe I am wrong, but I think that Cron Jobs are needed to index file, and my OC is probably performing bad as query on the database are WITHOUT INDICES (phpmyadmin continuously warns me about this situation)

[PVince81]

Are you guys still seeing this with 7.0.5 or 8.0.2 with sync client 1.8 ?

[loripino21]

I will update to 8.0.2 soon and let you know!

[avrhack]

I'm on 8.0.3 and 1.8.1 and performance is still unusably slow. This has been going on since v5, when are we actually going to get a fix as people are ditching own cloud for other options as a result of the incredible amount of time it is taking to address.......

[PVince81]

Related https://github.com/owncloud/core/issues/12001

[jospoortvliet]

This has sped up significantly with newer ownCloud releases.

On my server (first gen quad core Core i5 clocked at 1.2ghz, 8Gb ram, APCu, Redis, PHP 7, everything tuned quite well, MySQL on a SSD) it runs 5mb/s, which is the max line speed, for photo's, think filesize between 0.5 an 2 mb. So it handles several pics per second.
See it going:

That is with file locking enabled, not sure if that has a big impact.

For very many very small files, though, things are still not ideal:

See the server load maxing out:

It takes about 20 minutes to sync 1MB worth 10K files 100 bytes each. That means at least a win for @dragotin and his team: the time estimation of the client started low but went up quickly and turned out to be quite accurate ;-)

[guruz]

Thanks @jospoortvliet

(Note to client team: We could measure the latency and request completion time and then crank up the parallelism a bit .. )

[jospoortvliet]

@guruz not sure if more parallelism would help, the server, as you see, is quite loaded. There's still room for more, you can probably go to 8-10, which probably doubles the speed, but servers with many users might not be happy with the resulting load on the server. A few users doing this could bring a very strong server to its knees, I bet...

[jospoortvliet]

On recommendation of @LukasReschke I also enabled reddis memcache locking:
'memcache.locking' => '\\OC\\Memcache\\Redis',

This seems to lower the load on mysql, it now hovers around 80% rather than >90% and the time it takes to download the files is cut to about 13 minutes (30% faster). See it get started here:

warning: entirely unscientific tests! :kissing_heart:

[dragotin]

@jospoortvliet awesome test. Thanks.

[jospoortvliet]

For those who want to run this 'test' themselves, it is easy:
Generate a folder with 10.000 files of 100 random bytes and 1 'big' 1MB file:

mkdir SmallFiles && cd SmallFiles
dd if=/dev/urandom of=masterfile bs=1 count=1000000
split -b 100 -a 10 masterfile

You can of course remove the 'masterfile' to get rid of the one 1mb file ;-)

Now move (not copy, and ideally, do it on the same filesystem) this folder somewhere you sync with ownCloud and check the client having fun syncing it all.

[icewind1991]

21519 will give some more improvements

[jospoortvliet]

Right now, based on what I've seen, I would say that a reasonably well configured server/client setup can handle anything above 1 mb files at >10 mb/sec quite easily. There is room for some more optimization, of course.

However I wanted to note that if 10.000 files of 100 bytes is a scenario we want to support, realistically, there is probably no way around batching this on the client side by putting a large number of them in a single file and letting the server take it apart again (based on conversation with @icewind1991 )

As that's a lot of work I guess it is a question for @MTRichards if that is needed.

[guruz]

@jospoortvliet the 2.2 client will try to do more parallelism for _small_ files. You can try with a nightly build from this morning.. http://download.owncloud.com/desktop/daily/
(3rd of March)

[louisl]

I suffered with slow syncing of many small files too, after trying various optimisation and caching options I looked into the MySql queries, I noticed some really slow queries like:

SELECT `path`, `fileid` 
FROM 
`oc_filecache` 
WHERE `storage` = '3' 
AND `path` 
LIKE 'files\\_encryption/keys/files/Work/Websites/www.example.com/.git/objects/d0/e0aa5af4bd0227c83f7dd73db033709c8438af/%'

So I tried adding an index for 'path' on 'oc_filecache', this made a big improvement.

ALTER TABLE `oc_filecache` 
ADD INDEX `ix_path` (`path` ASC);

ownCloud: 8.2.2
MySql: 5.5.48
ownCloud client: Version 2.1.1 (build 3107) (OS X)

[Xenopathic]

@louisl We can't add an index for the path column, it's too long and causes some DBs to fall over. Instead we should be using path_hash, it's strange that you are seeing queries without path_hash though (cc @icewind1991 )

[louisl]

@Xenopathic FYI It added a 255 length index. I think that means it just indexes the first 255 chars.

The query I got from MySQL workbench by choosing a query with high 'Time' in 'Client connections' right clicking it and choosing 'Show in editor'.

I ran it it the editor and it took ages, so I added the index.

My ownCloud client changed from estimating days to minutes and was done in 5 mins.

Hopefully there's something useful in this for you.

[rullzer]

You are probably seeing the like in https://github.com/owncloud/core/blob/master/lib/private/files/cache/cache.php#L513

This is done since if we move a folder around we need to update all its children. Now the path is the only thing we have to find this. Else we migh end up with a huge number of queries if you try to move a folder with a lot of sub folders and files.

[louisl]

@rullzer That context does make sense as the initiator of the sync was me creating a release branch making changes, committing and switching back to the develop branch using gitflow.

[rullzer]

@icewind1991 maybe we should rewrite that part. To just get all the children... move those and then query all the childrenof subnodes etc. That way we can at least use indexes.

OK this results in a lot of queries if you have a huge subdir below it it will most likely be on a large installation. So LIKE is going to be even more horrible.

Then as a second step we could do as @dragotin suggested (in IRC). Have a separate table that lists all the parents of a file. That way we could do a very fast search.

[dragotin]

Yes, a table having all parents of either all files or only the directories would help tremendously.

[DeepDiver1975]

Different file cache table implementations have already been discussed in different places https://github.com/owncloud/core/issues/4209

Given the recent changes in 9.0 where the cache is now fully abstracted and alternative implementations are possible we could think about using different database schema.

Please note that such an effort has to be properly planned with @cmonteroluque and coordinated with @karlitschek and myself.

Thanks

[DeepDiver1975]

It added a 255 length index. I think that means it just indexes the first 255 chars.

this is actually an interesting approach - even if the significant change is no in the first 255 chars the set can still be dramatically reduced (I assume).

We should invest into using prefixes indices - http://dev.mysql.com/doc/refman/5.7/en/create-index.html

[karlitschek]

yes. this would be a huge task. the migration and testing efford alone is significant. something to consider for maybe 9.2

[icewind1991]

I suffered with slow syncing of many small files too, after trying various optimisation and caching options I looked into the MySql queries, I noticed some really slow queries like:
[snip]

It seems that it's not the PUT being slow but a MOVE

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[boettner-it]

Syncing large amounts of small files is still an issue here with 10.0.5, indexing oc_filecache didn´t bring real improvements. So I´d suggest to keep this issue open.

I´m ready to contribute with tests, my setup is as described above.

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[boettner-it]

Please reopen, see my previous comment above.

Syncing large amounts of small files is still an issue here with 10.0.5, indexing oc_filecache didn´t bring real improvements. So I´d suggest to keep this issue open.
I´m ready to contribute with tests, my setup is as described above.

[PVince81]

@mrow4a what was the conclusion of your research regarding bundling multiple files within a single PUT call ?

[guruz]

AFAIK it improved performance but @DeepDiver1975 and @ogoffart did not like the code complexity.
(And even with HTTP2 you're invoking one php for each file)

[DeepDiver1975]

The trade-off between code complexity and real life usecases is very limited.

What are real life scenarios for many small files being synced at once?
One un-tars the linux kernel source into the ownlcoud folder - not really real life.

Syncing thousands of pictures after exporting them from mobile of camera: more real life scenario. But give the size of today's picture: one picture has closely the size of one chunk.
You don't gain much in this scenario.

[boettner-it]

What are real life scenarios for many small files being synced at once?
One un-tars the linux kernel source into the ownlcoud folder - not really real life.

A real life scenario from my side is syncing software development files I hold locally between machines (e.g fixed desktop pc and notebook). Software projects almost anytime consist of hundreds of thousands of very small files as you always hold used libraries used in your projects in your development environment

[DeepDiver1975]

A real life scenario from my side is syncing software development files I hold locally between machines (e.g fixed desktop pc and notebook). Software projects almost anytime consist of hundreds of thousands of very small files as you always hold used libraries used in your projects in your development environment

TBH: this is what git and other software version control systems are made for.

Not a valid use case for owncloud in my pov

[boettner-it]

Hm, not necessarily. I ususally don´t include everything in my repos especially external libraries I only need for my IDE and the environment there.

[DeepDiver1975]

Hm, not necessarily. I ususally don´t include everything in my repos especially external libraries I only need for my IDE and the environment there.

no matter what software development workflow you choose - owncloud is not a tool for software development - I'm sorry to sound harsh but this is not a scenario we will optimize owncloud to be used for.

other sync'n'share solutions are faster compared to us with respect to many small files.
owncloud is focused on openness and the high ability to integrate with various storage backends. this variety brings shortcomings to the code base.

[boettner-it]

I don´t want to bother anyone and don´t see oc as a dev tool. Every solution has it´s limitations and if we reach one here it maybe is an uncovered use case and surely kind of natural.

A huge amount of small files may also occur in other scenarios I won´t limit that to software development. But if that´ is out of scope atm and won´t be covered anytime sooner or later than this issu might be closed. Thanks for taking care and all the other efforts put into owncloud!