Client: Broken sync time estimation - enterprise setup

[mrow4a]

Scenario

Imagine you are just doing your initial sync on one of the sync clients. Target server is enterprise scale setup, meaning so called "1 Byte PUT duration" takes relatively long compared to home scale server.

Imagine, that your initial sync folder contains 55000 files and 50GB of data.

Observed behaviour

Obviously, sync in this case will consist of small files and big files. When sync client syncs big files, it shows estimation for big files. However, when the client syncs very small files, the estimation shows the estimate for whole sync size compared to very small transfer rates. E.g. for 500B-5kB files, 5kB/s observed transfers for 55000 files and 55GB will show something around 2-3 months. This is obviously not true.. of course unless your sync client will sync 1GB file with speed of 5kB/s..

Expected behavious

Client does not show the dramatic and confusing staticstics. It could be very strange for non technical person to see estimate for 3 months, while the sync will take some hours.

Sync client should show estimate for files under "MINIMUM CHUNKING SIZE" which is to be defined for 1MB, and files over this value. Call it "small files and big files estimation".

Client should show how many big files are to be synced and how many small files to be synced currently. Correspondingly, it should show e.g. "Estimated 7h for 35000 small files - total 10GB" or "Estimation not available" instead of "3 months needed" or "No big files to sync". Or at least something less confusing.

I would love to drive a discussion about that and try to implement it in case it is accepted.
@hodyroff @ogoffart @jturcotte @guruz @butonic @felixboehm

[mrow4a]

Can be solved with https://github.com/owncloud/client/issues/5391

[davidjericho]

This is a must fix, and it doesn't appear too difficult to resolve.

We routinely have users complain the estimated time is massively broken, As an example, just this morning, I updated a collection of small files. My desktop is 10Gbps connected, I have less than 1ms to the server infrastructure, which is connected at over 360 Gbps, and it was still forecasting 46 minutes to update 157 files totaling 34MB. The estimated speed was 36kbyte/s when I checked.

Of course, it didn't actually take 46 minutes (rather, less than one minute), but I saw the time estimated jump between less than 10 minutes, to at one point, 37 hours.

This does confuse users, and they do complain about it.

[felixboehm]

I thought such would be solved with bundling. @mrow4a ??

[mrow4a]

@felixboehm Indeed.

[mrow4a]

Guys, I think this is not a high severity issue but is very painful thinking about reputation.

Imagine that someone just test cases (willing to but enterprise license) a newly created instance and puts all his private files there to test, boss decides to do the same.

Imagine the face of management non-tech guys when they see estimation ranging from some hours (which is possible) to few MONTHS (reputational loss).

We cannot wait for any bundling or other improvement to solve that, this has to be done NOW. @hodyroff @felixboehm

[ckamm]

This is #4428.

You currently get very broken estimates when small files are synced first: then there's no good bandwidth estimate and if the sync contains large files, the remaining time will be grossly overestimated (because the starting assumption is just 100kb/s).

Similarly, when large files are synced first, the client's guess for how many small files the server can process per second may be way off (starts with assuming 2 files/s). There may also be other scenarios in which it breaks.

Since we seem to usually overestimate right now, we could adjust the starting estimates. But a full fix is needed. See the other ticket for a proposed model.