Core: Detect chunked upload timeouts and retry

[pmaier1]

With IE uploads sometimes get stuck. Very important issue for the next server release.

[butonic]

uploads can get stuck with any browser. The IE case is a little bit different because IE sometimes just stops uploading. In the middle of a chunk. The ui freezes, but we already tested that the upload can be aborted using js. Afterwards the browser works fine again. So, the idea is to retry stalled chunks in general. It makes sense for all browsers, but for IE it even allows recovering from a UI freeze.

[ownclouders]

GitMate.io thinks possibly related issues are https://github.com/owncloud/core/issues/25480 (Detect DB connection timeout and retry connection), https://github.com/owncloud/core/issues/16278 (Problems with Chunked upload), https://github.com/owncloud/core/issues/5326 (web dav chunked upload tasks), https://github.com/owncloud/core/issues/14603 (Disable / detect mod_reqtimeout to avoid upload problems), and https://github.com/owncloud/core/issues/17996 (Prevent uploading the same chunk twice).

[butonic]

hm https://github.com/blueimp/jQuery-File-Upload/wiki/Chunked-file-uploads#resuming-file-uploads & https://github.com/blueimp/jQuery-File-Upload/wiki/Chunked-file-uploads#automatic-resume look promising. I already have the test code that aborts the stalled upload. It should be possible to customize these snippets to automatically resume the upload at the right chunk if we can get the current chunk or the number of uploaded bytes from the server.

@PVince81 IIRC you wanted to get rid of blueimp ...

[PVince81]

blueimp is not fully compatible with our way of doing chunking, so it might be rather tricky to make it resume. still worth a try ?

[felixboehm]

If a upload breaks for any reason, ideally we can just continue that upload.
Yet, on client side the state for "currently uploading fileXYZ" is lost, when the page is reloaded.
Does it make sense to store this state on the server to retry on stalled uploads?
@PVince81 @butonic

[PVince81]

At some point in the future we can consider saving upload state on the server, out of scope for this fix as it involves a lot of changes in the server to make the upload stateful. Additionally we'd need to make sure that the file being resume is exactly the same as the one we just started.

For the scope of this fix the goal is to make it resume without reloading the page as the state information is still there in some form.

[felixboehm]

ok, when the reason for the failing uploads is not needing a reload.

[PVince81]

In Phoenix let's experiment with resumable.js: https://github.com/owncloud/phoenix/issues/67.

If it turns out that it works well we could consider backporting...

[butonic]

• [x] disable trace

[PVince81]

the fix is causing trouble now on stable10 / v10.0.8RC2, reverting https://github.com/owncloud/core/pull/31185 for the release

[PVince81]

needs rescheduling

[PVince81]

Estimate: 1md to get this sorted out.

this time let's also try and add JS tests for chunking, even though it's difficult gymnastics because of the jquery.fileupload object and its various event handlers

[PVince81]

bringing back the old PR where work will continue: https://github.com/owncloud/core/pull/32170

[PVince81]

from the changelog, this is what did not work while testing:

when simulating chunk failure, the upload finishes with error about sum of chunks sizes not matching

[PVince81]

ok, seems the problem is solved: the PR itself was ok but the code that simulated failure wasn't behaving correctly and was returning success instead of an error code on timeout.

I've adjusted the "patch that simulates timeout" here: https://gist.github.com/PVince81/2978ed02a21876fd9e58f74beda7e5c0

In the future it would be nice to have something in the testing app so one would only need to set some oc_appconfig settings, see the idea here: https://github.com/owncloud/core/issues/32183

[PVince81]

PR was merged, will be in 10.0.10. closing