Dropzone: Chunks?

[bschmitt]

Hi,
I just want to ask if it is planned to implement chunk uploads to avoid server limits?

Maybe I'm wrong but currently each file is transferred to the server in a single request and large files will run into a server timeout if the max post size is lower than the file size.

HTML5's blob.webkitSlice(start, end) or blob.mozSlice(start, end) could be used to split the file.

Is somebody already working on such a feature? Otherwise I give it a try, although I'm a total noob when it comes to CoffeeScript...

[alancolyer]

I second this. Server upload limits can be a problem especially in a shared hosting environment where you are restricted in what you can change. If chunked uploads would get around that it would be the icing on the cake :)

[enyo]

Well.. correct me if I'm wrong, but you have to handle chunked file uploads differently on the server.

So: if Dropzone uploaded chunks of the files, the server would need to handle those chunks, and combine them. Which means that there would need to be a server implementation for this.
I can't think of an environment where you have the freedom to use such a server side service, but _can't_ change the max upload size.

Closing for now. Feel free to continue the discussion.

[alancolyer]

True, you'd need to write something server-side to handle it - but as far as I'm aware you should be able to do that with vanilla PHP without any special server configuration.

In my case, the shared hosting environment we often use is very limited in terms of changing php.ini directives or apache config directives but anything else is fine, you are mostly free to use anything you can write in PHP so long as it doesn't use shell_exec(); or similar.

Here's a question from stackoverflow, one of the answers to which details a server side PHP chunked-upload handling script: http://stackoverflow.com/questions/9011138/handling-pluploads-chunked-uploads-on-the-server-side

[bschmitt]

The server script is no big deal. You have to append the file parts which were chunked by dropzone in a temp file. E.g. as described here:
http://stackoverflow.com/questions/7853467/uploading-a-file-in-chunks-using-html5
http://creativejs.com/tutorials/advanced-uploading-techniques-part-1/

To guarantee that each chunk will result in the right target file, the dropzone chunk AJAX request has to be extended with an additional parameter (e.g. target=myUpload.zip).

I can't think of an environment where you have the freedom to use such a server side service, but can't change the max upload size.

Nearly every PHP webspace allows to store temp files but the fewest allow changing the max post size or upload limits.

But that's not my case. I don't want to extend the timeouts at our servers which is the reason if a user wants to upload a large file with a slow connection in a single request. Beyond if the upload limit is high and the run-time timeout long you can easily put the webserver in a false position.

[john-dev]

Hello, i would love to see chunked uploading too!!! It's actually a must have for such a nice uploader.
Cheers,
john

[mfrederico]

Looks like this guy may have created a solution:
https://github.com/bootle/DropZone

[haydnhkim]

+1

[vedmant]

I for this feature as well as I forced to use different upload plugin with chunking feature.

[slashrsm]

I can't think of an environment where you have the freedom to use such a server side service, but can't change the max upload size.

I kind of disagree... There are cases (specially VERY big files), where you want chunked uploads even if you're able to increase server's upload limits.

The server script is no big deal. You have to append the file parts which were chunked by dropzone in a temp file.

True. Back-end part is a no-brainer. It also makes sense to make cunked uploads optional. This will allow people that don't want to deal with chunks to keep using "standard" approach.

+1 to this feature. Would be great if this issue would be re-opened.

[blop]

+1

Chunk uploads can help with big files and connection loss also.
See http://www.resumablejs.com/ which has a nice approach on this.

You only need to make this optional and documented, people will implement the server part themselves in their favorite language.

[evoactivity]

Is there a reason this issue is closed? This would be a useful feature, I saw another issue where it was suggested @enyo might look at making it possible to integrate resumablejs, has any progress been made there?

[Marcovecchio]

In fact, using chunks is even better for people on shared hosts: people would be able to upload any size, with no need to touch the max upload size configuration.

On client side, the upload would have to be broken into several calls to the upload handling script, which would be very simple, something like this:

// Complete path of the uploaded file.
$filePath = "somewhere/uploads/".$_FILES["file"]["name"];;
// $chunks is the total number of chunks.
$chunks = isset($_GET["chunks"]);
// $chunk is the number of the current chunk (zero based).
$chunk = isset($_GET["chunk"]);

// Open output file, appending or overwriting.
$out = fopen($filePath.".part", $chunks ? "ab" : "wb");
// Open temp file for reading.
$in = fopen($_FILES["file"]["tmp_name"], "rb"));
// Copy contents.
while ($buff = fread($in, 4096)) { fwrite($out, $buff); }
// Close files.
fclose($out);
fclose($in);
// Check if the current chunk is the last one. If yes, rename the file to it's actual name.
if (!$chunks || $chunk == $chunks - 1) rename($filePath.".part", $filePath);

[jundl77]

So I think I got this working with resumable.js.

Of course you need to inlcude the resumable.js file, then change the
Dropzone.prototype.uploadFiles function in the following way:

Here is how I implemented it in my project.

Dropzone.prototype.uploadFiles = function (files) {
    var resumable = new Resumable ({
        target:'path/to/server/side/code',
        maxFiles: Dropzone.prototype.defaultOptions.maxFiles,
        simultaneousUploads: Dropzone.prototype.defaultOptions.parallelUploads,
        testChunks: true
    });

    if (resumable.support) {
        for (var j = 0; j < files.length; j++) {
            var fileLocal = files[j];
            resumable.addFile(fileLocal);
        }

        resumable.on('fileAdded', function (file) {
            resumable.upload();
        });

        resumable.on('fileProgress', function (file) {
            var progressValue = Math.floor(resumable.progress() * 100);
            Dropzone.prototype.defaultOptions.uploadprogress(file.file, progressValue, null);
        });

        resumable.on('fileSuccess', (function(_this) {
            return function (file) {
                return _this._finished([file.file], "success", null);
            }
        })(this));

        resumable.on('error', (function(_this) {
            return function (message, file) {
                return _this._errorProcessing([file.file], message, null);
            }
        })(this));
    }  else {
        // Otherwise use the old upload function
    }
}

The server side code (in php) can be found here.

[ghost]

@enyo huh? it has little to do with the freedom of environment, its pretty common to do chunked uploads, large bins, videos maybe? what do you suggest? change the nginx config & php config just to upload the videos and set it back to sensible settings? :8ball:

[ghost]

@jundl77 nice job

[nodefortytwo]

Agreeing with everyone here, chunked uploads should be supported. Large file uploads are pretty common these days, videos etc. posting at 5gb video isn't nice, upload limits can be changed but large timeouts can be detrimental to the rest of the app.

@jundl77's solution should do the trick though.

[eltoro]

I've implemented @jundl77's solution and it works but Dropzone isn't firing the success and error events. Anyone get it to work?

Dropzone.prototype.uploadFiles = function (files) {
    var resumable = new Resumable ({
         target: url,
         maxFiles: 1,
         testChunks: true
    });
    ...
    //Events
    resumable.on('fileSuccess', (function(_this) {
        return function (file, message) {
             return _this._finished(file.file, JSON.parse(message), null);
        }
    })(this));
}

[jundl77]

@eltoro It looks to me like you switched the file and message arguments since I had:

    return function (message, file) {
        return _this._errorProcessing([file.file], message, null);
    }

Not sure if that fixes it.

[eltoro]

This works for me:

    var _this = this;

    resumable.on('fileSuccess', function(file, message) {
        _this._finished.call(_this, [file.file], JSON.parse(message));
    })

[Abdualiym]

jundl77 good job, thank you

[shivanraptor]

Nice work @jundl77 . I'm creating DropzoneJS dynamically (e.g. 10 Dropzones in a page, with different URL). How can I use your codes to apply to all Dropzones ?

[eltoro]

I'm setting the url programmatically for a couple Dropzones like this. And then I set the options for each Dropzone:

    Dropzone.prototype.uploadFiles = function (files) {
            var _this = this;
            var resumable = new Resumable ({
                target: (updateURL) ? updateURL : apiUrl.getUrl("dataset"),
                maxFiles: 1,
                chunkSize: 5*1024*1024,
                simultaneousUploads: 5,
                testChunks: true
            });

            if (resumable.support) {
                for (var j = 0; j < files.length; j++) {
                    var fileLocal = files[j];
                    resumable.addFile(fileLocal);
                }

                resumable.on("fileAdded", function (file) {
                    resumable.upload();
                    mapUtils.openLoadingModal("Processing your Dataset Upload");
                });

                resumable.on("fileSuccess", function(file, message) {
                    if(message && message.length > 0) {
                        _this._finished.call(_this, [file.file], JSON.parse(message));
                    }
                });

                resumable.on("error", function(file, error) {
                    _this._errorProcessing.call(_this, [file.file], error);
                });
            }
        }

[Mr-Zer0]

+1
I also would like to get that feature. I don't wanna use other file upload js library. I really like dropzone. If I get chunk upload feature in dropzone, it will be perfect.

[reneklacan]

+1

[MrLei]

Hello eltoro only yours cod need to use dropzone chunked file ? of course js code not backend php?

[MrLei]

Ok.. example @eltoro work ok :-) but i have next problem when i chcek when all file finish
init:
function () {
this.on("maxfilesexceeded", function(file){ });
this.on("sending", function (file) { });
this.on("success", function (file) {
if (this.getQueuedFiles().length === 0 ) { ...

getQueuedFiles().length show 0 when first part of file stop upload.
How i can change to whait when all part file stop upload?
How i can modify status file?

[MrLei]

ok...i check other status Upload files when all stop: this.getUploadingFiles().length === 0

[kallex]

Seems to work cleanly! I only had to add cancel hook as well.

`
var currentResumable;
Dropzone.prototype.uploadFiles = function (files) {
var resumable = new Resumable ({
...
});
currentResumable = resumable;
...
};

Dropzone.prototype.cancelUpload = function () {
  currentResumable.cancel();
};

`

[patrickdorival]

Why has this been closed?
It would make dropzone rival fine uploader in every way then.

[gordonlhw]

Can't upload multiple files, if i select more than one files, but always upload 1 file

Any solution to solve? Please help... Thank you...

Dropzone.prototype.uploadFiles = function (files) {
var _this = this;
var resumable = new Resumable ({
target: (updateURL) ? updateURL : apiUrl.getUrl("dataset"),
maxFiles: 1,
chunkSize: 510241024,
simultaneousUploads: 5,
testChunks: true
});
...
...
}

[jundl77]

@doz87 I agree, it would be cool to see this implemented.

[Dodotree]

That is a really old thread, but Google directed me here when I was looking for server script to support chunks. At the end I rewrote script I used for Resumables and PHP. Hope somebody will find it useful.

https://github.com/Dodotree/DropzonePHPchunks/tree/master

[shivanraptor]

Since version 5.2.0, chunked upload is officially supported by Dropzone. See official documentation: https://gitlab.com/meno/dropzone/wikis/faq#chunked-uploads