Uppy: Move limiting to different point, to fix StatusBar and other UI issues

[ogtfaber]

I'm experiencing various issues uploading files in bulk using XHRUpload (through the AWSS3). Just a general question if Uppy is designed to work for uploading big amounts of files?

I'm getting timeout errors (upload stalled for 30 seconds). Does the timeout counter start from the moment it's "prepairing the uploads"? In the case of AWS or Gcloud, this means retreiving the signed urls, which can take some time if we have to do this for many files.

And I'm guessing the dashboard plugin is not the most logical choice in this case.

[kvz]

Generally speaking, I think it’d be nice to do a 10K milestone next year, where we try to upload 10k files as well as 10k MBs (so yeah, 10GB ;) through to the whole uppy/server/tusd/transloadit chain and make sure all components can handle it. Not that it’d be the primary/majority use case that we, but it might help uncover some bugs, memory leaks, optimization kinks that’d be nice to iron out before 1.0. It’s something we haven’t prioritized yet tho so that will explain why you’re hitting some issues with this number. What’s your timeframe on this?

[ogtfaber]

It's the primary use-case of the uploader, (CD's with MRI scans, to be precise), which can have 1000's of files on them.

I've figured out the uploader works a lot better if I put a limit to the concurrent uploads in the XHRUpload component, but this option isn't currently accessible from the AWSs3 plugin if I'm correct. However, if I do this the statusbar isn't displaying the uploads correctly, since it assumes it's only doing "limit"-number op uploads at the time. But I think with a few tweaks it'll be a lot easier on the browser.

[goto-bus-stop]

However, if I do this the statusbar isn't displaying the uploads correctly, since it assumes it's only doing "limit"-number op uploads at the time.

oh yeah that's a good find! we should apply the limiting at a different point ...

[ogtfaber]

I created a PR for passing XHR options https://github.com/transloadit/uppy/pull/471

[ogtfaber]

@goto-bus-stop That would be great! Let me know if you need any help

[ogtfaber]

@goto-bus-stop I assume once the limiting function is moved to another point, the "upload +### button" issue will be solved, too?

[goto-bus-stop]

Yeah, it should fix that too!

[ogtfaber]

@goto-bus-stop do you need any help moving the limiting to another point? I don't see it on the changelog list for .23, not sure how much work this is either. Anyway, happy to help - let me know.

[goto-bus-stop]

tackling this today!

[ogtfaber]

@goto-bus-stop thanks!