Uppy: Massive number of files upload

[zeus19900814]

Hi,

I was wondering about the scalability of Uppy so I tried to upload ~30k files by dragging and dropping a folder that contains these files. It took ~3 mins to render the file icons in the Uppy dashboard. Is there any way to speed up this step? I have set the option disableThumbnailGenerator to true.

It seems like, for me, the number of files under 500 wouldn't give me a noticeable slowdown while adding files.

I found that Uppy spent most of time rendering the icons of files. Can we create an icon for the folder instead of creating icons for all the files?

Thank you very much!

[steverob]

I'm looking to integrate Uppy into a project that has a similar requirement. Uppy & the page gets sluggish when I'm just at 400 or so images. Wondering how we can support adding thousands of images at a time.

[zeus19900814]

Also, I saw that no matter how fast the file is uploaded, "upload-progress" event will be triggered at least twice, 0 percent uploaded and 100 percent uploaded. This could cause a symptom that when uploading many files, they are already uploaded shown on the dashboard but "upload-progress" is still triggered. Looks like they are queued in a command queue and need to be executed. I set Tus limit option to 5.

I found #1693 and #1736 are kinda related but I am still not sure how to fix this issue. i.e. Stop emitting "upload-progress" if "upload-success" is called.

[arggh]

This issue is somewhat related: https://github.com/transloadit/uppy/issues/1775

One idea that comes to mind is lazy rendering. When presented with more than 10 images, render the first 10 and an indicator/spinner below them, start the upload, then render another 10. Rine and repeat. This way the CPU isn't burning hot for 3 minutes straight & the upload begins almost immediately.

[goto-bus-stop]

There are a bunch of performance bottlenecks right now. At least these:

• If you're not using the limit: option in your uploader plugin, Uppy is going to synchronously create hundreds of requests, which is expensive and locks up the browser for a while
  
  
  
  • We will likely set a conservative default limit in a future major release, something like 10
  
  
• Thumbnail generation is still kinda expensive
  
  
  
  • Only generating thumbnails once the image is visible in the dashboard could help
  
  
  • We could look into doing it in a worker again, so it would still be slow but not lock up the UI
  
  
• During uploads, each file emits lots of progress updates, which causes the dashboard to rerender often, and rerenders are not fast enough
  
  
  
  • Only rendering visible parts of the image grid could help here too, to save Preact a lot of diffing work
  
  

[arggh]

We could look into doing it in a worker again, so it would still be slow but not lock up the UI

I actually did some research in this direction, nothing to show yet though, but according to what I read, this would probably also have a positive impact on the speed of thumbnail generation.

Only generating thumbnails once the image is visible in the dashboard could help

As long as it gets generated before the file is uploaded, if the waitUntilThumbnailsGenerated flag is set. Related to this, I think it would be better to wait on a per-file-basis, not until all files have thumbnails. More about that here: https://github.com/transloadit/uppy/issues/1775#issuecomment-546513472

Only rendering visible parts of the image grid could help here too, to save Preact a lot of diffing work

Lazy rendering would probably be the silver bullet here. If it were my project, I would re-write the UI in Svelte instead of Preact. Svelte's performance is insane, startup overhead is ~zero and there is no actual runtime (only heavily treeshaken helper functions as-needed, if those count), or diffing for that matter.

[arturi]

Hi, @zeus19900814, @steverob, @arggh, https://github.com/transloadit/uppy/pull/1949 was merged and released in Uppy 1.8, it significantly improves performance of adding files. There’s still work to be done, but this is a big step already. Would you care to try and let us know how it goes for you?

[zeus19900814]

Thank you all for your effort! Unfortunately, I didn't see obvious speedup adding files with Uppy 1.8. Adding ~30k files still takes ~3mins for the UI to respond. Is there any setting that I need to configure?

[steverob]

Hi Zheng, we’ll try this out as well but more interested in handling the < 5k range. Do you mind reporting how it works for that kind of a load? Thanks :)

--
Regards
Steve Robinson

On 01-Feb-2020, at 4:52 AM, Zheng Qin [e-mail] wrote:


Thank you all for your effort! Unfortunately, I didn't see obvious speedup adding files with Uppy 1.8. Adding ~30k files still takes ~3mins for the UI to response. Is there any setting that I need to configure?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[zeus19900814]

Adding 5k files drops from ~25s with Uppy 1.6 to ~15s with Uppy 1.8.

[goto-bus-stop]

I guess it's more varying than I'd hoped! Adding 2k files went from around 20ish seconds to 6 seconds for me in Chrome, and from IIRC 8ish to less than 1s in Firefox, but your numbers are not nearly as good. But all of the gains so far were made in the code that adds the files to an internal data structure, so it may still spend a truckload of time rendering all those thousands of elements which are out of view. I think we should be able to get list virtualization and lazy thumbnails in the dashboard before the end of the month (so that would be in 1.10).

[arggh]

Lazy thumbnails would be a huge performance win! And by lazy, I mean, load only the ones in view, but also the ones that need EXIF-data to be extracted just before upload :)

I just took the latest version of Uppy for a spin. Adding just ~500 _largish_ (7-9Mb) images takes an eternity, completely stalling the upload process.

[mat5879]

Seeing massive improvement with Uppy 1.8 and the performance improvements. Thanks guys! ❤