Uppy: Multiple errors when uploading 300+ images at once

Created on 31 Jul 2019 · 8Comments · Source: transloadit/uppy

Is there any limit on how many files we can upload at once or how large that set can be?

I'm getting multiple errors when trying to upload 300+ JPG images (~5MB each) at once. That doesn't happen when trying to upload a small set of 10 images.

First couple hundred errors are as follows:

XMLHttpRequest cannot load https://master.tus.io/files/e335fe8ccbdb27a14b20740319fbecfa+cgrPnQ2KVCPXsDJRqIVQLXokc2ezFTSOrIk_2d6d2SOmlVeFgjpL.v8EY58EC_qF2rpK43D3nrKpbQ2XVgMb4AR8V5stcsrd993.jGAd0xYSYU4nJruQx_NGnNWiq4UI due to access control checks.
https://master.tus.io/files/e335fe8ccbdb27a14b20740319fbecfa+cgrPnQ2KVCPXsDJRqIVQLXokc2ezFTSOrIk_2d6d2SOmlVeFgjpL.v8EY58EC_qF2rpK43D3nrKpbQ2XVgMb4AR8V5stcsrd993.jGAd0xYSYU4nJruQx_NGnNWiq4UI
Then Failed to load resource: Origin null is not allowed by Access-Control-Allow-Origin. and Origin null is not allowed by Access-Control-Allow-Origin.

Then errors become similar to this: https://master.tus.io/files/d4113b9d7b617064ebc26fe5c0bf7fc5+7OMAGTSdZn3KWDzYDtcVJeqY8_C1xFWBMios6Fv5bs978gV7_r9x8g8yvupE92JQeDV1529wHXxhhuMJz3mSW3VhZiVNEn004kvCg_cBbvI95cvm2AhoYAjMANoVl0gt
Failed to load resource: the server responded with a status of 500 ()

When it gets around 4-5% of progress, the UI starts showing the Retry button.

Console indicates 3.3K requests and 2.3K errors.

I'm on a Mac (v10.14.6). Tried with both Safari (v12.1.2) and Chrome (v76.0.3809.87). Also, there is no significant peak of memory usage.

Our regular use consists of users uploading sets of 1K+ images and a few videos with up to 20GB each.

The code i'm refering to is the first sample from https://uppy.io/docs/

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Uppy</title>
    <link href="https://transloadit.edgly.net/releases/uppy/v1.3.0/uppy.min.css" rel="stylesheet">
  </head>
  <body>
    <div id="drag-drop-area"></div>

    <script src="https://transloadit.edgly.net/releases/uppy/v1.3.0/uppy.min.js"></script>
    <script>
      var uppy = Uppy.Core()
        .use(Uppy.Dashboard, {
          inline: true,
          target: '#drag-drop-area'
        })
        .use(Uppy.Tus, {endpoint: 'https://master.tus.io/files/'})

      uppy.on('complete', (result) => {
        console.log('Upload complete! We’ve uploaded these files:', result.successful)
      })
    </script>
  </body>
</html>

Bug Tus

Source

andrebr

Most helpful comment

I agree the non-zero default is a sensible choice, like maybe 10? @goto-bus-stop, what do you say?

arturi on 15 Aug 2019

👍2

All 8 comments

/cc @Acconut

kvz on 31 Jul 2019

Thanks for bringing this to our attention, that's very interesting! I don't have a concrete idea what could cause this right now but I will try to reproduce it.

In the meantime, could you try to record your network activity in a HAR file (see https://toolbox.googleapps.com/apps/har_analyzer/ for instructions)? This would help us a lot to analyze the individual requests and response to see what the actual problems are. If you don't want to post the HAR file publicly as it may contain personal information, you can also mail it to me personally at [email protected].

Besides that, do you know what the minimum number of images is to reproduce it? For example, does it also happen with 100 images or are 300 required to reproduce it.

@arturi Does Uppy have a feature to limit the number of concurrent uploads? That may also help in these situations, although it looks like a tusd problem for now.

Acconut on 1 Aug 2019

@Acconut There are limit options for every “official” uploader plugin, including tus: https://uppy.io/docs/tus/#limit-0. There are some issues with it, when the upload is canceled, but that is being worked on already here: https://github.com/transloadit/uppy/pull/1736.

arturi on 1 Aug 2019

@andrebr Did you have the opportunity to generate the HAR file yet?

I tried to reproduce the problem on by own but didn't get the same results as you did. I don't get CORS errors or 500 status codes but Chrome cancels some requests on its own. Now exactly sure what's happening but I will keep digging.

Acconut on 12 Aug 2019

@Acconut to be honest, we decided to move forward with AWS SDK JS as just need images and videos to be uploaded to S3, we don't need to support any other platform and don't want to display thumbs during progress, specially because we're sending thousands at once.

But even when implementing the AWS solution, had the same issue as with Uppy. It was trying to upload all the files at the same time. Once selecting the 300+ files to upload, it was kicking 300+ uploads in parallel, so they were fighting for bandwidth and end up with an error related to "not enough resources". For fixing that, we implemented a simple queue for controlling how many uploads we'd like to process in parallel to just 8.

That's probably what the limit config you mentioned earlier would take care of.

Here is the link for the HAR file: https://d.pr/free/f/naCPVm

Hope it'll help you guys debug it. Thanks!

andrebr on 14 Aug 2019

@andrebr Thank you very much the HAR file, it helped a lot! I hope your S3 solution is working better for you.

From what I can deduct from the network transfer, I come to the some conclusion as you. The network is getting overwhelmed by the 300+ uploads and the requests are just timing out, so tusd is starting to close connections to protect itself. The limit option would likely have prevented this.

@arturi What do you think about setting a non-zero default value for limit or emitting a warning if many uploads are started at the same time with limit: 0? This could many user/customers who do not test for this edge case.

Acconut on 15 Aug 2019

I agree the non-zero default is a sensible choice, like maybe 10? @goto-bus-stop, what do you say?

arturi on 15 Aug 2019

👍2

We’ve added this to backlog and will set a non-zero default in Uppy 2.0. Thanks!

arturi on 29 Sep 2020

Was this page helpful?

0 / 5 - 0 ratings