Meteor-files: Upload concurrency settings

[tuarrep]

Hi! Thanks for this useful package :+1:

I'm currently encountering a problem with uploading multiple files, after a certain number of concurrently files server answer with HTTP 403.

Problem solved see https://github.com/VeliovGroup/Meteor-Files/issues/245#issuecomment-250434530 & https://github.com/VeliovGroup/Meteor-Files/issues/245#issuecomment-250458020

I've a solution to propose.

The problem

Description

I've a file fileds which allows user to upload one or more files (multiple=true).
When user upload multiple files I loop over them and call FilesCollection.insert for each files.
It works pretty good util a certain amount a files (3 to 5 depending on size). In this case server reply with 413 Entity too large
After some trying all the files are finally uploaded correctly but it takes very long time comparing to upload these files individually.

Technical details

Versions

• Meteor 4.1.1

• MeteorFiles 1.7.3

  My code

• File collection creation:

export const Files = new FilesCollection({
  collectionName:  'files',
  allowClientCode: false,
  downloadRoute:   "./media",
  debug:           true,
                   onBeforeUpload, //Just like in doc (file check)
                   onAfterUpload // Nothing really interesting in our case
});

• Function called to insert a file:

const insertFile = (file, domain, siteId, meta) => {
  return new Promise((resolve, reject) => {
    Files.insert({
      file:      file,
      meta:      {
        domain,
        siteId,
        ...meta
      }
    }, false)
      .on('error', (error, fileRefs) => {
        reject(error, fileRefs);
      })
      .on('end', (error, fileRefs) => {
        if (!error) {
          resolve(fileRefs);
        } else {
          reject(error, fileRefs);
        }
      })
      .start();
  })
}

I loop over files and call this function. Files are processed before inserting (get md5 to find duplicates for example) so I don't put the code here but it can be considered as a basic forEach on the files array.

Example of failed request

In this request I'm trying to upload 8 files and the eight are uploading in the same request and the server reply with 413.

A solution?

Facing this I thought about using queues and I've found this package: https://github.com/d3/d3-queue

Concept

This package allow us to create a queue with async functions (simple example here: http://bl.ocks.org/mbostock/1696080). When creating this queue we define the concurrency (number of parallel calls). Each successful functions are removed from the queue until no one still. When the queue is empty a callback is called.

In our case

For me it can be a solution to the problem I describe above.
Each upload request are queued before starting sending them instead of grouping them into a payload. Next the package automatically control the "flow" of requests.

The concurrency could be set in constructor like the chunk size

We could imagine something like that:

const queue = d3.queue(settings.concurrency || 2);
files.forEach((file) => {
    queue.defer(fn(file)); // fn() manage file uploading
});
queue.awaitAll(/*Call a callback here*/);

Why I don't do this in my code?

First because I use Redux and all upload call are made by dispatching actions and each these actions are independent from the others. It's difficult to me to make between each files.
And because the request seems to already be aggregated by your package.

I would be glad to discuss this with you :smile:

[dr-dimitru]

Hi @mafzst

By your screenshot I see what somewhy DDP (WebSockets) is not enabled on your server. And SockJS (used internally in Meteor) falls back to XHR. Then nginx receives big body rejects it.

There is three ways to solve it:

1. Enable WebSockets
2. Tune nginx to accept bigger body on POST requests
3. Switch upload protocol from DDP to HTTP

It is not related to this package, but to your server environment setup

[tuarrep]

Damn you've right! Thanks for your help :+1:
BTW what do you think about this feature. It can be useful in case where user haven't access to the nginx configuration, no?

---

For information(for other people who find this issue), I've chosen your first option.
For this you must to edit nginx vhost file by adding

map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

before the server block and

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

in the location block (at the same level as proxy_pass)

[dr-dimitru]

It can be useful in case where user haven't access to the nginx configuration

That's why developers do code. And DevOps do environment. DevOps always has access to any configuration

[tuarrep]

Yes you've rigth :) Forget that