Meteor-files: Question meteor files with AWS adaptor

[haojia321]

Hello,
Im using Meteor-files + AWS adaptor in my website for video playing(206 status). I came up couple questions:

1. While playing video from browser, browser will download buffer in advance. But our issue is video will always need to wait for buffer to be downloaded. My question is, is there any config i can change to define the buffer size from Meteor-files level? We dont think it is internet speed issue.
2. We use AWS as adaptor, while upload file to AWS, will it first upload to Meteor server first?
3. While playing(download) video from AWS, will the data goes to Meteor server first then goes to user's browser? Or will it directly go from AWS to user's browser?

[dr-dimitru]

Hello @haojia321 ,
Haven't heard from you for a while :)

1. It's in control for the browser. Browser uses Range header to control buffer, and chunk size. Using interceptDownload hook you may alter the size of the chunk, by changing "end position". For more see AWS:S3 integration docs, you're looking for if (http.request.headers.range) { /*...*/ } block. __Note:__ not all browsers sends this header, and most of the browsers are willing to not set "end position", I recommend to start with logging Range header, to check how player (default or custom) manages this feature.
2. Yes, Browser -> Meteor server -> AWS:S3
3. If you follow AWS:S3 integration docs, then yes, AWS:S3 -> Meteor server -> Browser

2 and 3 is made this way for security reasons, upload/download flow control and users permissions control.

[haojia321]

Hello @dr-dimitru ,
Take play video from AWS for example. I know the file from meteor server to browser will be send as multiple chunks instead of one file.
My question is will the file send from AWS to meteor server in one file or chunks? Will this file be deleted automatically from meteor server(point to the code)? Where exactly is this file stored in meteor?

Our issue now is galaxy will crash when we play video. There is only one user and not a heavy traffic on our server when we faced the issue. From galaxy it says out of memory.

[dr-dimitru]

1. From AWS -> Meteor in chunks, see s3.getObject in current AWS integration guide
2. File pulled from AWS to Meteor is not saved anywhere and piped directly to Browser, maybe some bytes will remain for a moment in RAM due to difference in connection speed
3. Does Galaxy crash has a error? exception? what support told you?

P.S. Galaxy isn't a silver bullet, simple dedicated server or AWS EC2 will serve your needs with ease. Of course, you need to know Linux well...

[haojia321]

Galaxy only says "Application exited with signal: killed (out of memory)" from log. Any thoughts why this happens?

[haojia321]

I have attached collection code here. Would you mind help me take a look if there is any potential code can cause this issue please?

import { Meteor } from 'meteor/meteor';
import { _ } from 'meteor/underscore';
import { Random } from 'meteor/random';
import { FilesCollection } from 'meteor/ostrio:files';
import stream from 'stream';

import S3 from 'aws-sdk/clients/s3'; // http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html
// See fs-extra and graceful-fs NPM packages
// For better i/o performance
import fs from 'fs';

// Example: S3='{"s3":{"key": "xxx", "secret": "xxx", "bucket": "xxx", "region": "xxx""}}' meteor
if (process.env.S3) {
Meteor.settings.s3 = JSON.parse(process.env.S3).s3;
}

const s3Conf = Meteor.settings.s3 || {};
const bound = Meteor.bindEnvironment((callback) => {
return callback();
});

// Check settings existence in Meteor.settings
// This is the best practice for app security
if (Meteor.isServer && s3Conf && s3Conf.key && s3Conf.secret && s3Conf.bucket && s3Conf.region) {
// Create a new S3 object
const s3 = new S3({
secretAccessKey: s3Conf.secret,
accessKeyId: s3Conf.key,
region: s3Conf.region,
// sslEnabled: true, // optional
httpOptions: {
timeout: 6000,
agent: false
}
});

// Declare the Meteor file collection on the Server
OstrioVideo = new FilesCollection({
    debug: false, // Change to `true` for debugging
    storagePath: 'assets/app/uploads/uploadedFiles/videos',
    collectionName: 'OstrioVideo',
    // Disallow Client to execute remove, use the Meteor.method
    allowClientCode: false,
    chunkSize: 5 * 1024 * 1024,

    onBeforeUpload(file) {
        // Allow upload files under 10MB, and only in png/jpg/jpeg formats
        // Note: You should never trust to extension and mime-type here
        // as this data comes from client and can be easily substitute
        // to check file's "magic-numbers" use `mmmagic` or `file-type` package
        // real extension and mime-type can be checked on client (untrusted side)
        // and on server at `onAfterUpload` hook (trusted side)
        if (file.size <= 1024 * 1024 * 1024 && /mp4|mov/i.test(file.ext)) {
            return true;
        } else {
            return 'Please upload video, with size equal or less than 1GB';
        }
    },

    // Start moving files to AWS:S3
    // after fully received by the Meteor server
    onAfterUpload(fileRef) {
        // Run through each of the uploaded file
        _.each(fileRef.versions, (vRef, version) => {
            const ffprobe = require('ffprobe');
            const ffprobeStatic = require('ffprobe-static');
            ffprobe(vRef.path, {
                path: ffprobeStatic.path
            }, (err, info) => {
                if (err) {
                    console.error(err, info);
                }
                this.collection.update({
                    _id: fileRef._id
                }, {
                    $set: { 'meta.ffprobe': info }
                }, (updError) => {
                    if (updError) {
                        console.error(updError);
                    }
                    const filePath = 'videos/' + (Random.id()) + '-' + version + '.' + fileRef.extension;
                    s3.putObject({
                        // ServerSideEncryption: 'AES256', // Optional
                        StorageClass: 'STANDARD',
                        Bucket: s3Conf.bucket,
                        Key: filePath,
                        Body: fs.createReadStream(vRef.path),
                        ContentType: vRef.type,
                    }, (error, data) => {
                        bound(() => {
                            if (error) {
                                console.error(error);
                            } else {
                                const upd = { $set: {} };
                                upd['$set']['versions.' + version + '.meta.pipePath'] = filePath;
                                upd['$set']['meta.isUploaded'] = true;

                                // Update FilesCollection with link to the file at AWS
                                this.collection.update({
                                    _id: fileRef._id
                                }, upd, (updError) => {
                                    if (updError) {
                                        console.error(updError);
                                    } else {
                                        // Unlink original files from FS after successful upload to AWS:S3
                                        this.unlink(this.collection.findOne(fileRef._id), version);
                                    }
                                });
                            }
                        });
                    });
                });
            });

        });
    },


    // Intercept access to the file
    // And redirect request to AWS:S3
    interceptDownload(http, fileRef, version) {
        let path;

        if (fileRef && fileRef.versions && fileRef.versions[version] && fileRef.versions[version].meta && fileRef.versions[version].meta.pipePath) {
            path = fileRef.versions[version].meta.pipePath;
        }

        if (path) {
            // If file is successfully moved to AWS:S3
            // We will pipe request to AWS:S3
            // So, original link will stay always secure

            // To force ?play and ?download parameters
            // and to keep original file name, content-type,
            // content-disposition, chunked "streaming" and cache-control
            // we're using low-level .serve() method
            const opts = {
                Bucket: s3Conf.bucket,
                Key: path
            };

            if (http.request.headers.range) {
                const vRef = fileRef.versions[version];
                let range = _.clone(http.request.headers.range);
                const array = range.split(/bytes=([0-9]*)-([0-9]*)/);
                const start = parseInt(array[1]);
                let end = parseInt(array[2]);
                if (isNaN(end)) {
                    // Request data from AWS:S3 by small chunks
                    end = (start + this.chunkSize) - 1;
                    if (end >= vRef.size) {
                        end = vRef.size - 1;
                    }
                }
                opts.Range = `bytes=${start}-${end}`;
                http.request.headers.range = `bytes=${start}-${end}`;
            }

            const fileColl = this;
            s3.getObject(opts, function(error) {
                if (error) {
                    console.error(error);
                    if (!http.response.finished) {
                        http.response.end();
                    }
                } else {
                    if (http.request.headers.range && this.httpResponse.headers['content-range']) {
                        // Set proper range header in according to what is returned from AWS:S3
                        http.request.headers.range = this.httpResponse.headers['content-range'].split('/')[0].replace('bytes ', 'bytes=');
                    }

                    const dataStream = new stream.PassThrough();
                    fileColl.serve(http, fileRef, fileRef.versions[version], version, dataStream);
                    dataStream.end(this.data.Body);
                }
            });

            return true;
        }
        // While file is not yet uploaded to AWS:S3
        // It will be served file from FS
        return false;
    }
});

// Intercept FilesCollection's remove method to remove file from AWS:S3
const _origRemove = OstrioVideo.remove;
OstrioVideo.remove = function(search) {
    const cursor = this.collection.find(search);
    cursor.forEach((fileRef) => {
        _.each(fileRef.versions, (vRef) => {
            if (vRef && vRef.meta && vRef.meta.pipePath) {
                // Remove the object from AWS:S3 first, then we will call the original FilesCollection remove
                s3.deleteObject({
                    Bucket: s3Conf.bucket,
                    Key: vRef.meta.pipePath,
                }, (error) => {
                    bound(() => {
                        if (error) {
                            console.error(error);
                        }
                    });
                });
            }
        });
    });

    //remove original file from database
    _origRemove.call(this, search);
};

} else {
OstrioVideo = new FilesCollection({
collectionName: 'OstrioVideo',
allowClientCode: false,
chunkSize: 5 * 1024 * 1024,
onBeforeUpload(file) {
// Allow upload files under 10MB, and only in png/jpg/jpeg formats
// Note: You should never trust to extension and mime-type here
// as this data comes from client and can be easily substitute
// to check file's "magic-numbers" use mmmagic or file-type package
// real extension and mime-type can be checked on client (untrusted side)
// and on server at onAfterUpload hook (trusted side)
if (file.size <= 1024 * 1024 * 1024 && /mp4|mov/i.test(file.ext)) {
return true;
} else {
return 'Please upload video, with size equal or less than 1GB';
}
},
onAfterUpload(fileRef) {
_.each(fileRef.versions, (vRef, version) => {
const ffprobe = require('ffprobe');
const ffprobeStatic = require('ffprobe-static');
ffprobe(vRef.path, {
path: ffprobeStatic.path
}, (err, info) => {
if (err) {
console.error(err, info);
}
this.collection.update({
_id: fileRef._id
}, {
$set: {
'meta.ffprobe': info,
'meta.isUploaded': true
}
}, (updError) => {
if (updError) {
console.error(updError);
}
});
});
});
}
});
}

[haojia321]

ostrioVideo.js.zip

[dr-dimitru]

Any thoughts why this happens?

1. What size of the files you're trying to play?
2. What "network tab" in devtools says? What size of the chunks browser is requesting? My guess - player requesting whole file, you need custom player, which manages chunks in a better way. Plus to avoid server hitting its limits, change chunk size in if (http.request.headers.range) { /*...*/ } block as at AWS:S3 integration docs.

[dr-dimitru]

@haojia321 any news on this one?

[haojia321]

Hi @dr-dimitru
Sorry for the delay. We are still stuck here.

1. The size of the file is 27mb and chunk size is 5mb. Video duration is 13 seconds.
2. We are using videojs player.

We are working on deploy meteor app to a cloud server with a docker now.
We are concerned about if it will scale to deploy to a cloud server. We know it will always hit our meteor server when user play video. So even we host our video in aws s3 and supports multiple region, it will always come to the meteor server which might be a bottleneck for us.

[dr-dimitru]

1. I recommend in your case to use GridFS to store files, as anyways you won't be able to utilize AWS:S3 CDN features
2. Are you sure your browser (and videojs) are requesting video by 5mb chunks? What if you doing seek (by moving playhead on timeline)? Network Tab screenshot may help with this one...

[haojia321]

Below is a chunk from the network tab. It is 5mb in chuck and total 27mb.

Accept-Ranges:bytes
Cache-Control:public, max-age=31536000, s-maxage=31536000
Content-Disposition:inline; filename="2015-08-18%20372.MOV"; filename*=UTF-8''2015-08-18%20372.MOV; charset=UTF-8
Content-Length:5242880
Content-Range:bytes 18317312-23560191/27207738
Content-Type:video/quicktime
Date:Sun, 24 Dec 2017 02:27:35 GMT
Pragma:private
Trailer:Expires
Transfer-Encoding:chunked

[haojia321]

@dr-dimitru is below issue will happen if I use third party storage like AWS S3?https://github.com/VeliovGroup/Meteor-Files/wiki/MeteorUp-(MUP)-Usage

This question came when we were discussing to using MUP.

1. We want to compress/encode the video before upload to aws. Is this the best place to write the code?
   onAfterUpload(fileRef) {
   _.each(fileRef.versions, (vRef, version) => {
   //compress first here
   //then s3.putObject()
   });
   }

[dr-dimitru]

Yes, onAfterUpload hook is the right place for post-processing of uploaded files.

[haojia321]

Hi @dr-dimitru ,
Now I can use ffmpeg to compress the video in onAfterUpload function and I can successfully upload the compressed video to aws.
But I cannot play that video after upload to aws. Below is my onAfterUpload function. Could you help me to see if anything is wrong here please?

[haojia321]

onAfterUpload(fileRef) {
    _.each(fileRef.versions, (vRef, version) => {
        const filePath = 'files2/' + (Random.id()) + '-' + version + '.mp4';
        var self = this;
        var processedFilePath = fileRef._storagePath + '/processed-' + fileRef._id + '.mp4';
        var ffmpeg = require("ffmpeg");
        var ffmpegProcess = new ffmpeg(vRef.path);
        ffmpegProcess.then(function(video) {
            video.addCommand('-vf', '"scale=\'min(1280,iw)\':\'min(720,ih)\':force_original_aspect_ratio=decrease,pad=1280:720:(ow-iw)/2:(oh-ih)/2"');
            video.addCommand('-c:v', 'libx264');
            video.save(processedFilePath, function(error, file) {
                s3.putObject({
                    // ServerSideEncryption: 'AES256', // Optional
                    StorageClass: 'STANDARD',
                    Bucket: s3Conf.bucket,
                    Key: filePath,
                    Body: fs.createReadStream(processedFilePath),
                    ContentType: vRef.type,
                }, (error, data) => {
                    bound(() => {
                        if (error) {
                            console.error(error);
                        } else {
                            console.log(data);
                            // Update FilesCollection with link to the file at AWS
                            const upd = { $set: {} };
                            upd['$set']['versions.' + version + '.meta.pipePath'] = filePath;

                            self.collection.update({
                                _id: fileRef._id
                            }, upd, (updError) => {
                                if (updError) {
                                    console.error(updError);
                                } else {
                                    // Unlink original files from FS after successful upload to AWS:S3
                                    var f = self.collection.findOne(fileRef._id);
                                    //console.log(this.link(f));
                                    self.unlink(self.collection.findOne(fileRef._id), version);
                                }
                            });
                        }
                    });
                });
            });
        }, function(error) {
            console.log(error);
        });
    });
}

[dr-dimitru]

Paste code in here looks ugly so I uploaded the file here so that you can view more clearly.

I've fixed it. Consider reading this docs.

1. ffmpegProcess.then callbacks should be wrapped into bound(() => { /*...*/ });
2. video.save callback should be wrapped into bound(() => { /*...*/ });
3. Have you tried to play videos without using AWS:S3, does processed videos playing locally?
4. Have you updated metadata? With new:
   
   
   
   • Filesize
   
   
   • Mime-type (if changed)
   
   
   • Other options
   
   

[dr-dimitru]

But I cannot play that video after upload to aws.

1. Are you getting an error?
2. What error?
3. Are you getting error in a Browser or Server?
4. What "Network" tab can tell us? (headers, response code, etc.)

[haojia321]

ffmpegProcess.then callbacks should be wrapped into bound(() => { /.../ });
video.save callback should be wrapped into bound(() => { /.../ });

bound(() => { /.../ }) is called when file is uploaded to AWS. Here we need to first process the video, then upload to aws, right? So I think bound(() => { /.../ }) should inside video.save callback?

Have you tried to play videos without using AWS:S3, does processed videos playing locally?

I think the issue here is collection can only knows the original uploaded video. It doesnt aware of the processed video.

Have you updated metadata?

I didn't. Is this the cause? File size definitely smaller after compression.

[haojia321]

This is the source url from video player. If I paste it in a new tab. I see errors.

Request URL:http://localhost:3000/cdn/storage/AttachmentFile/e4kPM9tnCFfxdwGCA/original/e4kPM9tnCFfxdwGCA?play=true
Referrer Policy:no-referrer-when-downgrade
Request Headers
**Provisional headers are shown**
Accept-Encoding:identity;q=1, *;q=0
Range:bytes=458752-
Referer:http://localhost:3000/cdn/storage/AttachmentFile/e4kPM9tnCFfxdwGCA/original/e4kPM9tnCFfxdwGCA?play=true
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
Query String Parameters
view source
view URL encoded
play:true

[dr-dimitru]

@haojia321

1. As I've understood you aren't getting any errors, the video just not playing, right?
2. I highly recommend try to play processed video locally (without involving AWS:S3)
3. Have you tried to play video in another browser?
4. Have you tried to play video in a native player or other JS-player?

//cc @mksh-su need help to locate the issue, with reproducible steps.

[haojia321]

Hi @dr-dimitru

1. a. If I watch video from the page itself, then no errors from network. However if I inspect the video tag and copy src and open src url from a new tab. I see errors(see above comment).
2. Processed video can be found in assets/app/uploads/uploadedFiles. I can play it on local player without any issue. Since processed video is uploaded to AWS, I can also play it directly from AWS deeplink.
3. It doesnt work in Chrome but sometimes it works in firefox, sometimes it doesnt...
4. Native player works like mentioned in point 2.

[UPDATE] I see error in server now. Sorry for the wrong info before.

{ InvalidRange: The requested range is not satisfiable
W20180117-19:22:15.666(-5)? (STDERR)     at Request.extractError (/Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/services/s3.js:577:35)
W20180117-19:22:15.666(-5)? (STDERR)     at Request.callListeners (/Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
W20180117-19:22:15.666(-5)? (STDERR)     at Request.emit (/Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
W20180117-19:22:15.666(-5)? (STDERR)     at Request.emit (/Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/request.js:683:14)
W20180117-19:22:15.667(-5)? (STDERR)     at Request.transition (/Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/request.js:22:10)
W20180117-19:22:15.667(-5)? (STDERR)     at AcceptorStateMachine.runTo (/Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/state_machine.js:14:12)
W20180117-19:22:15.667(-5)? (STDERR)     at /Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/state_machine.js:26:10
W20180117-19:22:15.667(-5)? (STDERR)     at Request.<anonymous> (/Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/request.js:38:9)
W20180117-19:22:15.667(-5)? (STDERR)     at Request.<anonymous> (/Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/request.js:685:12)
W20180117-19:22:15.667(-5)? (STDERR)     at Request.callListeners (/Users/I839603/edueex/code/prototypes/videoaws/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
W20180117-19:22:15.668(-5)? (STDERR)   message: 'The requested range is not satisfiable',
W20180117-19:22:15.668(-5)? (STDERR)   code: 'InvalidRange',
W20180117-19:22:15.668(-5)? (STDERR)   region: null,
W20180117-19:22:15.668(-5)? (STDERR)   time: 2018-01-18T00:22:15.665Z,
W20180117-19:22:15.668(-5)? (STDERR)   requestId: 'B438173E579484AB',
W20180117-19:22:15.669(-5)? (STDERR)   extendedRequestId: 'O4/m4joo1c7lM8T7/3EwKuW345sXTEjcwFWlbP+6nakoCLjABzo2gKNprcpSeebnqEVEEXcoUAM=',
W20180117-19:22:15.669(-5)? (STDERR)   cfId: undefined,
W20180117-19:22:15.669(-5)? (STDERR)   statusCode: 416,
W20180117-19:22:15.669(-5)? (STDERR)   retryable: false,
W20180117-19:22:15.669(-5)? (STDERR)   retryDelay: 49.171270532532475 }

[haojia321]

If you see onAfterUpload function with code Body: fs.createReadStream(processedFilePath), Im not uploading the original video to AWS. Im uploading the processed video to AWS. This actually brings inconsistency between the collection info in mongodb and the processed video file. For example I have converted the video from MOV to MP4. But in mongodb record, type is still video/quicktime. Size is also not consistent. Will this be the reason to cause the issue?

Any suggestions how to fix the inconsistency?

[dr-dimitru]

Will this be the reason to cause the issue?

As I've said before you should update file's meta-data.

Any suggestions how to fix the inconsistency?

Update file's meta-data or update MongoDB's record with new file's subversion. See this wiki doc for reference.

[haojia321]

How does Meteor-Files getting a file's size under the hood?
Looks like ffmpeg cannot get a video files size.

[dr-dimitru]

Not funny, it's node.js, - use fs.stat.
This discussion is already far beyond ostrio:files package.

[haojia321]

Thanks for your help. The issue is I need to update the size from mongodb. Video will play after update it.
Since this issue is not ostrio:files related, we can close this issue now.

[haojia321]

@dr-dimitru I have one more question related to ostrio:files. Below is a copy of file record from mongodb. We already have file info inside the versions array. Why do we still need to put similar file info like size, extension in the root of this record?

{
    "_id" : "d4AusDrgCmNWgNSNm",
    "size" : 6613307,
    "type" : "video/mp4",
    "name" : "output_5173.mp4",
    "meta" : {},
    "ext" : "mp4",
    "extension" : "mp4",
    "extensionWithDot" : ".mp4",
    "mime" : "video/mp4",
    "mime-type" : "video/mp4",
    "userId" : null,
    "path" : "assets/app/uploads/uploadedFiles/d4AusDrgCmNWgNSNm.mp4",
    "versions" : {
        "original" : {
            "path" : "assets/app/uploads/uploadedFiles/d4AusDrgCmNWgNSNm.mp4",
            "size" : 6809260,
            "type" : "video/mp4",
            "extension" : "mp4",
            "meta" : {
                "pipePath" : "files2/cGequZeMknDrMAH9X-original.mp4"
            }
        }
    },
    "_downloadRoute" : "/cdn/storage",
    "_collectionName" : "AttachmentFile",
    "isVideo" : true,
    "isAudio" : false,
    "isImage" : false,
    "isText" : false,
    "isJSON" : false,
    "isPDF" : false,
    "_storagePath" : "assets/app/uploads/uploadedFiles",
    "public" : false
}

[dr-dimitru]

1. It happened historically, before versioning was implemented;
2. You don't need to publish all those fields to the Client, so only needed fields could be picked for publishing;