Meteor-files: How to access files when S3 is integrated?

[vanshady]

So this previously works before S3, I can access the file in client with no problem, but after I followed S3 tutorial, when I click Collection.findOne(...).link(), it says File Not Found :(.

I checked that file is uploaded to S3 and the versions.original.meta.pipePath is correct. Am I accessing it incorrectly with link() or is there anything wrong with my setup?

Thanks!

Meteor-Files: 1.9.10
Meteor: 1.6.1
OS: Mac Browser:Chrome

Debug info:

I20180419-01:56:00.295(-4)? [FilesCollection] [Upload] [DDP Start Method] Got #-1/1 chunks, dst: ***.pdf
I20180419-01:56:00.388(-4)? [FilesCollection] [Upload] [DDP] Got #1/1 chunks, dst: ***.pdf
I20180419-01:56:00.393(-4)? [FilesCollection] [Upload] [DDP] Got #-1/1 chunks, dst: ***.pdf
I20180419-01:56:00.394(-4)? [FilesCollection] [Upload] [finish(ing)Upload] -> resumes/nCQJtLq7RBX3GZHEj.pdf
I20180419-01:56:00.398(-4)? [FilesCollection] [Upload] [finish(ed)Upload] -> resumes/nCQJtLq7RBX3GZHEj.pdf
I20180419-01:56:00.428(-4)? [FilesCollection] [_preCollectionCursor.observe] [changed]: nCQJtLq7RBX3GZHEj
I20180419-01:56:00.432(-4)? [FilesCollection] [_preCollectionCursor.observe] [removed]: nCQJtLq7RBX3GZHEj
I20180419-01:56:00.637(-4)? [FilesCollection] [unlink(nCQJtLq7RBX3GZHEj, original)]
I20180419-01:56:04.202(-4)? [FilesCollection] [find({}, undefined)]
I20180419-01:56:06.938(-4)? [FilesCollection] [download(/cdn/storage/resumes/Rbosirjxbv3q9CyY9/original/Rbosirjxbv3q9CyY9.pdf, original)]
I20180419-01:56:06.939(-4)? [FilesCollection] [download(/cdn/storage/resumes/Rbosirjxbv3q9CyY9/original/Rbosirjxbv3q9CyY9.pdf)] [_404] File not found

[dr-dimitru]

Hello @vanshady ,

1. Can you go to S3 and observe uploaded file, to confirm it's moved to S3?
2. S3 requires bucket, IAM, and policy setup, described here, have followed this procedure?

[vanshady]

@dr-dimitru
Yes for both 1 and 2, I can download the file that I uploaded and I made sure that everything is configured as your tutorial

[vanshady]

For reference, here is my collection file

import { FilesCollection } from 'meteor/ostrio:files';
import { Meteor } from 'meteor/meteor';
import { _ } from 'meteor/underscore';
import { Random } from 'meteor/random';
import stream from 'stream';
import S3 from 'aws-sdk/clients/s3';
import fs from 'fs';

const s3Conf = Meteor.settings.s3 || {};
const bound = Meteor.bindEnvironment(callback => callback());
let Resumes;
// Check settings existence in `Meteor.settings`
// This is the best practice for app security
if (s3Conf && s3Conf.key && s3Conf.secret && s3Conf.bucket && s3Conf.region) {
    // Create a new S3 object
    const s3 = new S3({
        secretAccessKey: s3Conf.secret,
        accessKeyId: s3Conf.key,
        region: s3Conf.region,
        // sslEnabled: true, // optional
        httpOptions: {
            timeout: 6000,
            agent: false,
        },
    });

    // Declare the Meteor file collection on the Server
    Resumes = new FilesCollection({
        debug: true,
        storagePath: Meteor.settings.public.resumesStoragePath,
        collectionName: 'resumes',
        // Disallow Client to execute remove, use the Meteor.method
        allowClientCode: false,

        onBeforeUpload(file) {
            if (file.size <= 10485760 && /pdf|doc|docx|jpg|png|pages/i.test(file.extension)) {
                return true;
            } else {
                return 'Please upload PDF/DOC/DOCX resume, with size equal or less than 10MB';
            }
        },

        // Start moving files to AWS:S3
        // after fully received by the Meteor server
        onAfterUpload(fileRef) {
            // Run through each of the uploaded file
            _.each(fileRef.versions, (vRef, version) => {
                // We use Random.id() instead of real file's _id
                // to secure files from reverse engineering on the AWS client
                const filePath = Meteor.settings.public.current_event + '/' + (Random.id()) + '-' + version + '.' + fileRef.extension;

                // Create the AWS:S3 object.
                // Feel free to change the storage class from, see the documentation,
                // `STANDARD_IA` is the best deal for low access files.
                // Key is the file name we are creating on AWS:S3, so it will be like files/XXXXXXXXXXXXXXXXX-original.XXXX
                // Body is the file stream we are sending to AWS
                s3.putObject({
                    // ServerSideEncryption: 'AES256', // Optional
                    StorageClass: 'STANDARD',
                    Bucket: s3Conf.bucket,
                    Key: filePath,
                    Body: fs.createReadStream(vRef.path),
                    ContentType: vRef.type,
                }, (error) => {
                    bound(() => {
                        if (error) {
                            console.error(error);
                        } else {
                            // Update FilesCollection with link to the file at AWS
                            const upd = { $set: {} };
                            upd.$set['versions.' + version + '.meta.pipePath'] = filePath;

                            this.collection.update({
                                _id: fileRef._id,
                            }, upd, (updError) => {
                                if (updError) {
                                    console.error(updError);
                                } else {
                                    // Unlink original files from FS after successful upload to AWS:S3
                                    this.unlink(this.collection.findOne(fileRef._id), version);
                                }
                            });
                        }
                    });
                });
            });
        },

        // Intercept access to the file
        // And redirect request to AWS:S3
        interceptDownload(http, fileRef, version) {
            let path;

            if (fileRef && fileRef.versions && fileRef.versions[version] && fileRef.versions[version].meta && fileRef.versions[version].meta.pipePath) {
                path = fileRef.versions[version].meta.pipePath;
            }

            if (path) {
                // If file is successfully moved to AWS:S3
                // We will pipe request to AWS:S3
                // So, original link will stay always secure

                // To force ?play and ?download parameters
                // and to keep original file name, content-type,
                // content-disposition, chunked "streaming" and cache-control
                // we're using low-level .serve() method
                const opts = {
                    Bucket: s3Conf.bucket,
                    Key: path,
                };

                if (http.request.headers.range) {
                    const vRef = fileRef.versions[version];
                    const range = _.clone(http.request.headers.range);
                    const array = range.split(/bytes=([0-9]*)-([0-9]*)/);
                    const start = parseInt(array[1]);
                    let end = parseInt(array[2]);
                    if (isNaN(end)) {
                        // Request data from AWS:S3 by small chunks
                        end = (start + this.chunkSize) - 1;
                        if (end >= vRef.size) {
                            end = vRef.size - 1;
                        }
                    }
                    opts.Range = `bytes=${start}-${end}`;
                    http.request.headers.range = `bytes=${start}-${end}`;
                }

                const fileColl = this;
                s3.getObject(opts, function (error) {
                    if (error) {
                        console.error(error);
                        if (!http.response.finished) {
                            http.response.end();
                        }
                    } else {
                        if (http.request.headers.range && this.httpResponse.headers['content-range']) {
                            // Set proper range header in according to what is returned from AWS:S3
                            http.request.headers.range = this.httpResponse.headers['content-range'].split('/')[0].replace('bytes ', 'bytes=');
                        }

                        const dataStream = new stream.PassThrough();
                        fileColl.serve(http, fileRef, fileRef.versions[version], version, dataStream);
                        dataStream.end(this.data.Body);
                    }
                });

                return true;
            }
            // While file is not yet uploaded to AWS:S3
            // It will be served file from FS
            return false;
        },
    });

    // Intercept FilesCollection's remove method to remove file from AWS:S3
    const origRemove = Resumes.remove;
    Resumes.remove = function (search) {
        const cursor = this.collection.find(search);
        cursor.forEach((fileRef) => {
            _.each(fileRef.versions, (vRef) => {
                if (vRef && vRef.meta && vRef.meta.pipePath) {
                    // Remove the object from AWS:S3 first, then we will call the original FilesCollection remove
                    s3.deleteObject({
                        Bucket: s3Conf.bucket,
                        Key: vRef.meta.pipePath,
                    }, (error) => {
                        bound(() => {
                            if (error) {
                                console.error(error);
                            }
                        });
                    });
                }
            });
        });

        // remove original file from database
        origRemove.call(this, search);
    };
} else {
    // Declare the Meteor file collection on the Server
    Resumes = new FilesCollection({
        storagePath: Meteor.settings.public.resumesStoragePath,
        collectionName: 'resumes',
        // Disallow Client to execute remove, use the Meteor.method
        allowClientCode: false,

        onBeforeUpload(file) {
            if (file.size <= 10485760 && /pdf|doc|docx|jpg|png|pages/i.test(file.extension)) {
                return true;
            } else {
                return 'Please upload PDF/DOC/DOCX resume, with size equal or less than 10MB';
            }
        },
    });
}

if (Meteor.isClient) {
    Meteor.subscribe('resumes.all');
}

if (Meteor.isServer) {
    Meteor.publish('resumes.all', () => Resumes.find().cursor);
}

export default Resumes;

[dr-dimitru]

Hello @vanshady

1. Please post Client logs from browser's console
2. Post screenshot of "Network" Tab in DevTools, with opened "Headers" tab on the failed request

[vanshady]

Weirdly, I didn't change anything, and it suddenly works, maybe it's because of meteor cache? I'm closing the issue.

[dr-dimitru]

Weirdly, I didn't change anything, and it suddenly works

🎉👍👏

maybe it's because of meteor cache?

No, Meteor has only bundler cache. Your changes wasn't applied.

Have you ran meteor reset?

[vanshady]

No, I didn't. It's so weird

[dr-dimitru]

Okay... Happening sometimes with Meteor.

Please, support this project by:

• Star on GitHub
• Star on Atmosphere
• Check other packages we have made
• Share on Facebook and Twitter
• Donate via PayPal

[vanshady]

Just donated 😉Thank you for such a great project and keep up the good work.

[dr-dimitru]

@vanshady are you ok with appearing in our supporters list?

[vanshady]

definitely, my pleasure

[dr-dimitru]

Done on dev
@vanshady thank you for support, it means a lot for us. And thank you for choosing our lib.