Meteor-files: Create native GridFS integration (or document recipes)

[hluz]

How can we specify the GridFS collection name (not the FilesCollection collectionName)? It defaults to fs. and I cant get it to accept new name.

[dr-dimitru]

Hello @hluz ,
The name of the collections in GridFS is out of our control, as it hard-coded in MongoDB's specs.

If you've followed our GridFS integration, where we're using gridfs-stream npm package you can see there is no place to change collection's name.

More about GridFS can be found at this article.

Feel free to close it in case if the issue is solved on your end.

[hluz]

GridFS is out of our control, as it hard-coded in MongoDB's specs

@dr-dimitru, not exactly... the fs. prefix is just a default. Multiple named GridFS collections are ok from Mongo's perspective. Have a read at https://github.com/aheckmann/gridfs-stream/issues/102

[dr-dimitru]

@hluz thank you for comment on this, I thought you want to change fs. part.

Anyways as we use gridfs-stream directly, you can use the, method described in the issue you've referenced. As I've initially said (and this is the primary concept of this package) - it's out of this package boundaries, we only provide docs and tutorials on how to extend package capabilities before/after a file is uploaded, feel free to use any other solution.

Feel free to close it in case if the issue is solved on your end.

[hluz]

@dr-dimitru Thanks for the comment. I now realise that the gridfs-stream package is not related to Meteor-Files... I somehow assumed it was. gridfs-stream appears to be "abandonware", with many unattended important PRs and no owner activity for a long time. Will try to look for another package provides similar functionality and expose the gridfs collection name properly. Or may have to do it myself :-(

[dr-dimitru]

@hluz no worries.

Actually we wish to fork gridfs-stream and keep it in up-to-date shape :) . We all were sad to discover it in a such badly maintained state.
Or maybe @aheckmann could give us a push rights to repository, or better transfer it to us (including NPM)? @aheckmann if you're reading this, take a look at our experience, for years we keep many packages in up-to-date state, taking care of every ticket and PR. Consider giving your child into good hands :)

[hluz]

@dr-dimitru That would be a good idea had not been for the fact that the node.js mongo driver has a GridFS API (GridFSBucket) with up-to-date support for node.js streams since v2.2 (http://mongodb.github.io/node-mongodb-native/2.2/api/GridFSBucket.html)

;-)

This means you can drop the gridfs-streams package and do something like this instead:

import { Meteor } from 'meteor/meteor';
import { FilesCollection } from 'meteor/ostrio:files';
import fs from 'fs';              // Required to read files initially uploaded via Meteor-Files
import { MongoInternals } from 'meteor/mongo';

let gridFSBucket;
if (Meteor.isServer) {
  gridFSBucket = new MongoInternals.NpmModule.GridFSBucket(
    MongoInternals.defaultRemoteCollectionDriver().mongo.db,
    { bucketName: 'attachments' }, // root name for GridFS collections - attachments.files, attachments.chunks
  );
}

export const FileAttachments = new FilesCollection({
  debug: false, // Change to `true` for debugging
  collectionName: 'attachments',
  allowClientCode: false, // Disallow remove files from Client
  onBeforeUpload(file) {
    if (file.size <= 10485760) { // Allow upload files under 10MB
      return true;
    }
    return 'Please upload file, with size equal or less than 10MB';
  },
  onAfterUpload: function (file) {
    // Move file to GridFS
    Object.keys(file.versions).forEach(versionName => {
      const metadata = {...file.meta, versionName, fileId: file._id};
      fs.createReadStream(file.versions[versionName].path)
        .pipe(gridFSBucket.openUploadStream(
          file.name,
          {
            contentType: file.type || 'binary/octet-stream',
            metadata,
          }
        ))
        .on('error', error => {
          console.log('Error on writing to GridFS ==> ', error)
        })
        .on('finish', Meteor.bindEnvironment(ver => {
          const property = `versions.${versionName}.meta.gridFsFileId`;
          this.collection.update(file._id, {
            $set: {[property]: new Mongo.ObjectID(ver._id.toString())}
          });
          this.unlink(this.collection.findOne(file._id), versionName); // Unlink files from FS
        }));
    });
  },
});

Or even better, maybe you can just provide a wrapper of these for the most common scenarios when using Meteor-Files...

[hluz]

@dr-dimitru, maybe you want to use this as a base. It works for me...

import { Meteor } from 'meteor/meteor';
import { FilesCollection } from 'meteor/ostrio:files';
import fs from 'fs';              // Required to read files initially uploaded via Meteor-Files
import { MongoInternals } from 'meteor/mongo';

let gridFSBucket;
let ObjID;

if (Meteor.isServer) {
  gridFSBucket = new MongoInternals.NpmModule.GridFSBucket(
    MongoInternals.defaultRemoteCollectionDriver().mongo.db,
    { bucketName: 'attachments' }, // override default 'fs' bucketName -> attachments.files, attachment.chunks
  );
  ObjID = MongoInternals.NpmModule.ObjectID;
}

export const FileAttachments = new FilesCollection({
  debug: false, // Change to `true` for debugging
  collectionName: 'attachments',
  allowClientCode: false, // Disallow remove files from Client
  onBeforeUpload(file) {
    if (file.size <= 10485760) { // Allow upload files under 10MB
      return true;
    }
    return 'Please upload file, with size equal or less than 10MB';
  },
  onAfterUpload: function (file) {
    // Move file to GridFS
    Object.keys(file.versions).forEach(versionName => {
      const metadata = {...file.meta, versionName, fileId: file._id};
      fs.createReadStream(file.versions[versionName].path)
        .pipe(gridFSBucket.openUploadStream(
          file.name,
          {
            contentType: file.type || 'binary/octet-stream',
            metadata,
          }
        ))
        .on('error', err => { throw err; })
        .on('finish', Meteor.bindEnvironment(ver => {
          const property = `versions.${versionName}.meta.gridFsFileId`;
          this.collection.update(file._id, {
            $set: {[property]: ver._id.toHexString(),
            }});
          this.unlink(this.collection.findOne(file._id), versionName); // Unlink files from FS
        }));
    });
  },
  interceptDownload(http, file, versionName) {
    const gridFsFileId = (file.versions[versionName].meta || {}).gridFsFileId;
    if (gridFsFileId) {
      const readStream = gridFSBucket.openDownloadStream(new ObjID(gridFsFileId));
      readStream.on('error', err => { throw err; });
      http.response.setHeader('Content-Disposition', `inline; filename="${file.name}"`);
      readStream.pipe(http.response);
    }
    return Boolean(gridFsFileId); // Serve file from either GridFS or FS if it wasn't uploaded yet
  },
  onAfterRemove(files) {
    files.forEach(file => {
      Object.keys(file.versions).forEach(versionName => {
        const gridFsFileId = (file.versions[versionName].meta || {}).gridFsFileId;
        if (gridFsFileId) {
          gridFSBucket.delete(new ObjID(gridFsFileId), err => { if (err) throw err; });
        }
      });
    });
  }
});

if (Meteor.isServer) {
  FileAttachments.deny({
    insert() {
      return true;
    },
    update() {
      return true;
    },
    remove() {
      return true;
    }
  });
}

[jankapunkt]

@dr-dimitru I am currently updating to the GridFSBucket, as @hluz proposes. If there are no bigger issues I would update the documentation accordingly.

[macrozone]

a small improvement over @hluz ' suggestion above:

// interceptDownload
 .on('error', err => { throw err; })

this is bad, because if it can't read the file, this will crash the server. Better handle it:

interceptDownload(http, file, versionName) {
      const { gridFsFileId } = file.versions[versionName].meta || {};
      if (gridFsFileId) {
        const readStream = gridFSBucket.openDownloadStream(new ObjID(gridFsFileId));
        readStream.on('data', (data) => {
          http.response.write(data);
        });

        readStream.on('end', () => {
          http.response.end('end');

        });
        readStream.on('error', () => {
          // not found probably
          // eslint-disable-next-line no-param-reassign
          http.response.statusCode = 404;
          http.response.end('not found');
        });
        http.response.setHeader('Cache-Control', this.cacheControl);
        http.response.setHeader('Content-Disposition', `inline; filename="${file.name}"`);
      }
      return Boolean(gridFsFileId); // Serve file from either GridFS or FS if it wasn't uploaded yet
    },

EDIT: i forget to set the cache control!

[jankapunkt]

@macrozone @hluz anyone interested in opening a small PR to update the documentation?

[hluz]

I will see what I can do, but will probably be only towards the end of June... If anyone else wants to jump in, feel free... ;-)

[jankapunkt]

For everyone on this issue thread @hluz @macrozone @dr-dimitru I have implemented the example of @hluz together with the comment of @macrozone in this example repo PR: https://github.com/VeliovGroup/files-gridfs-autoform-example/pull/1

I think we can leave the gridfs-stream example in the documentation and mark it as deprecated and add a link to the example repo to point to the latest example using GridFSBucket.

Any thoughts on that?

[dr-dimitru]

I think we can leave the gridfs-stream example in the documentation and mark it as deprecated and add a link to the example repo to point to the latest example using GridFSBucket.

Please, PR to the docs is more than welcome, do not forget to list it in toc.md. And I'll take care of wiki

[jankapunkt]

I finally managed to create a PR on this topic: https://github.com/VeliovGroup/Meteor-Files/pull/732

@macrozone @hluz

[kakadais]

Cool functions.
I can't see where the deprecate message from,
but I wish I could help if there it is-

[jankapunkt]

@kakadais I added the deprecation message into the gridfs integration wiki file but it is not officially deprecated but it can be considered as deprecated (or at least abandoned) since the package is not maintained since 2017

[kakadais]

@jankapunkt I see the stories, but so sad news definately ;(. I believe this could be one of the good alternative for a file storage system for small to mid size system so far. Is there any chance to revoke gridfs-stream package alive or use another grifs package maintained? I should benchmark and report more to show this availabilty for hesitating people to use this way.

[dr-dimitru]

Closing with #731, and update wiki:

• Up-to-date: https://github.com/VeliovGroup/Meteor-Files/wiki/GridFS-Bucket-Integration
• Legacy: https://github.com/VeliovGroup/Meteor-Files/wiki/GridFS-Integration

Thanks to @jankapunkt and @kakadais

Feel free to reopen it in case if the issue is still persists on your end.