Meteor-files: File that was uploded on server-side using write method is corrupt. How can I fix issue?

[yushaUzumo]

1. ostrio:[email protected]
2. [email protected]
3. Fedora 26
4. Server

I am inserting a file into db.images, db.fs.files and db.fs.chunks using meteor-files or Ostrio-files (from VeliovGroup or keenethics). I am storing the files on GridFS. I installed gridfs-stream. I already have a meteor client app code that reads files from meteor-files and display them via a browser - this is how I view my newly inserted files, server-side. My server-side code for inserting files is as follows:

Images.write(this.bodyParams.file, {
  fileName: 'SignUpTermsAndConditions_' + this.bodyParams.name,
  type: 'application/vnd.oasis.opendocument.text',
  meta: {owner: this.userId,createdAt: new Date()}
}, function (error, fileRef) {
  if (error) {
    throw error;
  } else {
    console.log(fileRef.name + ' is successfully saved to FS. _id: ' + fileRef._id);
  }
},proceedAfterUpload=true);

I am able to insert a file client-side properly and when I view that file, the file is not corrupt.
My client-side code for inserting into db.images, db.fs.files and db.fs.chunks is as follows:

Images.insert({
    file: file,  // where var file = e.currentTarget.files[0];
    fileName: fileName + e.currentTarget.files[0].name,
    onStart: function () {
    },
    onUploaded: function (error, fileObj) {
      if (error) {
        alert('Error during upload: ' + error);
      } else {
      }
    },
    streams: 'dynamic',
    chunkSize: 'dynamic'
});

If you want me to provide the code that reads the file from GridFS client-side let me know.

As stated above my issue is that my server-side code for inserting a file, results in a corrupt file. Please help.

For your info: I am passing the file in a REST api post; I am passing the file in the body of the post.

For your info this is the error I get when trying to open the file in LibreOffice:

The file 'nSQGqasoSy6JF3DkJ-2.odt' is corrupt and therefore cannot be opened. LibreOffice can try to repair the file.

The corruption could be the result of document manipulation or of structural document damage due to data transmission.

We recommend that you do not trust the content of the repaired document.
Execution of macros is disabled for this document.

Should LibreOffice repair the file?

For your info I am reading the file in node.js using fs.readFile method. I have also tried reading the file locally within meteor.js instead but still results in a corrupt file. Here is my node.js code for fs.readFile and post

    fs.readFile('./Statement.odt', function read(err, data) { //where fs = require('fs')
        if (err) {
            throw err;
        }
        var file = data;

        request.post({
          uri: url, 
          headers:   {
            'X-User-Id': userId,
            'X-Auth-Token': authToken
          },
          form: {
              file: file,
              name:"Statement.odt"
          }
        }, function(err, httpResponse, body) {
          if (err) {
            return console.error('post failed:', err);
          }

          console.log('Get successful!  Server responded with:', body);
        });
    });

For your info: I have tried reading a pdf document instead of a LibreOffice document but also results in a corrupt file. Please help!

[dr-dimitru]

Hello @yushaUzumo ,

This issue looks very clear to me:

I am passing the file in a REST api post; I am passing the file in the body of the post.

Looks like incoming data isn't a Buffer, or a Buffer with wrong encoding. If you follow .write() API docs, first argument __must__ be a Buffer.
Could you please do console.log(this.bodyParams.file), right before this line (to make sure it is a Buffer)?:

Images.write(this.bodyParams.file, {/*...*/});

[yushaUzumo]

Thanks for your prompt reply. You are correct. I went over my testing/test fs.readFile on the server side where I am running meteor.js and fixed it by making sure there are no errors while reading the file; the file that is loaded on GridFS is not corrupt anymore. So yes my issue is with reading the file on the node.js client-side of things as per the code above. I followed this example in this answer on stackoverflow for my fs.readFile.If you do know how to fix my node.js code to read the buffer into memory and also send the buffer via post I'll appreciate your help. Otherwise if I don't see your reply/closing-of-issue in two days time I'll automatically close this issue. Thanks a ton for your help.

[dr-dimitru]

Hey @yushaUzumo ,

If you do know how to fix my node.js code to read the buffer into memory and also send the buffer via post I'll appreciate your help.

It depends from what type of data you have on incoming connection. Can you do console.log(this.bodyParams.file)?

[yushaUzumo]

Here is the log file:
files.txt

The uploaded file starts where there is PK near the top.

[dr-dimitru]

? @yushaUzumo attached file is binary encoded.
All I need to help you is console.log(this.bodyParams.file)

[yushaUzumo]

The backend logs are also binary encoded.The logs on the meteor.js backend are a bit messy. But as far as I can see the post itself has not modified the file further. Here are the first few bytes of the console.log(this.bodyParams.file):

PK\00\00\00\00\00U�L^�2'\00\00\00'\00\00\00\00\00\00mimetypeapplication/vnd.oasis.opendocument.textPK\00\00\00\00\00U�L'�V�\00\00\00\00\00\00\00Thumbnails/thumbnail.png�PNG

[dr-dimitru]

Ok, then give a code you're using to send a file via POST

[yushaUzumo]

Sorry I had the utf8 encoding set on fs.readFile. Now here is the log on the client:

The code is now:

    fs.readFile('./Statement.odt', function read(err, data) {
        if (err) {
            throw err;
        }
        console.log(data);
        var file = data;

        request.post({
          uri: url, 
          headers:   {
            'X-User-Id': userId,
            'X-Auth-Token': authToken
          },
          form: {
              file: file,
              name:"Statement.odt"
              //name:"Terms and conditions.pdf"
          }
        }, function(err, httpResponse, body) {
          if (err) {
            return console.error('post failed:', err);
          }

          console.log('Get successful!  Server responded with:', body);
        });
    });

console.log(this.bodyParams.file) is however still binary encoded not sure why. Here are the first few bytes:

PK\00\00\00\00\00U�L^�2
'\00\00\00'\00\00\00\00\00\00mimetypeapplication/vnd.oasis.opendocument.textPK\00\00\00\00\00U�L'�V�
\00\00
\00\00\00\00\00Thumbnails/thumbnail.png�PNG

[dr-dimitru]

@yushaUzumo try this code:

    fs.readFile('./Statement.odt', function read(err, data) {
        if (err) {
            throw err;
        }
        console.log(data);
        var file = data;

        request({
          uri: url,
          method: 'POST',
          headers:   {
            'X-User-Id': userId,
            'X-Auth-Token': authToken
          },
          body: data,
          encoding: null
        }, function(err, httpResponse, body) {
          if (err) {
            return console.error('post failed:', err);
          }

          console.log('Get successful!  Server responded with:', body);
        });
    });

[yushaUzumo]

Thanks I am busy with it. I'll let you know.

[yushaUzumo]

@dr-dimitru, thanks for the code. How do I access the body on the meteor server?

[yushaUzumo]

I am using restivus for the REST api and I have tried this.request.body and it gives me [object Object]

[dr-dimitru]

Use webapp

[yushaUzumo]

Can you make webapp api to force authentication? Can you please refer me to a thread that does this or give me an example. Can the authentication be linked with the meteor app authentication like restivus does?

I was using restivus because the word out there is that simple-REST which is forked from restivus will become the de facto REST package for meteor once it is complete.

[dr-dimitru]

@yushaUzumo sorry I didn't get how it is connected to OP and Meteor Files

[yushaUzumo]

Found the answer. I am posting the file in the body itself as per your advice and then setting json to true. My client side code changes as follows:

    url = 'http://localhost:3000/api/v1/images/';
    fs.readFile('./Statement.odt', function read(err, data) {
        if (err) {
            throw err;
        }
        console.log(data);
        //var file = new Buffer(data).toString('base64');
        var file = data;
        //var file = fs.readFileSync('./Statement.odt',{ encoding: 'base64' });

        var req = request.post({
          uri: url, 
          headers:   {
            'X-User-Id': userId,
            'X-Auth-Token': authToken
          },
          /*form: { //Post to the body instead of the form
              file: file,
              name:"Statement.odt",
          },*/
          body: file, //Post to the body instead of the form
          json:true  //Set json to true
          //encoding: null
        }, function(err, httpResponse, body) {
          if (err) {
            return console.error('post failed:', err);
          }

          console.log('Get successful!  Server responded with:', body);
        });
    });

And here is how I access the json body on the server-side:

var buf = JSON.stringify(this.request.body);
var buf1 = JSON.parse(buf).data
var bufferOriginal = Buffer.from(buf1)

Note that when you do console.log(bufferOriginal) you get an output of the file encoded in utf8 but when you reference/use bufferOriginal anywhere in the code it is recognized as a file buffer that looks somenhing like:

<Buffer 50 4b 03 04 14 00 00 08 00 00 00 55 f6 4c 5e c6 32 0c 27 00 00 00 27 00 00 00 08 00 00 00 6d 69 6d 65 74 79 70 65 61 70 70 6c 69 63 61 74 69 6f 6e 2f ... >

[dr-dimitru]

@yushaUzumo why first you do .stringify(), then .parse()? It should work this way too:

var bufferOriginal = Buffer.from(this.request.body.data);

Note that when you do console.log(bufferOriginal) you get an output of the file encoded in utf8 but when you reference/use bufferOriginal anywhere in the code it is recognized as a file buffer

All correct, this is the way it should be, as it comes in form of the text, and using Buffer.from() you're actually transforming it to Buffer.

[yushaUzumo]

You are correct. Silly of me.

[yushaUzumo]

And thanks for nudging me in the right direction.