Meteor-files: Thumbnails generation error

[programthis]

I'm having an issue:

• generating thumbnails
• Meteor-Files: Version 1.9.5
• Meteor Version: 1.6.0.1
• Appearing on Mac/Linux (not tested on windows)
• this issue is appearing server side

I am using the thumbnail generation example found here.

When I upload several images at once, most of the images create thumbnails. However, for a handful of the images, the thumbnails are not generated. I receive the error shown below:

[159.203.61.14][createThumbnails] [img.resize] { Error: Command failed: 
[159.203.61.14]    at ChildProcess.onExit (/bundle/bundle/programs/server/npm/node_modules/gm/lib/command.js:301:17)
[159.203.61.14]    at emitTwo (events.js:125:13)
[159.203.61.14]    at ChildProcess.emit (events.js:213:7)
[159.203.61.14]    at maybeClose (internal/child_process.js:927:16)
[159.203.61.14]    at Socket.stream.socket.on (internal/child_process.js:348:11)
[159.203.61.14]    at emitOne (events.js:115:13)
[159.203.61.14]    at Socket.emit (events.js:210:7)
[159.203.61.14]    at Pipe._handle.close [as _onclose] (net.js:545:12) code: null, signal: 'SIGKILL' }
[159.203.61.14]{ Error: Command failed: 
[159.203.61.14]    at ChildProcess.onExit (/bundle/bundle/programs/server/npm/node_modules/gm/lib/command.js:301:17)
[159.203.61.14]    at emitTwo (events.js:125:13)
[159.203.61.14]    at ChildProcess.emit (events.js:213:7)
[159.203.61.14]    at maybeClose (internal/child_process.js:927:16)
[159.203.61.14]    at Socket.stream.socket.on (internal/child_process.js:348:11)
[159.203.61.14]    at emitOne (events.js:115:13)
[159.203.61.14]    at Socket.emit (events.js:210:7)
[159.203.61.14]    at Pipe._handle.close [as _onclose] (net.js:545:12) code: null, signal: 'SIGKILL' }
[159.203.61.14][FilesCollection] [Upload] [DDP] Got #-1/2 chunks, dst: IMG_4746-1.jpg
[159.203.61.14][FilesCollection] [Upload] [finish(ed)Upload] -> /images/SNygdJC3G647dxLM7.jpg
[159.203.61.14][FilesCollection] [Upload] [finish(ing)Upload] -> /images/v59YFLdZXW8bp4y5f.jpg
[159.203.61.14][FilesCollection] [_preCollectionCursor.observe] [changed]: iSfykHMSdENSLeMGR
[159.203.61.14][FilesCollection] [Upload] [DDP] Got #-1/3 chunks, dst: IMG_4741-1.jpg
[159.203.61.14][FilesCollection] [Upload] [finish(ed)Upload] -> /images/v59YFLdZXW8bp4y5f.jpg
[159.203.61.14][FilesCollection] [_preCollectionCursor.observe] [changed]: i7ZC63i8GHEwuJpFc
[159.203.61.14][FilesCollection] [Upload] [finish(ing)Upload] -> /images/r7T7ZzYux2tjrZahW.jpg
[159.203.61.14][FilesCollection] [_preCollectionCursor.observe] [removed]: ppixaEvznRennF7GW
[159.203.61.14][FilesCollection] [Upload] [DDP] Got #-1/2 chunks, dst: IMG_4739-1.jpg
[159.203.61.14][FilesCollection] [Upload] [finish(ed)Upload] -> /images/r7T7ZzYux2tjrZahW.jpg
[159.203.61.14][FilesCollection] [Upload] [finish(ing)Upload] -> /images/McpEckrGhQ2gZZmEB.jpg
[159.203.61.14][FilesCollection] [_preCollectionCursor.observe] [removed]: L6ARAcbLqAbAtzDWk
[159.203.61.14][FilesCollection] [Upload] [DDP] Got #-1/2 chunks, dst: IMG_4754-1.jpg
[159.203.61.14][FilesCollection] [Upload] [finish(ed)Upload] -> /images/McpEckrGhQ2gZZmEB.jpg
[159.203.61.14][FilesCollection] [_preCollectionCursor.observe] [changed]: SNygdJC3G647dxLM7
[159.203.61.14][FilesCollection] [Upload] [finish(ing)Upload] -> /images/YDJx5P3KLGyyMRx95.jpg
[159.203.61.14][FilesCollection] [_preCollectionCursor.observe] [changed]: v59YFLdZXW8bp4y5f
[159.203.61.14][FilesCollection] [Upload] [DDP] Got #-1/2 chunks, dst: IMG_4752-1.jpg
[159.203.61.14][FilesCollection] [Upload] [finish(ed)Upload] -> /images/YDJx5P3KLGyyMRx95.jpg
[159.203.61.14][FilesCollection] [_preCollectionCursor.observe] [removed]: iSfykHMSdENSLeMGR
[159.203.61.14][FilesCollection] [Upload] [finish(ing)Upload] -> /images/tfWsoZkjw9d55v9TB.jpg
[159.203.61.14][FilesCollection] [_preCollectionCursor.observe] [removed]: i7ZC63i8GHEwuJpFc
[159.203.61.14][FilesCollection] [Upload] [DDP] Got #-1/2 chunks, dst: IMG_4751-1.jpg
[159.203.61.14][FilesCollection] [Upload] [finish(ed)Upload] -> /images/tfWsoZkjw9d55v9TB.jpg
[159.203.61.14][FilesCollection] [Upload] [finish(ing)Upload] -> /images/RE2c9g7YZhCiizZJn.jpg
[159.203.61.14][FilesCollection] [_preCollectionCursor.observe] [changed]: r7T7ZzYux2tjrZahW

[dr-dimitru]

Hello @programthis ,

Please update this thread following our issue template.

[programthis]

@dr-dimitru done.

[dr-dimitru]

Server and client logs with "debug" enabled, please...

[programthis]

@dr-dimitru Sorry, I've now updated with the debug enabled.

[dr-dimitru]

Hello @programthis , sorry for delay, very busy now...
I recommend to take a look on how we create thumbs at our demo app, should give you a clue where you may made a mistake

[programthis]

@dr-dimitru I believe I may have found where the error is coming from.

When I am subscribing to all the images, the thumbnail works fine but when I only subscribe to a subset of images, I end up missing thumbnails.

Here's my subscription that is causing this error:

Meteor.publish('galleryImages', function() {
        let imageArray = [];
        imageSets = ImageSets.find({}, {sort: {orderIndex: -1}});
        imageSets.forEach(function(imageSet) {
            imageArray.push(imageSet.coverImageId);
        });
        return Images.find({_id: {$in: imageArray}}).cursor;
});

Could something like this be affecting the image processing client side? I had assumed this thumbnail generation was purely a server side functionality.

[dr-dimitru]

@programthis

1. Thumbnails created on Server side only, (it's possible to create thumbs on a Client, but such use-case is not covered in our wiki);
2. Fetching data from collection is not interfere with creating thumbs, it's two separate unrelated tasks
3. Not sure how publish is related to creating thumbs, but at first you should check what data is in imageArray, could you console.log it?

[programthis]

@dr-dimitru 1 and 2- Ok that is what I thought.
3- I also didn't think publish was related to creating thumbs though subscribing to all images does seem to resolve this issue. imageArray is just a collection of image ids (Every image set has a cover image, so I am returning just the cover image ids instead of returning all the images--so these ids are unrelated to the images being uploaded).

[paulincai]

Hi @programthis I am having some files from my project with V1.8.3. I hope this can help you.

I tried to keep the folder structure so that you find easy which goes where.
Together the files follow a phone camera workflow. In the components it uses a Cordova camera, takes the shot, saves it to the state, then on press of button starts saving to Files with a thumbnail and does upload to S3. I am long due to put these together in a demo for the Wiki of this repo... In the ImageMagic I generate "polaroids" (and on the server I do the thumbnails).

Let me know if you want to go through these files together in case things are too cumbersome. As there is always place for improvement (in my case there is a lot), if you get to look at this and find it useful and find ways to improve, let us know please as a better version of it will try to make it to the wiki.

As mentioned this is V1.8.3. I am not aware whether there have been any breaking changes since.
Cordova camera - Ostri0Files - S3.zip

[dr-dimitru]

@paulincai v1.9.x major changes:

• __Deprecated__: globally available Meteor.Files
• Move events interface to eventemitter3 package
• Solve #500 removing 25ms delay before setting auth-token-cookie for protected downloads, thanks to @vbelolapotkov for pointing this out. Not sure if this will help for initial issue which stated First download attempt, as Meteor's login is async and happens only after DDP connection is established.
• Fix this.user in onBeforeRemove hook
• Deprecate (temporarily) throttle option, not sure if it was working from the beginning

[programthis]

@paulincai As far as I can tell, I do not see any major differences between what you are doing and what I am doing. Additionally, I do not necessarily think there is an error in the code as it does generate the thumbnails correctly in most circumstances. I am starting to think this may be related to cpu usage as I'm seeing that this fails in production while local development seems to be just fine. I will continue to investigate as to why subscribing to all images seems to resolve this though.

[dr-dimitru]

@programthis Error points to this place: /bundle/bundle/programs/server/npm/node_modules/gm/lib/command.js:301:17 could you cat and paste that file here?

[programthis]

@dr-dimitru

/**
 * Module dependencies.
 */

var spawn = require('cross-spawn');
var utils = require('./utils');
var debug = require('debug')('gm');
var series = require('array-series');
var PassThrough = require('stream').PassThrough;

/**
 * Error messaging.
 */

var noBufferConcat = 'gm v1.9.0+ required node v0.8+. Please update your version of node, downgrade gm < 1.9, or do not use `bufferStream`.';

/**
 * Extend proto
 */

module.exports = function (proto) {

  function args (prop) {
    return function args () {
      var len = arguments.length;
      var a = [];
      var i = 0;

      for (; i < len; ++i) {
        a.push(arguments[i]);
      }

      this[prop] = this[prop].concat(a);
      return this;
    }
  }

  function streamToUnemptyBuffer(stream, callback) {
    var done = false
    var buffers = []

    stream.on('data', function (data) {
      buffers.push(data)
    })

    stream.on('end', function () {
      var result, err;
      if (done)
        return

      done = true
      result = Buffer.concat(buffers)
      buffers = null
      if (result.length==0)
      {
          err = new Error("Stream yields empty buffer"); 
          callback(err, null);
      } else {
          callback(null, result);
      }
    })

    stream.on('error', function (err) {
      done = true
      buffers = null
      callback(err)
    })
  }

  proto.in = args('_in');
  proto.out = args('_out');

  proto._preprocessor = [];
  proto.preprocessor = args('_preprocessor');

  /**
   * Execute the command and write the image to the specified file name.
   *
   * @param {String} name
   * @param {Function} callback
   * @return {Object} gm
   */

  proto.write = function write (name, callback) {
    if (!callback) callback = name, name = null;

    if ("function" !== typeof callback) {
      throw new TypeError("gm().write() expects a callback function")
    }

    if (!name) {
      return callback(TypeError("gm().write() expects a filename when writing new files"));
    }

    this.outname = name;

    var self = this;
    this._preprocess(function (err) {
      if (err) return callback(err);
      self._spawn(self.args(), true, callback);
    });
  }

  /**
   * Execute the command and return stdin and stderr
   * ReadableStreams providing the image data.
   * If no callback is passed, a "through" stream will be returned,
   * and stdout will be piped through, otherwise the error will be passed.
   *
   * @param {String} format (optional)
   * @param {Function} callback (optional)
   * @return {Stream}
   */

  proto.stream = function stream (format, callback) {
    if (!callback && typeof format === 'function') {
      callback = format;
      format = null;
    }

    var throughStream;

    if ("function" !== typeof callback) {
      throughStream = new PassThrough();
      callback = function (err, stdout, stderr) {
        if (err) throughStream.emit('error', err);
        else stdout.pipe(throughStream);
      }
    }

    if (format) {
      format = format.split('.').pop();
      this.outname = format + ":-";
    }

    var self = this;
    this._preprocess(function (err) {
      if (err) return callback(err);
      return self._spawn(self.args(), false, callback);
    });

    return throughStream || this;
  }

  /**
   * Convenience function for `proto.stream`.
   * Simply returns the buffer instead of the stream.
   *
   * @param {String} format (optional)
   * @param {Function} callback
   * @return {null}
   */

  proto.toBuffer = function toBuffer (format, callback) {
    if (!callback) callback = format, format = null;

    if ("function" !== typeof callback) {
      throw new Error('gm().toBuffer() expects a callback.');
    }

    return this.stream(format, function (err, stdout) {
      if (err) return callback(err);

      streamToUnemptyBuffer(stdout, callback);
    })
  }

  /**
    * Run any preProcessor functions in series. Used by autoOrient.
    *
    * @param {Function} callback
    * @return {Object} gm
    */

  proto._preprocess = function _preprocess (callback) {
    series(this._preprocessor, this, callback);
  }

  /**
    * Execute the command, buffer input and output, return stdout and stderr buffers.
    *
    * @param {String} bin
    * @param {Array} args
    * @param {Function} callback
    * @return {Object} gm
    */

  proto._exec = function _exec (args, callback) {
    return this._spawn(args, true, callback);
  }

  /**
    * Execute the command with stdin, returning stdout and stderr streams or buffers.
    * @param {String} bin
    * @param {Array} args
    * @param {ReadableStream} stream
    * @param {Boolean} shouldBuffer
    * @param {Function} callback, signature (err, stdout, stderr) -> * 
    * @return {Object} gm
    * @TODO refactor this mess
    */

  proto._spawn = function _spawn (args, bufferOutput, callback) {
    var appPath = this._options.appPath || '';
    var bin = this._options.imageMagick
      ? appPath + args.shift()
      : appPath + 'gm'

    var cmd = bin + ' ' + args.map(utils.escape).join(' ')
      , self = this
      , proc, err
      , timeout = parseInt(this._options.timeout)
      , disposers = this._options.disposers
      , timeoutId;

    debug(cmd);
    //imageMagick does not support minify (https://github.com/aheckmann/gm/issues/385)
    if(args.indexOf("-minify") > -1 && this._options.imageMagick){
      err = new Error("imageMagick does not support minify, use -scale or -sample. Alternatively, use graphicsMagick");
      return cb(err);
    }
    try {
      proc = spawn(bin, args);
    } catch (e) {
      return cb(e);
    }
    proc.stdin.once('error', cb);

    proc.on('error', function(err){
      if (err.code === 'ENOENT') {
        cb(new Error('Could not execute GraphicsMagick/ImageMagick: '+cmd+" this most likely means the gm/convert binaries can't be found"));
      } else {
        cb(err);
      }
    });

    if (timeout) {
      timeoutId = setTimeout(function(){
        dispose('gm() resulted in a timeout.');
      }, timeout);
    }

    if (disposers) {
      disposers.forEach(function(disposer) {
        disposer.events.forEach(function(event) {
          disposer.emitter.on(event, dispose);
        });
      });
    }

    if (self.sourceBuffer) {
      proc.stdin.write(this.sourceBuffer);
      proc.stdin.end();
    } else if (self.sourceStream) {

      if (!self.sourceStream.readable) {
        err = new Error("gm().stream() or gm().write() with a non-readable stream.");
        return cb(err);
      }

      self.sourceStream.pipe(proc.stdin);

      // bufferStream
      // We convert the input source from a stream to a buffer.
      if (self.bufferStream && !this._buffering) {
        if (!Buffer.concat) {
          throw new Error(noBufferConcat);
        }

        // Incase there are multiple processes in parallel,
        // we only need one
        self._buffering = true;

        streamToUnemptyBuffer(self.sourceStream, function (err, buffer) {
          self.sourceBuffer = buffer;
          self.sourceStream = null; // The stream is now dead
        })
      }
    }

    // for _exec operations (identify() mostly), we also
    // need to buffer the output stream before returning
    if (bufferOutput) {
      var stdout = ''
        , stderr = ''
        , onOut
        , onErr
        , onExit

      proc.stdout.on('data', onOut = function (data) {
        stdout += data;
      });

      proc.stderr.on('data', onErr = function (data) {
        stderr += data;
      });

      proc.on('close', onExit = function (code, signal) {
        if (code !== 0 || signal !== null) {
          err = new Error('Command failed: ' + stderr);
          err.code = code;
          err.signal = signal;
        };
        cb(err, stdout, stderr, cmd);
        stdout = stderr = onOut = onErr = onExit = null;
      });
    } else {
      cb(null, proc.stdout, proc.stderr, cmd);
    }

    return self;

    function cb (err, stdout, stderr, cmd) {
      if (cb.called) return;
      if (timeoutId) clearTimeout(timeoutId);
      cb.called = 1;
      if (args[0] !== 'identify' && bin !== 'identify') {
    self._in = [];
    self._out = [];
      }
      callback.call(self, err, stdout, stderr, cmd);
    }

    function dispose (msg) {
      var message = msg ? msg : 'gm() was disposed';
      err = new Error(message);
      cb(err);
      if (proc.exitCode === null) {
        proc.stdin.pause();
        proc.kill();
      }
    }
  }

  /**
   * Returns arguments to be used in the command.
   *
   * @return {Array}
   */

  proto.args = function args () {
    var outname = this.outname || "-";
    if (this._outputFormat) outname = this._outputFormat + ':' + outname;

    return [].concat(
        this._subCommand
      , this._in
      , this.src()
      , this._out
      , outname
    ).filter(Boolean); // remove falsey
  }

  /**
   * Adds an img source formatter.
   *
   * `formatters` are passed an array of images which will be
   * used as 'input' images for the command. Useful for methods
   * like `.append()` where multiple source images may be used.
   *
   * @param {Function} formatter
   * @return {gm} this
   */

  proto.addSrcFormatter = function addSrcFormatter (formatter) {
    if ('function' != typeof formatter)
      throw new TypeError('sourceFormatter must be a function');
    this._sourceFormatters || (this._sourceFormatters = []);
    this._sourceFormatters.push(formatter);
    return this;
  }

  /**
   * Applies all _sourceFormatters
   *
   * @return {Array}
   */

  proto.src = function src () {
    var arr = [];
    for (var i = 0; i < this._sourceFormatters.length; ++i) {
      this._sourceFormatters[i].call(this, arr);
    }
    return arr;
  }

  /**
   * Image types.
   */

  var types = {
      'jpg': /\.jpe?g$/i
    , 'png' : /\.png$/i
    , 'gif' : /\.gif$/i
    , 'tiff': /\.tif?f$/i
    , 'bmp' : /(?:\.bmp|\.dib)$/i
    , 'webp': /\.webp$/i
  };

  types.jpeg = types.jpg;
  types.tif = types.tiff;
  types.dib = types.bmp;

  /**
   * Determine the type of source image.
   *
   * @param {String} type
   * @return {Boolean}
   * @example
   *   if (this.inputIs('png')) ...
   */

  proto.inputIs = function inputIs (type) {
    if (!type) return false;

    var rgx = types[type];
    if (!rgx) {
      if ('.' !== type[0]) type = '.' + type;
      rgx = new RegExp('\\' + type + '$', 'i');
    }

    return rgx.test(this.source);
  }

  /**
   * add disposer (like 'close' of http.IncomingMessage) in order to dispose gm() with any event
   *
   * @param {EventEmitter} emitter
   * @param {Array} events
   * @return {Object} gm
   * @example
   *   command.addDisposer(req, ['close', 'end', 'finish']);
   */

  proto.addDisposer = function addDisposer (emitter, events) {
    if (!this._options.disposers) {
      this._options.disposers = [];
    }
    this._options.disposers.push({
      emitter: emitter,
      events: events
    });
    return this;
  };
}

[dr-dimitru]

1. Check if machine has enough resources to perform image manipulation;
2. Check if ImageMagic or GraphicsMagic is installed on your Linux/Mac machine, try to run simple commands right via CLI;
3. Make sure there is no permission errors when node.js (meteor) is trying to read source image, and write new thumbs to FS.

[programthis]

@dr-dimitru I have a feeling it might be number one as the other two seem to be fine. I'll have to do some experimenting with a bigger CPU and see if it makes any difference.

[dr-dimitru]

@programthis needs to be mentioned, what I'm very familiar with IM/GM and I've never seen it hitting CPU to fail, but it's common to hit RAM to fail. I recommend to start with adding more RAM or even simpler a SWAP.

[programthis]

@dr-dimitru Thank you. I will up the RAM first in that case and see if that resolves the issue.

[dr-dimitru]

@programthis one more thing related to RAM usage, - set streams option to 1, this will avoid high RAM usage on server. When streams set to 1 incoming chunks will be written to FS immediately, so memory will be quickly released.

Also you may tune chunkSize option, try to set it to something small, like 1KB: 1024

[dr-dimitru]

Hello @programthis ,

Have you solved this one? If so, then how?

[mksh-su]

Hello @programthis ,

Is this issue still relevant? Shall we close it?

[dr-dimitru]

Closed due to silence at issue owner end.
Feel free to reopen it in case if the issue still persists on your end.