Sharp: to buffer and back again

[smremde]

I was looking for a way of applying a chroma key to the image (matching certain colour pixels and turning them transparent.) In fact I just want to turn black pixels transparent so fairly straight forward.

I can't see a way to do this currently - I could output to a raw buffer and do the operation, but then I couldn't turn that raw buffer back into a image (unless I have missed something).

Do you have any plans to add this support? Is it even possible and if so would you welcome a contribution?

[Nateowami]

I'm also stuck at the same place, though from the docs it looks like it's possible to get back to an image from a raw buffer. Unfortunately I can't figure out how it works. I'm sending output from sharp through jsfeat, but sharp doesn't define the format for the input and output (I think there are conventions for this, but I can't seem to get sharp to accept the data).

See the docs for the cunstructor. One thing it doesn't say, but I figured out from error messages and peaking into the source code, is how to specify the raw parameter on the options object. Here's the docs:

raw an Object containing width, height and channels when providing uncompressed data. See raw() for pixel ordering.

I assumed that the channels field was for supplying the channels (i.e. image data for each channel), but actually it's the number of channels (in the range 1 to 4).

Higher up, the docs for the constructor say:

input, if present, can be one of:

• Buffer containing JPEG, PNG, WebP, GIF, SVG, TIFF or raw pixel image data, or
• String containing the path to an JPEG, PNG, WebP, GIF, SVG or TIFF image file.

I think this means you can send in the raw image data as the input parameter. However, it's not accepting the format I get back from jsfeat (which is an array-like object with width*height elements, each from 0 to 255 (I think) (it's single-channel)).

I'm getting the raw image data from sharp with img.raw().toBuffer().then(...).
It appears to be array-like, with a length of widthheightchannels, which makes sense (one byte for each color for each pixel) and is how the HTML5 canvas API works (as I understand it at least).

So my questions are:

• What exactly do raw() and toBuffer() do? I see raw().toBuffer() in the docs, and I'm using it, but I don't fully understand the relationship between these two.
• What format is "raw"? Is it an array-like object with widthheightchannels elements, with each one in the range 0 to 255? Sharp is rejecting what I'm quite sure is just that (Unsupported input object at Sharp._createInputDescriptor (/home/l/git/nodeoverlay/node_modules/sharp/index.js:186:11).
• Given I have data described in the above point, how do I construct a sharp object?

[smremde]

I read the docs a bit more carefully. I think this is the intended use but I have not had time to test:

var width, height, channels;
image.metadata(function (metadata) {
   width = metadata.width;
   height = metadata.height;
   channels = metadata.channels;
}).raw().toBuffer().then(function (buffer) {
   // alter buffer

   // reconstruct 
   sharp(buffer, { width: width, height: height, channels: channels})
});

[Nateowami]

I've made a little progress from digging into the code a bit. Here's how the constructor works:
EDIT: This is not how it works; I was mistaken; this simply has the effect of passing in no image data.

// Where raw is the pixel data, as a flattened array (see my previous 
// comment describing this data structure)
raw.width = someWidth;
raw.height = someHeight;
raw.channels = numChannels;
sharp(null, {
  raw: raw;
});

It accepts this. However, afterwards doing .toFile('somefile.jpg', function(cb)), it never calls the callback. Promises never get fulfilled either.

Here's what is returned, as reported by JSON.stringify(): {"isFulfilled":false,"isRejected":false}.

And here's what console.log() prints:

Promise {
  _bitField: 0,
  _fulfillmentHandler0: undefined,
  _rejectionHandler0: undefined,
  _promise0: undefined,
  _receiver0: undefined }

I'm not very good with promises, but I thought Bluebird (the promise implementation used by sharp) was supposed to make sure nothing silently failed. So I'm stuck again and uncertain whether this relates to the previous issue I was facing. I did get sharp working with opening images, modifying them, and saving, but in this case I can't get it to work.

@smremde Can you get an image constructed the way I have? Does it successfully save?

EDIT: By the way, I interpreted the docs the same way you have, but It didn't accept it that way.

[smremde]

Just tested and this actually works fine for me:

sharp('/mnt/tank/temp/x.jpg').metadata(function(err, metadata) {
    width = metadata.width;
    height = metadata.height;
    channels = metadata.channels;
}).raw().toBuffer().then(function(buffer) {
    for (var i = 0; i < buffer.length; i += 3) 
        buffer[i] = 255;
    sharp(buffer, {
        raw: {
            width: width,
            height: height,
            channels: channels
        }
    }).toFile('/mnt/tank/temp/y.jpg', function(cb) {
        console.log('cb')
    });
}); 

[Nateowami]

Changing y to buffer your code works for me too. So now I guess I need to find the root of the problem. I still don't know what format sharp wants, so maybe I have it wrong still. I'd be nice if it threw an error though.

[smremde]

sharp(new Buffer([255, 0, 0]), {
    raw: {
        width: 1,
        height: 1,
        channels: 3
    }
}).toFile('/mnt/tank/temp/test.jpg', function(err) {
    console.log(err);
});

Works fine for me - it creates a 1x1 pixel red image. The format seems to be RGB

[Nateowami]

Yes, the color order is specified in the docs where it talks about the raw() function, and it's RGBA (A for alpha, when applicable). And it _does_ specify what the raw() format is, better than I remembered, but I'm still confused about it's relation to buffer().

I was wrong about how the constructor worked; apparently I was simply passing in no data at all, hence the behavior of the toFile() function.

I just tried converting my array-like object (which I get back from jsfeat) to a Buffer, but that fails. I need to look more at the data structure of jsfeat; I thought it was identical to the raw output that sharp produces.

Anyway, I hope I didn't lead you down the wrong path. :frowning: And thanks for helping figure out how the API works. :smile:

[smremde]

Anyway, I hope I didn't lead you down the wrong path. :frowning: And thanks for helping figure out how the API works. :smile:

Far from it - you helped me solve me problem!

[Nateowami]

Just for anyone who stumbles across this and has the same problem integrating with jsfeat, here's what worked for me:

// img is a jsfeat matrix_t
sharp(new Buffer(img.buffer.u8), {
  raw: {
    width: width,
    height: height,
    channels: 1
  }
})

Keep in mind that u8 means an unsigned 8-bit array. If you're working with more than one channel, this may differ.

I figured out the data structure by logging the data I was trying to pass to sharp, redirecting the process's output to a file, and then opening the file with nano (other text editors may become unresponsive trying to open 42MB of text). Then it was pretty easy to see the data structure. This same approach should help regardless of where you're getting the data you send to sharp.

[lovell]

Hello, in my absence it looks like you've both solved your respective problems. I should go on holiday more often :)

[adamwilbert]

Just tested and this actually works fine for me:

sharp('/mnt/tank/temp/x.jpg').metadata(function(err, metadata) {
    width = metadata.width;
    height = metadata.height;
    channels = metadata.channels;
}).raw().toBuffer().then(function(buffer) {
    for (var i = 0; i < buffer.length; i += 3) 
        buffer[i] = 255;
    sharp(buffer, {
        raw: {
            width: width,
            height: height,
            channels: channels
        }
    }).toFile('/mnt/tank/temp/y.jpg', function(cb) {
        console.log('cb')
    });
}); 

Why does this work? I'm not sure I understand what the manipulation to the buffer is for.

[marzious]

I would like to follow on this task as I can't seem to get current solution work for me. I have an sample.raw file, so I read them from filesystem and try to pass the buffer to sharp, let it convert to .jpg file.

fs.readFile("sample.raw", "utf8", function(err, buffer) {
sharp(buffer, {
raw: {
width: 7920,
height: 6004,
channels: 1
}
}).toFile('sample.png', function(cb) {
console.log(cb)
});
});

My cb is returning missing input file. Can anyone point to me how to properly read an .raw from file and convert it to other file format ?

[lovell]

@marzious Raw support refers to uncompressed pixel data without metadata and not "raw" camera images. Please open a new issue with a sample image if you're still having problems.