Hammerspoon: Add hs.image:toByteArray()

[latenitefilms]

@asmagill - I've got the following NodeJS code that I want to try and replicate in Hammerspoon. Any ideas or suggestions? I'm thinking I might need to adding something like hs.image:asHex()?

var fs = require('fs');
var PNG = require('pngjs').PNG;

let ongoing2 = false;
let pixels2 = Buffer.alloc(240*240*3);

fs.createReadStream('test.png').pipe(
    new PNG({
        filterType: 4
    }
)).on('parsed', function() {

    for (let pixel = 0; pixel < 240*240; pixel++) {
        pixels2.writeUInt8(this.data[(pixel*4) + 0], (pixel*3) + 0 );
        pixels2.writeUInt8(this.data[(pixel*4) + 1], (pixel*3) + 1 );
        pixels2.writeUInt8(this.data[(pixel*4) + 2], (pixel*3) + 2 );
    }
    console.log(pixels2.toString('hex'))
});

[asmagill]

Check out hs.image:encodeAsURLString. IIRC, it converts the image to base64 with a specific suffix that allows the string to be imbedded directly into HTML.

Would parsing out the prefix, then using hs.base64 to convert it to a string of binary data be sufficient?

[latenitefilms]

That's a really clever idea - will try it out! Thanks heaps!

[latenitefilms]

Thanks for the suggestion @asmagill ! I played around, but wasn't able to make it work, as I really just need the pixel RGB values - whereas hs.image:encodeAsURLString will just give me the bytes of the encoded PNG (including metadata, etc).

After a bit of trial and error, I came up with the following, which works, but is impossibly slow:

function rgb2col(r, g, b)
   return ((g >> 5) & 7) | (((r >> 3) & 31) << 3) | (((b >> 3) & 31) << 8)
end

function mod.middleScreenImage()
    -- IMAGE MUST BE 360x270:
    local img = image.imageFromPath("test.png")
    local size = img:size()

    local data = {}
    local drawStart = "ff10210041000000000168010e"
    local drawEnd = "050f210041"

    for y=size.h-1, 0, -1 do
        for x=size.w-1, 0, -1 do
            local imageRGB = img:colorAt({x=x, y=y})

            local r =  imageRGB and math.floor(imageRGB.red * 255) or 0
            local g =  imageRGB and math.floor(imageRGB.green * 255) or 0
            local b =  imageRGB and math.floor(imageRGB.blue * 255) or 0

            local rgb = string.format("%04x", rgb2col(r, g, b))
            table.insert(data, rgb)
        end
    end
    local dataString = table.concat(data)

    send(fromHex(drawStart .. dataString))
    send(fromHex(drawEnd))
end

Any ideas or suggestions on how I can improve the performance in Lua-land, before I attempt to fix it in Objective-C land?

[asmagill]

You may be right about needing to add a method to convert an nsimage into an array of rgb(a) bytes; otherwise you're spending a lot of time going back and forth between Lua and ObjC with an approach like the one above.

Looking at the implementation of hs.image:colorAt, it should be fairly easy to write a method which just loops through the image and builds your array. It'll be faster that above because all but the transfer of the array will be handled in Objective-C, and you could optimize further by allowing a rect as a parameter specifying the region of the image you care about.

However, because it's looping and converting one pixel at a time, there are probably better ways to do it. The first part of this looks promising, though it will likely be a couple of days before I have a chance to really spend much time on it.

[latenitefilms]

Legend, thanks for the advice! I'll have a play and see what I can come up with.

[latenitefilms]

After a lot of experimentation, we finally came up with this method, which works for our purposes:

/// hs.image:getLoupedeckArray() -> table
/// Method
/// Translates an `hs.image` object into an RGB array suitable for the Loupedeck CT device.
///
/// Parameters:
///  * None
///
/// Returns:
///  * A string containing the RGB data
static int getLoupedeckArray(lua_State* L) {
    LuaSkin *skin = [LuaSkin shared] ;
    [skin checkArgs:LS_TUSERDATA, USERDATA_TAG, LS_TBREAK] ;

    NSImage *image = [skin luaObjectAtIndex:1 toClass:"NSImage"] ;

    NSRect imageRect = NSMakeRect(0, 0, image.size.width, image.size.height);

    CGImageRef inImage = [image CGImageForProposedRect:&imageRect context:NULL hints:nil];

    // Get image width, height. We'll use the entire image.
    size_t pixelsWide = CGImageGetWidth(inImage);
    size_t pixelsHigh = CGImageGetHeight(inImage);

    // Declare the number of bytes per row. Each pixel in the bitmap in this
    // example is represented by 4 bytes; 8 bits each of red, green, blue, and
    // alpha.
    unsigned long bitmapBytesPerRow   = (pixelsWide * 4);
    unsigned long bitmapByteCount     = (bitmapBytesPerRow * pixelsHigh);

    // Use the generic RGB color space.
    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();

    if (colorSpace == NULL)
    {
        return luaL_error(L, "error allocating color space") ;
    }

    // Allocate memory for image data. This is the destination in memory
    // where any drawing to the bitmap context will be rendered.
    void *bitmapData = malloc( bitmapByteCount );
    if (bitmapData == NULL)
    {
        CGColorSpaceRelease( colorSpace );
        return luaL_error(L, "memory not allocated") ;
    }

    // Create the bitmap context. We want pre-multiplied ARGB, 8-bits
    // per component. Regardless of what the source image format is
    // (CMYK, Grayscale, and so on) it will be converted over to the format
    // specified here by CGBitmapContextCreate.
    CGContextRef context = CGBitmapContextCreate (bitmapData,
                                    pixelsWide,
                                    pixelsHigh,
                                    8,      // bits per component
                                    bitmapBytesPerRow,
                                    colorSpace,
                                    kCGImageAlphaPremultipliedFirst);

    // Make sure and release colorspace before returning
    CGColorSpaceRelease( colorSpace );

    if (context == NULL)
    {
        free (bitmapData);
        return luaL_error(L, "context not created") ;
    }

    size_t w = CGImageGetWidth(inImage);
    size_t h = CGImageGetHeight(inImage);
    CGRect rect = {{0,0},{w,h}};

    // Draw the image to the bitmap context. Once we draw, the memory
    // allocated for the context for rendering will then contain the
    // raw image data in the specified color space.
    CGContextDrawImage(context, rect, inImage);

    // Setup the output array which we'll eventually send to Lua-land:
    NSMutableData *output = [NSMutableData dataWithCapacity: w * h * 2];

    // Now we can get a pointer to the image data associated with the bitmap
    // context.
    unsigned char* data = CGBitmapContextGetData (context);
    if (data != NULL) {
        for (unsigned y = 0; y < h; y++)
        {
            for (unsigned x = 0; x < w; x++)
            {
                //offset locates the pixel in the data from x,y.
                //4 for 4 bytes of data per pixel, w is width of one row of data.
                int offset = 4*((w*round(y))+round(x));
                //int alpha =  data[offset];
                int red = data[offset+1];
                int green = data[offset+2];
                int blue = data[offset+3];

                uint16_t color = ((green >> 5) & 0x07) | (((red >> 3) & 0x1F) << 3) | (((blue >> 3) & 0x1F) << 8);

                uint8_t little = color & 0xFF;
                uint8_t big = (color >> 8 ) & 0xFF;
                [output appendBytes:&big length:1];
                [output appendBytes:&little length:1];
            }
        }
    }

    // Free image data memory for the context
    if (data) {
            free(data);
    }
    if (context) {
        CGContextRelease(context);
    }

    [skin pushNSObject:output withOptions:LS_NSLuaStringAsDataOnly];
    return 1;
}

It's currently very specific to our needs (i.e. sending RGB data to a Loupedeck CT via websockets), so not sure it's really relevant to the wider Hammerspoon community - until we finish hs.loupedeckct, so I haven't submitted a pull request.

@randomeizer is going to attempt to make the function a bit more generic if he gets a chance.

He's toying with the idea of having a generic function that we can pass a 'converter' function into, for example:

myImage:toByteArray(16, function(r,g,b,a) return rbg2col(r,g,b) end)

...where 16 is indicating each pixel will be 16 bits. The function takes the 8-bit rgba values and returns either a number or a string.

If you have any ideas or suggestions @asmagill, let us know.

Thanks again for all your help and support!

[randomeizer]

If you have a good example of an extension where Lua passes a function into Objective-C code?

[latenitefilms]

@asmagill - Whilst you're online, if you happen to have a good example/reference of how you can pass Lua functions back into Objective-C code, that would be awesome.

[asmagill]

How does this differ from setting a callback function?

If I'm understanding what you mean here, I'm assuming you're passing a lua function in as a parameter to a function/method that is written in objective-c, correct?

Basically, whcn checking parameters use LUA_TFUNCTION in the skin checkArgs method (this recognizes both actual functions and tables with a __call metamethod that allows the table to act like a function). Then, it depends upon whether you're going to use it in a blocking function that makes calls back to lua before it returns, or whether you're wanting to use the function in a callback or from something that will be running in another thread...

Within a blocking function:
~~~objc
// ... arg check, setup, etc. happened above here ...

// the following can be repeated in a for/while loop, just make sure to start with `lua_pushvalue` and end with popping your results
lua_pushvalue(L, <index of function on lua stack>) ; // copy function onto top of lua stack
// push arguments to be passed to lua function onto stack here
if (![skin protectedCallAndTraceback:argCount nresults:resultCount]) {
    // do what you want for an error in/from the lua function here
    // if not returning with luaL_error or similar, then make sure to do `lua_pop(L, 1) ;` to clear message from stack
} else {
    // do what you want with the results
    // make sure to pop them from the stack when done `lua_pop(L, resultCount) ;`
}

// .. rest of objc function

~~~

If you're going to be returning (non-blocking) while the callback gets invoked when necessary, then do something like this:

~~~objc
// ... arg check, setup, etc. happened above here ...

int fnRef = [skin pushRef:refTable] ; // push function into registry and store reference to it
// if returning a userdata, make sure fnRef is stored in one of it's properties; otherwise you'll
// need to store it somwhere that you can get it again when needed (if using a single global callback, then in a file static variable is good enough

// return to user
return 1 ;

}

// now, in the code where you want to use the function, you do something like this:

if (fnRef != LUA_NOREF) {
    // the following can be repeated in a for/while loop, just make sure to start with `pushLuaRef ` and end with popping your results
    [skin pushLuaRef:refTable ref:fnRef] ;
    // push arguments to be passed to lua function onto stack here
    if (![skin protectedCallAndTraceback:argCount nresults:resultCount]) {
        // do what you want for an error in/from the lua function here
       // if not returning with luaL_error or similar, then make sure to do `lua_pop(L, 1) ;` to clear message from stack
   } else {
       // do what you want with the results
       // make sure to pop them from the stack when done `lua_pop(L, resultCount) ;`
    }
}

// somewhere you'll need to release fnRef so it can be properly garbage collected; if stored in
// a userdata, then the __gc method of the userdata is the proper place; otherwise, you'll need
// to decide when/how to invoke the following:

fnRef = [skin luaUnref:refTable ref:fnRef] ;

~~~

[latenitefilms]

Legend, thanks heaps! Will give it a bash, and see what we can come up with.

[asmagill]

Oh, I should also note that if you’re going the nonblockong route and it may be invoked on another thread, wrap the lua callback stuff with a dispatch_(a)sync... you can find examples in the Hammerspoon extensions.

[randomeizer]

I think it would be a blocking call, since it will be doing a conversion of each pixel to add to the returnees array.

That said, not that I think about it, Lua is waiting for the call to the C toByteArray function to return. Will it block execution of the passed-in function until it returns?

Also, the intent of this idea is to put the more expensive array processing into objective-c rather than Lua. But the specific type of encoding of the pixel data seems to be unique to this particular piece of hardware, so building that into the shared API seems a bit wrong.

Do you have any alternate suggestions for this?

[cmsj]

I would suggest that doing the pixel conversion in a Lua function is going to be dramatically slower than doing it in C, if only because of the number of times you’re going to be jumping back and forth between the two.

[cmsj]

How about for now this exists in the loupedeck module and if we see a need for more generic versions later, we refactor it into hs.image?

[randomeizer]

Yeah, was wondering about that.

Main issue is Lua is slow at byte array manipulation. And we've been avoiding making the Loupedeck module Objective-C. Might have to be done though.

[asmagill]

@cmsj is likely right, but to answer your question @randomeizer, by "blocking" I mean that the lua interpreter is actively doing something (executing the objective-c function) rather than sitting idle waiting for us to tell it do something, either rom the console or a timer or other callback from notifications, delegates, hotkeys, etc.

The objective-c function is still executing within the lua state as "the function lua is executing at this instant", and like any lua function, it can call another lua function. We talk about them having separate states because there is overhead that is handled for us automatically when passing data back and forth, and it's easier to think of them as "separate", but essentially the objective-c function isn't acting outside of the lua environment -- it's what the lua environment is doing right now, instead of interpreting some lua function we've written or some other lua built-in function.

If there is truly nothing "running" in your Hammerspoon configuration (for example if you've only set up hotkeys but zero timers, or other modules that rely on callbacks from the OS) then until you hit enter in the console or press one of your defined hotkeys, the lua interpreter is actually idle -- not even garbage collection is occurring. Lua is only actually doing something when lua_call is invoked (if you dig into the source of LuaSkin's protectedCall... methods and lua's lua_pcall function, it all eventually reaches a lua_call and it is during this that the input from the console or the stuff Objective-C delegates have pushed onto the stack gets dealt with. And some other maintenance like garbage collection, etc. occurs as well.

Sorry this turned into a mini-lecture... it took me a while to really grasp this and understand how this all worked and what the implications were... the biggest one being that a mostly idle Hammerspoon (i.e. no timers, etc.) may not actually be the best at releasing memory or resources because garbage collection only occurs when lua is actually doing something rather than when it's not.

[asmagill]

As to a generic pixel-by-pixel function, I'd say, if what you need for Loupedeck is pretty static (i.e. doesn't change every time you need it, or if it does, only in terms of parameter values and not convoluted logic) then code it entirely in Objective-C -- you'll get the fastest results that way.

If we do want something more generic, perhaps for basic image manipulation or proof of concept things like this in the future, we can add it then.

I've also played around a little with writing a CIFilter module for manipulating images that might be worth resurrecting in a couple of months after we get a few things finished and move to v2 first.

(FWIW, the entirety of the CIFilter class is way beyond the scope of what we can likely support in Hammerspoon, but there are some interesting methods in the class for creating controls and user interface elements for us automatically based on the specific filters chosen that might make creating a few of the more common and useful ones easy enough to be worthwhile.)

[latenitefilms]

We've solved our specific Loupedeck CT issue, and there wouldn't be much demand for a generic pixel-by-pixel function, so I'm just going to close this for now.