Opencv4nodejs: Poor performance from MAT.getDataAsArray()

[Tom-Hudson]

When using MAT.getDataAsArray(), it takes around 1 second to execute on a 1280 x 720 image. I am running this on a pretty high powered machine and it feels like this shouldn't take this long.

My rationale behind needing the MAT as an array is that I need the raw RGB values for each pixel to be stored in a flattened array.

Mostly for fun, I used the MAT.at(row,col) method in the most performant way I could (within JavaScript) to see if I could do any better manually. This method was slower and took 1.5 seconds, but I was surprised it wasn't slower than that since MAT.getDataAsArray() is calling C++ code.

const imageData = new Array((image.rows * image.cols))

let index = 0

for (let row = 0; row < image.rows; row++) {
    for (let col = 0; col < image.cols; col++) {        
        imageData[index] = image.at(row, col)
    }
}

Is there anything that can be done to speed this up or could you add a MAT.getDataAsArrayAsync() method to at least free up the Node Event Loop (or both ideally :satisfied:)?

Side note: this is probably the best npm module I have come across. Keep up the good work! Your efforts are appreciated!

[justadudewhohacks]

Its expected that getDataAsArray takes that long, because every single pixel has to be converted to a javascript array. For such a big image I would rather use getData to retrieve the node buffer and convert it to a Uint8Array.

[Tom-Hudson]

@justadudewhohacks - thanks for the tip! Never thought to use the buffer data directly.

But this puzzles me a little so perhaps you can shed some light on it for me. based on your suggestion, I wrote the code below which I think is correct to get an array of RGB values in the imageData array. This code executes in roughly 10-12ms. I could easily change this to output the same array as MAT.getDataAsArray() but within 10-12ms vs 1000ms+. Why is MAT.getDataAsArray() not using this method?

const imageBuffer = image.getData();

const ui8 = new Uint8Array(imageBuffer);

const imageData = new Array((image.rows * image.cols))

let index = 0

for (let i = 0; i < ui8.length; i += 3) {
    imageData[index] = [ui8[i], ui8[i + 1], ui8[i + 2]];
    index++;
}

This may answer my question above but I can see subtle differences in the rgb values from both methods. Why would this be?

[justadudewhohacks]

getDataAsArray() was one of the first methods I ever implemented in this package, mainly for debugging reasons. If this could be reimplemented with a javascript wrapper using getData() as you did, that would be nice.

We would have to figure out, how to deal with different Mat types and pixel formats, however. For example I am not sure if the image data in the cv.Mat you retrieve with getData is changing it's layout for different pixel layouts.

For example, by default cv.Mats are usually stored in BGR format, when read from a VideoCapture or with cv.imread, but the data itself might actually have a RGB layout in memory, which might also be, why you see differences in both approaches (maybe the R and B channels are swapped). But one would have to investigate more into the inner workings of OpenCV to figure this out.

Furthermore depending on the mat type, you have to convert the data buffer differently, depending on, whether the values are chars, half ints, ints or floats and the number of channels. But for image data (cv.CV_8UC3) your approach should be correct I'd assume.

[rainabba]

@Tom-Hudson That approach just saved me more than 60% processing time across my image (applying my own LAB space processing logic) Thanks!

[rainabba]

Hmm. Need to turn imageData back into a mat so I'm trying the following, but getting a segmentation fault with the last line. I do something wrong?

function preProcess( mat ) {
    const matProc = mat.cvtColor(cv.COLOR_BGR2Lab), // Convert to Lab for processing
      ui8 = new Uint8Array( matProc.getData() ),
      img = new Array( matProc.rows * matProc.cols ),
      index = 0;
    for (let i = 0; i < ui8.length; i += 3) {
      let c = [ui8[i], ui8[i + 1], ui8[i + 2]], L = c[0];
      // Go do my thing with c (L, a, and b) then...
      img[index] = [L, c[1], c[2]];
      index++;
    }

    return new cv.Mat( imageData, cv.CV_8UC3 ).cvtColor( cv.COLOR_Lab2BGR ) //Convert back to BGR
}

[justadudewhohacks]

What does the shape of imageData look like? Check out this, which should give you an intuition, how to create a Mat from a JS array.

[Tom-Hudson]

@rainabba imageData is a flat array of RGB values in my code [[r,g,b],[r,g,b]].

I think you will need to modify it to output the values in their respective columns and rows [[[r,g,b],[r,g,b]], [[r,g,b],[r,g,b]]].

Hope this helps!

[rainabba]

Yesterdays post wasn't good after all. Here is a working version. In my case, processing time for a 4k image goes from ~5600ms to ~3200ms so it's not a small improvement :)

I feel like this needs to at least be in the README, if not as this function, but I'm not sure where it's appropriate so I haven't done a PR, though I'm willing.

function copyMatByPixelc3( mat ) {
  let ui8 = new Uint8Array( mat.getData() ),
    img = createArray( mat.rows, mat.cols ), index = 1;

  for (let i = 1; i <= ui8.length; i += 3 ) {
    let c = [ ui8[i-1], ui8[i], ui8[i + 1] ];

    // Process Pixel here

    img[ Math.floor( ( index - 1 ) / mat.cols ) ][ ( index - 1 ) % mat.cols ] = [ c[0], c[1], c[2] ];
    index++;
  }

  return new cv.Mat(img, cv.CV_8UC3);
}

[Tom-Hudson]

@rainabba, if you're looking to improve performance further, I would suggest you get some benchmarks on the createArray function you have created. Recursion is great, and your function is clever but using a for loop will be significantly faster to create the array. I would say that's the only part of your code that could be further optimised without doing a C++ module for this. Do a bit of experimenting, and make sure to post back here. Your journey has been an interesting one to follow.

[rainabba]

@Tom-Hudson You were dead-on and I feel silly now that I've actually looked more at the array structure (instead of just copy/pasting grin).

The old function was taking ~95ms, my new one needs <2ms consistently I've been able to simplify it down to new Array( mat.cols ).fill( new Array( mat.rows ) )

Going to see about speeding up this for loop with Array.map() now. Something tells me there may be one more gain to be had with Matrix math, but that's a barely-educated guess from ~20 years of working around it in 3D graphics, but never really learning it.

[Tom-Hudson]

@rainabba that's a pretty decent gain if this is working on large batches of images. It's also a lot more readable than the old method :smile:

I am nowhere near my laptop, and won't be till Monday, and I may be completely wrong but I think you don't have to pass mat.getData() into a Uint8Array constructor (I know I put that in my code :stuck_out_tongue_closed_eyes:), as a buffer is already a subclass of an Uint8Array. So I think you could do const imageData = mat.getData(), and work with that in your loop. This might shave a few more milliseconds off the execution time.

I am fairly sure Array.map will be slower than a for loop, but you're a clever guy and clearly know what you're doing, so give it a go and post back here! I am looking forward to seeing what you come up with.

[Tom-Hudson]

@rainabba - I have just noticed you have made a mistake:

new Array( mat.cols ).fill( new Array( mat.rows ) )

should be

new Array(mat.rows).fill(new Array(mat.cols))

This will have been making an array larger than you need...

[rainabba]

@Tom-Hudson Thanks for the help along here. I'm stumped. Somewhere along 25 commits, I found my approach wasn't returning what I expected and I finally just figured out the behavior, but not the cause. Following is my current generic method (this represents some 40 hours of work embarrassingly) and no matter how I look at it, the logic appears solid and I get a valid mat, but somehow it seems that every row has the exact same value as just the last. I created a sample image to confirm. Would love to know where I've gone wrong and if you see any more optimization opportunities here. :)

function parsePixelsLabc3( mat ) {
  let ui8 = mat.cvtColor(cv.COLOR_BGR2Lab).getData(),
    img = new Array( mat.rows ).fill( new Array( mat.cols ).fill( 0 ) );

  for (let i = 1; i <= ui8.length; i += 3 ) {
    let L = ui8[i-1], // subpixel 0
      a = ui8[i], // subpixel 1
      b = ui8[i + 1] // subpixel 2
      oP = Math.floor( (i-1) / 3 );

    // Do processing of L, a, b here

    img[ Math.floor( oP / mat.cols ) ][ oP % mat.cols ] = [ L, a, b ];
  }

  return new cv.Mat(img, cv.CV_8UC3).cvtColor(cv.COLOR_Lab2BGR);  // Result appears to repeat just the last row, but back up to the top
}

[rainabba]

Here are the sample input/outputs that confirm the behavior above.

[Tom-Hudson]

@rainabba okay, I decided to stop just dishing advice and put my money where my mouth is.

The issue you are having (which I didn't realise until running the code), is that img = new Array( mat.rows ).fill( new Array( mat.cols ).fill( 0 ) ) is filling all rows with a reference to a single array instance 😄 . So you would find that all rows were filled out within the first iteration of the columns and then updated with each pass after that.

I have tried a few things to enhance performance as much as possible and found that creating the arrays for the rows and pushing the column arrays was faster than setting up the column arrays and re-assigning. This also removed the need for more code to identify the column. The code below ran in 400ms on a 1280 x 720 image. Let me know how it does for you!

function transformMat(mat) {
    const imageData = mat.getData(),
        transformedImageData = Array.from(new Array(mat.rows), () => []);

    for (let i = 1; i <= imageData.length; i += 3) {
        const row = Math.floor((i / 3) / mat.cols);

        // Do stuff with the values

        transformedImageData[row].push([imageData[i - 1], imageData[i], imageData[i + 1]]);
    }

    return new cv.Mat(transformedImageData, cv.CV_8UC3);
}

Any additional optimisations will be minor. I even tried doing everything inside 2 Array.from methods to create the arrays and populate them within the same loop but it was on average 70-80ms slower.

Don't worry about all the messages and declaring victory prematurely, we all do it at some point! hopefully you can get on with writing the rest of your app now 😃

[rainabba]

With that, I finally broke through. Thank you. I think the expected array structure here just wasn't making sense to me (I pictured an array of cols, each with an array of pixels for the row).

Here's my final function (pass in your pixelProcessor). Thanks @Tom-Hudson for the help!

function parsePixelsLabc3(mat, pxlFnc) {
  const imageData = mat.cvtColor(cv.COLOR_BGR2Lab).getData(),
    newMatData = Array.from(new Array(mat.rows), () => []);

  for (let i = 1; i <= imageData.length; i += 3) {
    newMatData[Math.floor((i / 3) / mat.cols)].push( pxlFnc([imageData[i - 1], imageData[i], imageData[i + 1]]));
  }

  return new cv.Mat(newMatData, cv.CV_8UC3).cvtColor(cv.COLOR_Lab2BGR);
}

[rainabba]

Here's another useful snippet based on our explorations:

img = new Array( mat.rows ).fill( new Array( mat.cols ).fill( 0 ) );
can now be
img = matrix( mat.rows, mat.cols, 0 )

function matrix( rows, cols, defaultValue){
  return Array.from(new Array(rows), () => Array.from(new Array(cols), () => defaultValue ));
}

[Tom-Hudson]

@rainabba although it's useful, it won't be very performant I'm afraid. Array.from iterates. So for a 1920 x 1080 image, it will iterate 2,073,600 times in total to create this matrix. I'm not sure how to express this in big o notation, but it wouldn't be good.

Although your efforts to make generic functions are good (and totally valid), within the context of image analysis/manipulation, performance is usually an important factor, and as I found during optimisation, this method was slower than pushing column arrays into the row arrays from the raw data loop.

[rainabba]

@Tom-Hudson Fair enough. Don't lose hope in me though. :) I was using that create a 5x5 for .dilate (needed something easy to tweak as I tested) and the code here pixels on a Mat.

[Tom-Hudson]

@rainabba no hope lost at all my friend! :+1:

I learnt a lot helping you. Mostly that Array.fill doesn't quite work as I had anticipated when giving it a new array :rolling_on_the_floor_laughing:

[rainabba]

@Tom-Hudson It's interesting trying to break my OOP/reuse patterns to go for max performance :) Things are coming together amazingly thanks largely to yourself and @justadudewhohacks

[Tom-Hudson]

@rainabba I was looking through the documentation of opencv4nodejs and found a line of code which triggered me to look back at this issue, and my statement in relation to further performance enhancements to the solution I provided:

Any additional optimisations will be minor

I was wrong...

function newParsePixelsLabc3(mat, pxlFunc) {
    const imageData = mat.cvtColor(cv.COLOR_BGR2Lab).getData();

    for (let i = 1; i <= imageData.length; i += 3) {
        // do whatever you need to with the values and simply mutate the buffer instance :)
        const [r, g, b] = pxlFunc(imageData[i - 1], imageData[i], imageData[i + 1]);

        imageData[i - 1] = r;
        imageData[i] = g;
        imageData[i + 1] = b;
    }

    return new cv.Mat(imageData, mat.rows, mat.cols, cv.CV_8UC3).cvtColor(cv.COLOR_Lab2BGR);
}

We were so fixated on creating that 'correct' array structure that we missed the fact you can directly create a mat from a Buffer instance (as shown in the quick guide - right below how to create a mat from an array 🤣).

Now, the best bit! If I ran the old function using the column/row structure on a 4k image, on my machine this took on average ~4000ms. The new function I have provided above does it in ~230ms. Yeah, seriously!!!

Think of this as a early Christmas present 🎁. I hope you find this useful 😃

UPDATE: Forgot to mention that I changed your pxlFunc to accept the values individually as arguments rather than in an array pxlFunc(imageData[i - 1], imageData[i], imageData[i + 1]). It cuts down on creating arrays this way since I then breakout the returned array into the r,g,b values.

[rainabba]

Santa!? I look forward to trying this first thing in the AM!

[Tom-Hudson]

Let me know how you get on @rainabba!

[rainabba]

@Tom-Hudson Sorry I haven't had feedback yet. Life is hitting me hard the last couple weeks and I'm playing catch up. With some luck, I get to back to this project today.

[rainabba]

@Tom-Hudson Finally got back to this project and implemented your suggestion. I went form ~6.5sec times for the full job (of which this is the biggest part as these numbers indicate, but not the whole), down to ~3.5sec.

Nice find and thank you!

[Tom-Hudson]

@rainabba you took your time buddy :smile:.

Glad it's been useful. I do love an optimisation challenge!