Depthai: 2021 OpenCV Spatial AI Competition Feature Discovery and Requests

[Luxonis-Brandon]

Start with the why:

In the 2020 OpenCV Spatial AI Competition, we got a bunch of good feedback on features/etc. needed. And we made a central to track these (so that they're easier to note quickly and not lose) and then broke those out into individual issues as we started work on them. See https://github.com/luxonis/depthai/issues/183.

The feedback from that competition was very helpful, and we've now implemented most of those, and the remaining are very close. But we expect from this 2021 competition that we'll get another round of valuable feedback on the API, needed functionality/etc.

So, let's do that again for the 2021!

Move to the how:

We'll add feature requests/etc. in here as notes (in the what) and then link to specific issues as we get the bandwidth to attack these.

Move to the what:

To kick this off, gtech888AU in our Discord (join here, then see this post) provided a nice starter list of onboard CV-processing nodes that are desired on DepthAI:

I have written down the OpenCV function names below with features I think would be immensely useful running directly on the OAK:

• Threshold (performing binary and other types of common thresholding)
• BilateralFilter (for bluring images whilst maintaining their edges) Github issue: https://github.com/luxonis/depthai/issues/215
• Canny (for finding all edges in an image, not just corners). A note on this: It looks like we can run Canny onboard at about 185ms/frame for a 1280x720 grayscale image. Just an approximate from doing quick math on this. Actual implementation will vary.
• ConnectedComponentsWithStats (finding the centroid of objects and filtering objects by size eg number plate letters)

[gtech888AU]

Nice to haves: _(These seem to have dropped off the list from the original post - reposted for completeness)_

• createCLAHE (to help correct illumination issues from the camera images with less than ideal lighting conditions)
• bitwise_and (for performing odd shaped / non-rectangular masking on images)

The OpenCV online documentation shows the complete description and features of any of these above-listed OpenCV functions.

[Luxonis-Brandon]

Thanks! I meant to add those and forgot. I appreciate you circling back.

[gtech888AU]

The following two simple functions can act as a simple background subtractor and would be very useful to be implemented onboard for simple motion tracking and finding changes between two images.

For maximum flexibility the node may need to be able to buffer the value of the previous frame to use as image 1 and current frame as image 2, to detect changes between the frames.

absdiff(image1, image2, dst);

https://docs.opencv.org/3.4/d2/de8/group__core__array.html#ga6fef31bc8c4071cbc114a758a2b79c14

// pseudocode absdiff
Mat image1, image2, dst;
...
for each row : r
for each column : c
for each channel : ch
dst(r,c,ch) = abs(image1(r,c,ch) - image2(r,c,ch))

subtract(image1, image2, differenceImg1);

https://docs.opencv.org/3.4/d2/de8/group__core__array.html#gaa0f00d98b4b5edeaeb7b8333b2de353b

// pseudocode subtract
Mat image1, image2, dst;
...
for each row : r
for each column : c
for each channel : ch
dst(r,c,ch) = (image1(r,c,ch) - image2(r,c,ch))

[Luxonis-Brandon]

Thanks!