Depthai: Arbitrary crop/rescale/reformat and ROI return

[Luxonis-Brandon]

Start with the why:

Often in CV/AI pipelining there is some ROI (region of interest), or series of ROIs, which should be used in the pipeline - whether set dynamically based on results in the pipeline - or set statically based on some a-priori or tacitly-understood application information.

So being able to specify via the DepthAI API (USB, SPI (https://github.com/luxonis/depthai/issues/140), or microPython (https://github.com/luxonis/depthai/issues/207)) for the results from some other node (e.g. the color camera output) to be cropped, rescaled and/or reformatted and for this ROI to be returned (internally, to a next stage, or to the host) is very valuable.

And example is ignoring the hood of a car when making a smart dash-cam (as YOLO will identify the hood as 'car', and well, that is already known).

Move to the how:

Implement support for at arbitrary crop/rescale/reformat Gen2 pipeline builder node (https://github.com/luxonis/depthai/issues/136) which can be controlled by DepthAI API via USB, SPI, or on-board microPython.

Move to the what:

Support arbitrary crop/rescale/reformat nodes as part of the Gen2 pipeline builder.

[neixlo]

Hi Luxonis-Team,

if I understand this feature right, it will enable the depthai device to crop a defined ROI and send this ROI with high resolution (uncompressed) to the host.

For example:
I configure a pipeline with Gen2 pipeline builder to 1. detect a ROI of the color stream, 2. crop this ROI based on the detection, 3. send this ROI with full resolution to the host at higher frame rates then the full resolution of the color stream.

Is this possible with Gen2 pipeline builder and the december release?
If so, can you allready estimate a release date?

Im highly interessed in this feature and of cause your tremendous efforts in depthai hard- and software.
Thanks for making this possible!

[Luxonis-Brandon]

Thanks @neixlo !

if I understand this feature right, it will enable the depthai device to crop a defined ROI and send this ROI with high resolution (uncompressed) to the host.

Yes, that's exactly right and exactly what we had envisioned this for.

For example:
I configure a pipeline with Gen2 pipeline builder to 1. detect a ROI of the color stream, 2. crop this ROI based on the detection, 3. send this ROI with full resolution to the host at higher frame rates then the full resolution of the color stream.

Yes, that's exactly right.

Is this possible with Gen2 pipeline builder and the december release?
If so, can you allready estimate a release date?

I think we _just_ finished this actually. But I need to check about when it will be integrated and usable in the Gen2 branch (it might be there already). @themarpe can you comment? And CC: @VanDavv

Here's the latest internally:

Currently it can do:

• [x] crops
• [x] resizing
• [x] thumbnailing.

For future features, we want to add:

• [ ] color space conversion
• [ ] grayscale
• [ ] fp16 normalization
• [ ] flip

Im highly interessed in this feature and of cause your tremendous efforts in depthai hard- and software.
Thanks for making this possible!

Thank you for the kind words! There have been products in my past where I think "Man, I sure am happy that someone made this." This is exactly what we are trying to build here. So hearing this is great, and we hope to make it even more valuable over time.

Thanks again,
Brandon

[Luxonis-Brandon]

This is now merged into gen2_develop (where we soak things before going into core gen2).

And here is an example in C++:
https://github.com/luxonis/depthai-core/blob/gen2_develop/example/main.cpp#L933:L821

And we'll be breaking these out to smaller examples on how to use these sorts of things (and then also integrating these examples into our auto-QA/CI system to test to make sure new builds/features don't cause regressions on core functionalities like this).

[Luxonis-Brandon]

And on the support for C++/Python, the API is mostly the same with just the syntax differing.

C++:
colorCam->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);
Python:
colorCam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)

And then for documenting example usage of core functionality like this, we're thinking something like below:

Just C++ / Python instead (or Python / C++ heh)

Thoughts?

Thanks,
Brandon

[Luxonis-Brandon]

Here is the Python example of how to use this:
https://github.com/luxonis/depthai-python/blob/gen2_develop/examples/14_color_camera_control.py

[neixlo]

Awesome!
I tried, it worked. Good example to see how the host can change the crop while runtime.

[Luxonis-Brandon]

Thanks for the update!