Depthai: OCR Support on DepthAI (and megaAI)

[Luxonis-Brandon]

Start with the why:

Text detection combined with optical character recognition (OCR) is extremely useful in all sorts of automation and perception applications, but is also extremely resource heavy. So being able to offload this to DepthAI would be a very valuable feature.

Move to the how:

The EAST text detector seems quite useful and has the capability return rotated boxes (example below) so it allows functioning where the camera's horizon is not oriented with the text.

This region of interest can then be warped back to a straight/horizontal text, which using Gen2 Pipeline Builder (https://github.com/luxonis/depthai/issues/136), can be fed into a second-stage neural network (or otherwise) that does the OCR on the horizontal text.

Move to the what:

Support an out-of-the-box capability to do text detection and OCR of the detected text.

[Luxonis-Brandon]

This looks of interest too:
https://docs.openvinotoolkit.org/latest/_demos_text_detection_demo_README.html

[MXGray]

Hi @Luxonis-Brandon,

Looks really interesting. :)
And I hope we can think of something to go around this issue:

QUOTE:
NOTE: On VPU devices (Intel® Movidius™ Neural Compute Stick, Intel® Neural Compute Stick 2, and Intel® Vision Accelerator Design with Intel® Movidius™ VPUs) this demo is not supported with any of the Model Downloader available topologies.
/QUOTE
SOURCE: https://docs.openvinotoolkit.org/latest/_demos_text_detection_demo_README.html

[Luxonis-Brandon]

Thanks. I noticed that after I posted it and meant to circle back. I think text-detection-0001 did say that it was VPU compatible. But I'm not finding details on it right now.

https://docs.openvinotoolkit.org/2018_R5/_docs_Retail_object_detection_text_pixel_link_mobilenet_v2_0001_tf_desc_text_detection_0001.html

[Luxonis-Brandon]

So @rt-luxonis was able to get the text detector working even though the documentation says it doesn't work. It seems to be pretty good, although it runs at about 1FPS.

[MXGray]

@Luxonis-Brandon

Cool! :)
Just a quick question though:

Can models that work in OpenCV DNN be used, just like how NCS1 and NCS2 are used with RPi for these OpenCV DNN models?
• My purpose in asking is that there are a few other pre-trained text detector models, like the EAST text detector that Adrian tackled in this blog post ...

Please advise. Thanks! :)

[Luxonis-Brandon]

Hi @MXGray ,

So most any network that works with the NCS2 will work on DepthAI. Only very large models might struggle (as on the NCS2 it only has to do neural processing, and on DepthAI we do all the other work of dealing w/ the image sensors/etc. so if the model uses all 512MB of RAM on the NCS2, it might not fit in DepthAI). Most models are not so big though so will work.

We will see if it's possible to convert that model using OpenVINO to run on the NCS2/DepthAI... starting probably by Googling to see if anyone has already converted this to run on the NCS2.

Thanks for pointing us to this!

Best,
Brandon

[Luxonis-Brandon]

We now have EAST text detection running on DepthAI:

[Luxonis-Brandon]

@alex-luxonis is working on the crop+warp necessary to take the EAST text detection output and feed into the OCR second stage network. CC: @rt-luxonis

[Luxonis-Brandon]

This is now implemented here: https://github.com/luxonis/depthai-experiments/tree/master/gen2-ocr