Depthai: UVC Output as a Pipeline Node

[Luxonis-Brandon]

Start with the why:

In some cases it may be desirable to feed UVC (USB Video Class, e.g. a webcam) output to have compatibility with existing software stacks/etc. To provide instant integration sort of compatibility, particularly if working with closed-source tools (or hard to modify tools) which already work with UVC-compatible inputs.

This becomes increasingly powerful when the UVC Output is a node in the pipeline. As instead of the whole video feed being output as a UVC video stream, any video stream in the pipeline can be output. Take for using an object detector to guide digital PTZ (https://github.com/luxonis/depthai/issues/135), and then outputting this pan/tilt/zoomed stream directly out UVC.

This way, the UVC output would appear to a computer as if someone is actually just moving a camera around.

Move to the how:

An initial way to support this is to keep XLINK as is (i.e. used for control and high-speed data sending back/forth), and use one of the total (3) high speed endpoints to be a UVC endpoint. This would allow 1 UVC output.

A longer-term solution may be to develop XLINK support over the USB control endpoint (working at the message level: host request -> device reply), which would then not consume a high bandwidth pipe (i.e. would not consume 1 of the dedicated endpoints) for control. Internal notes here.

In this case, instantiating 3x UVC outputs may be possible.

To start with though, it would be easiest to support 1x UVC output so likely this will be the approach.

Move to the what:

Implement a UVC output node in the Gen2 Pipeline Builder (https://github.com/luxonis/depthai/issues/136). This will allow any node which produces video to be output over USB Video Class to a USB host.

[Luxonis-Brandon]

We have this initially working now. Need to see if the code is externally usable now.

[dhruvsheth-ai]

Eager to test it out

[Luxonis-Brandon]

This should work on Linux, but may have issues on other OSes:
https://github.com/luxonis/depthai-python/blob/gen2_uvc/examples/19_uvc_video.py

python3 -m pip install depthai==0.0.2.1+e70becfe1c1908c6148f842006b9986ebab39cab --extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-snapshot-local

[dhruvsheth-ai]

Thanks! Will report how it goes

[ORippler]

Works for me with x64 chip under Ubuntu 18.04 and the above linked snapshot + python file. Tested it out with Zoom.

[dhruvsheth-ai]

edit: (Glitches). Works now

[cafemoloko]

@Luxonis-Brandon can we close this issue as it seems resolved?

[Luxonis-Brandon]

So let's leave it open for now as it's not formally integrated with DepthAI so far (I think). Actually, let's see what @alex-luxonis has to say on this. He knows better than me.

[mx1up]

I can confirm it works as expected with OBS studio. I will try it out for streaming. I commented out https://github.com/luxonis/depthai-python/blob/e70becfe1c1908c6148f842006b9986ebab39cab/examples/19_uvc_video.py#L15 in order to have autofocus.
this sample or code does not seem to be integrated yet in the main distribution

[mx1up]

@Luxonis-Brandon which would be the most efficient video format: YUV420, BGR3 or YV12? (or maybe it does not really matter for the device, but rather which format my video card can render fastest?)

[Luxonis-Brandon]

Thanks for the test and information here @mx1up !

On which is the most efficient, I'm not sure actually. I'm curious to know as well. And I don't know WRT if it would be host/video-card dependent either (but great question/point). @alex-luxonis is probably the most likely among us to know on both.

[alex-luxonis]

At the moment only the YUV420 (YU12) format is exposed. Some programs like guvcview show those additional formats, not sure if that's an issue with those programs, or somehow caused by the USB descriptor. It's listed properly by this command:
v4l2-ctl -d /dev/video0 --list-formats

ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture

    [0]: 'YU12' (Planar YUV 4:2:0)

It is not merged yet to mainline, as we don't have a clean way to expose the UVC interface only when the UVC node is added to the pipeline. With the depthai version here, the UVC device always shows up on host, but it operational only with the UVC node linked in the pipeline. (It's not harmful otherwise to the normal operation of XLink VSC interface, for pipelines not using UVC, but might be confusing.)

For a clean integration, we need these (WIP, I think @themarpe is looking at):

• pre-boot config, to send the Gen2 pipeline (or a part thereof) to device prior to it booting up the DepthAI FW;
• dynamically constructing the USB descriptor, based on the interfaces required by the nodes/config in the Gen2 pipeline.

[mx1up]

thanks for the explanation.
FYI: OBS studio does mention "(emulated)" after BGR3 and YV12. I did some simplistic performance measurements (both windowed and fullscreen on an intel chipset) using vmstat and it seems YUV420 and YV12 perform more or less the same while BGR3 is definitely slower.

for example:
BGR3 (emulated)

 us  sy  id  wa  st
  5   3  92   0   0
  5   3  92   0   0
  6   4  90   0   0
  6   2  92   0   0
  6   4  90   0   0
  6   3  91   0   0
  6   4  90   0   0

YUV420

 us  sy  id  wa  st
  2   3  94   0   0
  3   4  92   0   0
  2   5  93   0   0
  2   4  94   0   0
  3   4  93   0   0
  3   5  92   0   0
  3   4  93   0   0

YV12 (emulated)

 us  sy  id  wa  st
  2   3  94   0   0
  3   4  92   0   0
  2   5  93   0   0
  2   4  94   0   0
  3   4  93   0   0
  3   5  92   0   0
  3   4  93   0   0

[Luxonis-Brandon]

Oh that's neat! Thanks for the data-points here.

[TannerGilbert]

@Luxonis-Brandon is https://github.com/luxonis/depthai-python/blob/gen2_uvc/examples/19_uvc_video.py still the newest version available and is there already an implementation for the depth output?

[alex-luxonis]

Yes, that is still the latest for now. But soon we'll start to work again on it and integrate it in mainline.

About depth, it may be possible for now to link other streams to uvc.input, but the UVC descriptor is hardcoded to 1920x1080 NV12 (12 bits per pixel), so the host could reject the frames (uvcvideo in newer Linux kernels may do that, and configuring it with nodrop=1 as here could be a workaround, but will need custom frame decoding). We'll also properly configure the UVC descriptor to match with the frame format.

[TannerGilbert]

@alex-luxonis Thanks for the quick reply. I'm looking forward to the integration.

[AChangXD]

@alex-luxonis @Luxonis-Brandon What is the timeline for this working on Windows?

[Luxonis-Brandon]

So we haven't done anything more on it directly. But we've done a ton of bootloader work which is the crux of getting it to work properly.
So I don't know when the Windows setup will work better.
That said, if you wanted you could get a OAK-SoM-IoT and install it on an OAK-D and it can be pre-flashed with UVC pipeline so that then it will work with Windows now @AChangXD .
https://shop.luxonis.com/collections/iot

Or simpler you could get one of these and flash them with a UVC pipeline, and they'll boot up right away as UVC so will work fine with Windows as well.

Once the underpinnings are there (the total rework of the bootloader), I'm just not sure yet how long doing the refactoring on the UVC node will take to get Windows to be OK with it. The flashed-device route will definitely take care of it though, and will work now.

Thoughts?

Thanks,
Brandon