Yarp: realsense2 device doesn't work with D435i camera(two RGB)

[PeterBowman]

I'm trying to start the realsense2 device with the following configuration (realsense2.ini):

device       RGBDSensorWrapper
subdevice    realsense2
name         /depthCamera

[SETTINGS]
depthResolution (640 480)
rgbResolution   (640 480)
framerate       30
enableEmitter   true

[HW_DESCRIPTION]
clipPlanes (0.2 10.0)

The launch command is yarpdev --from realsense2.ini, it crashes with a [ERROR]realsense2Driver: Setting param rgbResolution failed... quitting.. See attached output file.

I was suspicious about this line:

[INFO]realsense2Driver: Device consists of 3 sensors

According to the following code excerpt, the device reports one depth sensor present and two RGB cameras, one of which (the second one, which is highly relevant) seems faulty since barely no options are present. The driver device loops over these sensors and assigns depth and color handles accordingly:

https://github.com/robotology/yarp/blob/32ba9f4d4bf7ec8d70fc586a148b491d4e51dab9/src/devices/realsense2/realsense2Driver.cpp#L515-L528

The RGB handle is linked with the last (=second) RGB camera. If I tweak this code so that to a pointer to the first camera is assigned to m_color_sensor (e.g. make the loop end at m_sensors.size() - 1), the device initializes correctly.

Configuration:

• Intel RealSense D435i
• OS: Ubuntu 16.04 (Xenial)
• yarp version: 3.1.1
• compiler: GCC 7

[PeterBowman]

BTW I noticed that, regardless of the depth resolution set in the .ini file, the resolution of the depth frames always matches the chosen RGB resolution.

[Nicogene]

Hi @PeterBowman

First of all I'd suggest you to change the name of the ini file, it is exactly the same name of the ini file used by yarpdev to load the plugin, it should be ok, but I would not risk to make them clash, rename it in something like realsense2Conf.ini

I was suspicious about this line:
[INFO]realsense2Driver: Device consists of 3 sensors

Here in the lab we have only D435 and D415, we don't have the D435i to test it. With the D435 the output is:

 Name                 : Intel RealSense D435
  Serial Number        : 829212070409
  Firmware Version     : 05.10.06.00
  Recommended Firmware Version : 05.10.03.00
  Physical Port        : /sys/devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.0/video4linux/video1
  Debug Op Code        : 15
  Advanced Mode        : NO
  Product Id           : 0B07
  Camera Locked        : N/A
  Usb Type Descriptor  : 3.2

[INFO]realsense2Driver: Device consists of 2 sensors 

The fix you propose probably make work the D435i but breaks the D435 and D415 since they have only 2 sensors. Maybe your second RGB camera is somehow broken.
BTW I need to test it on the camera D435i and D435 before proceeding fixing it, to be sure to not break other RS cameras.

BTW I noticed that, regardless of the depth resolution set in the .ini file, the resolution of the depth frames always matches the chosen RGB resolution.

This is due to the needAlignment parameter that by default is true (see http://www.yarp.it/classyarp_1_1dev_1_1realsense2Driver.html for the list of parameter). The device aligns the depth image over the RGB one, in this way the depth one is down/up-scaled to match the rgb one.

[PeterBowman]

The fix you propose (...)

It was only meant to be a quick test to prove that the sensor count in this model is causing trouble. I'll try to find out what's going on with this second RGB sensor.

This is due to the needAlignment parameter that by default is true (see http://www.yarp.it/classyarp_1_1dev_1_1realsense2Driver.html for the list of parameter). The device aligns the depth image over the RGB one, in this way the depth one is down/up-scaled to match the rgb one.

Thanks!

@Nicogene if you don't mind me adding a short off-topic: have you tried to launch the old OpenNI2 device with a RealSense2 camera? I'm interested in skeleton tracking, mainly, and I found that YARP 3 (this device was dropped in said release) doesn't provide a means to retrieve and track human skeletons (attempted standard interface at https://github.com/robotology/assistive-rehab/pull/2#issuecomment-377492395).

[Nicogene]

@Nicogene if you don't mind me adding a short off-topic: have you tried to launch the old OpenNI2 device with a RealSense2 camera

Do you mean use the D435(i) with the deprecated openni2 device ?

The depthcamera device still use openni2 if you need it.

I'm not sure that the realsense devices are openni compatible, I would say no, looking here it seems that there is not an official support for realsense, there is only this wrapper.

I never tried to launch a realsense device with depthcamera driver, but I don't think it will work.

But why do you need to run it with openni2 ? @pattacini and @vvasco are doing skeleton tracking using the realsense2 device driver.

I'm interested in skeleton tracking, mainly, and I found that YARP 3 doesn't provide a means to retrieve and track human skeletons

It doesn't, but I can discuss with @pattacini if we can standardize and integrate in YARP what they did in assistive-rehab. I'm ok with that

[pattacini]

Hi @PeterBowman

If you want to try out what we've been working on you're more than welcome. You could start off with our documentation: https://robotology.github.io/assistive-rehab/doc/mkdocs/site/index.html.

@Nicogene we haven't planned to devote resources to standardizing C++ classes to then put them back in YARP. We don't have time to go through these steps right now since we have to concentrate on our project objectives.

That said, we will carefully look at all possible contributions.

[Nicogene]

I've tried and openni2::DeviceOpen fails as expected:

[ERROR]depthCameraDriver: Couldn't open device,         DeviceOpen using default: no devices found

[PeterBowman]

HI, thank you for those pointers, the official OpenNI2 driver looks very promising. OpenPose-based skeleton tracking uses 2D frames. If I'm correct, you merge this information with depth data, which is something we can consider later on - our main focus is to explore the capabilities of the RealSense SDK with pure 3D information. The legacy YARP/OpenNI2 device seems like a ready-to-use solution with a simple and known API, therefore we'd try to explore it first.

[pattacini]

@PeterBowman out of sheer curiosity, is the following video showing the output of the SDK you mentioned?

https://www.youtube.com/watch?v=gMPtV4NXtUo

I've poked around and found out that Nuitrack makes use of the SDK.

[PeterBowman]

@pattacini Nuitrack is another option we are considering, thanks! In case there is no alternative to writing our own code, this wrapper might yield better results than the official RealSense SDK. So far, I have successfully built the OpenNI2 driver and can use it with the legacy YARP driver, but NiTE reports that it's unable to start user tracking (ref). I'll investigate that further.

Edit: must force same RGB resolution+framerate equal and depth resolution+framerate, e.g. --colorVideoMode 45 --depthVideoMode 10 (use --printVideoModes).

[pattacini]

Just wondering because from the video it looks like that the skeleton is available only when a correct segmentation in 3D is obtained, whereas with OpenPose the detection is always at hand. Perhaps, this condition is satisfied in your setting and you're happy with that.

Anyway, I don't want to pollute this thread with a discussion not relevant to solving the issue. If interested, we could keep talking in https://spectrum.chat/icub/technicalities.

[PeterBowman]

According to the following code excerpt, the device reports one depth sensor present and two RGB cameras

I forgot to clarify this in the PR. For future reference, the third sensor present in the D435i model is an IMU, mistakenly identified by the YARP device as an RGB camera.