Zed-ros-wrapper: [BUG] Object Detection won't start when using multiple cameras

[jorgemiar]

Describe the bug
I'm trying to use object detection on two zed cameras connected to a Xavier NX board.

Ideally, I want for the object detection on both cameras to be disabled and then I can turn OD on one camera on when I want to, using the services available.

I have tried launching the nodes with object detection enabled from the start for both of them (by setting the yaml file to true) and also have tried starting the object detection using rosservices, yet object detection will not start.

It works fine when launching only one camera but with two there is no response, even if I try to enable object detection on a single camera using rosservice.

I do get an "[WARN] Elaboration takes longer..." message at times, probably cause the Xavier is struggling to process so much stuff. Could this prevent the OD from starting?

Device information:

• OS: Ubuntu
• OS version: 18.04
• CPU: Xavier NX
• GPU Nvidia: Xavier NX
• ZED SDK Version: latest
• ROS Wrapper version: latest

To Reproduce
Launch two cameras using zed multicam.

Expected behavior
* Starting Object Detection should appear in the terminal.

Additional context
My launch files look like this:

<?xml version="1.0"?>

<launch>

    <!-- CAMERA 1 -->
    <arg name="node_name_1"          default="zed_node_A" />
    <arg name="camera_model_1"       default="zed2" />    
    <arg name="camera_name_1"        default="zedA" />
    <arg name="serial_number_1"      default="20971998"  />

    <!-- CAMERA 2 -->
    <arg name="node_name_2"          default="zed_node_B" />
    <arg name="camera_model_2"       default="zed2" />
    <arg name="camera_name_2"        default="zedB" />
    <arg name="serial_number_2"      default="29867557"  />

    <group ns="$(arg camera_name_1)">
        <include file="$(find navvy_accs)/launch/zed_camera.launch"> <!-- Will need to update this -->
            <arg name="camera_name"         value="$(arg camera_name_1)" />
            <arg name="node_name"           value="$(arg node_name_1)" />
            <arg name="camera_model"        value="$(arg camera_model_1)" />
            <arg name="serial_number"        value="$(arg serial_number_1)" />
            <arg name="publish_tf"          value="false" /> <!-- Robot Localization will publish TF -->
            <arg name="camera_id"           value="0" />
        </include>
    </group>

    <group ns="$(arg camera_name_2)">
        <include file="$(find navvy_accs)/launch/zed_camera.launch"> <!-- Will need to update this -->
            <arg name="camera_name"         value="$(arg camera_name_2)" />
            <arg name="node_name"           value="$(arg node_name_2)" />
            <arg name="camera_model"        value="$(arg camera_model_2)" />
            <arg name="serial_number"        value="$(arg serial_number_2)" />
            <arg name="publish_tf"          value="false" /> <!-- Robot Localization will publish TF -->
            <arg name="camera_id"           value="1" />
        </include>
    </group>

</launch>

<launch>
    <!-- Camera Model and Name -->
    <arg name="camera_name" /> <!-- The name you want -->
    <arg name="camera_model" /> <!-- 'zed' or 'zedm' or 'zed2' -->
    <arg name="node_name" />
    <arg name="serial_number" />

    <!-- Load SVO file -->
    <arg name="svo_file"              default="" /><!-- <arg name="svo_file" default="path/to/svo/file.svo"> -->

    <!-- Remote stream -->
    <arg name="stream"                default="" /> <!-- <arg name="stream" default="<ip_address>:<port>"> -->

    <arg name="camera_id"             default="0" />
    <arg name="gpu_id"                default="-1" />

    <arg name="publish_tf"          />

    <!-- Launch node -->
    <node name="$(arg node_name)" pkg="zed_wrapper" type="zed_wrapper_node" output="screen" required="true"><!-- launch-prefix="valgrind" -->

    <!-- General camera parameters for node -->
        <rosparam file="$(find navvy_accs)/params/common.yaml" command="load" />
        <!-- ZED2 Thresholds and OD parameters -->
        <rosparam file="$(find navvy_accs)/params/$(arg camera_model).yaml" command="load" />

        <!-- Camera name -->
        <param name="general/camera_name"               value="$(arg camera_name)" />

        <!-- Serial Number -->
        <param name="general/serial_number"               value="$(arg serial_number)" />

        <!-- SVO file path - Are these used anywhere? -->
        <param name="general/svo_file"                  value="$(arg svo_file)" />

        <!-- Remote stream - Are these used anywhere?  -->
        <param name="general/stream"                    value="$(arg stream)" />

        <!-- Camera ID - A is passed thru as 0 and B as 1 -->
        <param name="general/zed_id"                    value="$(arg camera_id)" />

        <!-- GPU ID -->
        <param name="general/gpu_id"                    value="$(arg gpu_id)" />

        <!-- TF -->
        <param name="pos_tracking/publish_tf"             value="$(arg publish_tf)" />
        <param name="pos_tracking/publish_map_tf"             value="$(arg publish_tf)" />
    </node>

</launch>

and my yaml:

# params/zed2_yaml
# Parameters for Stereolabs ZED2 camera
---

general:
    camera_model:               'zed2'

depth:
    min_depth:                  0.2             # Min: 0.2, Max: 3.0 - Default 0.7 - Note: reducing this value wil require more computational power and GPU memory
    max_depth:                  20.0            # Max: 40.0

pos_tracking:
    imu_fusion:                 true            # enable/disable IMU fusion. When set to false, only the optical odometry will be used.

sensors:
    sensors_timestamp_sync:     false           # Synchronize Sensors messages timestamp with latest received frame

object_detection:
    od_enabled:                 true           # True to enable Object Detection [only ZED 2]
    confidence_threshold:       30              # Minimum value of the detection confidence of an object [0,100]
    object_tracking_enabled:    true            # Enable/disable the tracking of the detected objects
    people_detection:           true            # Enable/disable the detection of persons
    vehicle_detection:          false           # Enable/disable the detection of vehicles

[Myzhar]

Hi @jorgemia
the Xavier NX cannot optimize two Object Detection models at the same time, so be sure that they are ready to be executed before running two parallel instances of the OD module running the command $ ZED_Diagnostic -o.
In this way the models will be downloaded and optimized for the Jetson board.

My concerns about running two instances of OD on the same Xavier NX is about RAM, I'm not sure that it is enough to load two AI models at the same time. I'm going to test it and I will let you know.

[jorgemiar]

When I ran $ ZED_Diagnostic -o

I got:

Cannot find CUDNN Diagnostic : 35%
- Running ZED SDK Diagnostic : Warning
  warning:
    Object detection module corrupted. Please reinstall the ZED SDK and enable the Object Detection option.

Should I reinstall the ZED SDK then? Seems to work fine with one camera and there is an objects_1.3.model file in the usr/local/zed/resources folder.

My concerns about running two instances of OD on the same Xavier NX is about RAM, I'm not sure that it is enough to load two AI models at the same time. I'm going to test it and I will let you know.

I have two cameras but only want to run the model on one of them at a time, for example robot moving forward would only use the front camera object detection. If a robot then moves backwards, it stops the front object detection and starts the back camera object detection.

If I'm only running one model at a time, will the Xavier still have to load both models? I thought with the stop_object_detection service I would be able to save memory/processing power.

[Myzhar]

Is the Jetson connected to Internet?

[jorgemiar]

Yes

[jorgemiar]

@Myzhar I can try and reinstall the ZED SDK. What the best way to remove my current version? Or will a new install just overwrite the current one?

[Myzhar]

You can simply reinstall to overwrite the old version.

[jorgemiar]

I've reinstalled both SDK and ROS Wrapper.

Single camera still works fine and I can start/stop OD with ros services. Two cameras doesn't work. I also still get this when running $ ZED_Diagnostic -o

Cannot find CUDNN Diagnostic : 35%

• Running ZED SDK Diagnostic : Warning
  warning:
  Object detection module corrupted. Please reinstall the ZED SDK and enable the Object Detection option.

I tried lowering the quality to VGA, but doesn't seem to change much.

[Myzhar]

I tested running two OD modules running at the same time and I can confirm that the NX does not have enough power to handle both.
About the error with ZED_Diagnostic I signal it and we will investigate

[jorgemiar]

Ok thank you.

Did you try to start a single OD module from one camera, whilst having two cameras connected? Is that also not feasible?

Have you tried on a full sized Xavier by any chance?

(Side note, I remember reading somewhere all processing eg. depth and pointcloud computation for the ZED cameras is done on the computer (NX) as well, is that right or is some processing done onboard the camera? Maybe a future camera can do more processing on board to free it up on the main computer)

[Myzhar]

Yes, the processing is performed by the GPU on the host, the camera only provides a synchronized stereo stream and sensors data.
Maybe in the future new devices will perform more computation onboard.

[jorgemiar]

That would be amazing. The intel D435 and T265 I believe do the processing on board. With how great the stereolabs support, SDK and documentation is and how well it integrates with ROS and the Jetson devices, it would really be a dream if more computation was done on board, would tick all the boxes.

Please let me know if in the future you test with two cameras on the AGX Xavier or get at least one OD module working on the Xavier NX. Thanks for the help.

[jorgemiar]

I tried launching the cameras with separate launch files and somehow managed to get object detection working on both at the same time and controllable using the rosservices on both.

I'm not quite sure what changed in my launch files that would allow me to start object detection now (or that wasn't letting me start before). I will report back if I find out what the problem was and with the performance I'm experiencing (Don't know what the best way to measure this is? Where can I find the fps other than rviz?). Two cameras running and only one doing object detection seems to work quite well and I've also been able to run it on both at the same time.

I sometimes get the Elaboration takes longer (0.0684223 sec) than requested by the FPS rate (0.066666667 sec). Please consider to lower the 'frame_rate' setting or to reduce the power requirements reducing the resolutions. warnings, but they appear much more rarely (mainly when I try to run OD on both cameras). If I try to visualise the pointclouds and both OD streams in rviz then the Xavier NX complains of insufficient memory.

[jorgemiar]

When I launch the cameras using my zed_multicam custom launch file and then try and start object detection on one of the cameras withrosservice call zedA/zed_node_A/start_object_detection 30 True True False, for some reason, it starts Positional Tracking:

[ INFO] [1599141834.132734158]: *** Starting Positional Tracking ***
[ INFO] [1599141834.133083470]:  * Waiting for valid static transformations...

If launch the cameras individually with the stock files, it starts Object Detection as it should.

[Myzhar]

Positional tracking is started when the camera starts if some condition are verified.
Enabling object detection is one of them:
https://github.com/stereolabs/zed-ros-wrapper/blob/master/zed_nodelets/src/zed_nodelet/src/zed_wrapper_nodelet.cpp#L3019-L3026

[jorgemiar]

Ok, it seems my problem was being caused by something to do with URDF and I guess those conditions you mentioned @Myzhar.

I had removed all the urdf stuff from my custom zed launch files, as this is contained in my robot description package.

I was trying to run the zed cameras without loading this robot description and the robot state publisher so object detection wasn't working. I have now tried launching both cameras at the same time with my custom multicam launch file whilst the robot state publisher was already running and it finally works!!! 🎉🎉

The stock launch files already provide the camera position and urdf and load the robot state publisher etc so that's why it was working.

[Myzhar]

Yes, if you disable TF publishing you need that some other node publishes at least the odom TF, otherwise the positional tracking cannot properly work.

[jorgemiar]

Yes it seems the positional tracking was preventing the OD from starting, as from the messages it seems the pos tracking is loaded first:

[ INFO] [1599143844.105092808]: *** Starting Positional Tracking ***
[ INFO] [1599143844.105354216]:  * Waiting for valid static transformations...
[ INFO] [1599143844.105720391]: Static transform Sensor to Base [zed2_left_camera_frame -> base_link]
[ INFO] [1599143844.106007303]:  * Translation: {0.000,-0.060,0.000}
[ INFO] [1599143844.106237766]:  * Rotation: {0.000,-0.000,0.000}
[ INFO] [1599143844.106481062]: Static transform Sensor to Camera Center [zed2_left_camera_frame -> zed2_camera_center]
[ INFO] [1599143844.108832131]:  * Translation: {0.000,-0.060,0.000}
[ INFO] [1599143844.109693921]:  * Rotation: {0.000,-0.000,0.000}
[ INFO] [1599143844.110622336]: Static transform Camera Center to Base [zed2_camera_center -> base_link]
[ INFO] [1599143844.110934432]:  * Translation: {0.000,0.000,0.000}
[ INFO] [1599143844.111148095]:  * Rotation: {0.000,-0.000,0.000}
[ INFO] [1599143844.212214607]: Initial ZED left camera pose (ZED pos. tracking): 
[ INFO] [1599143844.212550159]:  * T: [0,0.06,0]
[ INFO] [1599143844.212795087]:  * Q: [0,0,0,1]
[ INFO] [1599143844.338360380]: *** Starting Object Detection ***
[ INFO] [1599143848.521075668]: Advertised on topic /zed2/zed_node/obj_det/objects
[ INFO] [1599143848.523201425]: Advertised on topic /zed2/zed_node/obj_det/object_markers
[ INFO] [1599143848.587916789]: Odometry aligned to last tracking pose

Thanks for the help. Is there anyway of checking the FPS I'm getting with OD?

[Myzhar]

I suggest you to test the feat_async_obj_det branch, it will be soon merged in the master branch.
The ZED Diagnostic information contains the real FPS of the OD module

[jorgemiar]

Ok thanks!