Zed-ros-wrapper: Desync between depth and rgb even though timestamps show exact match

[dennisVi]

Although a recent update of the zed-ros-wrapper repository has significantly reduced the effect, I'm still observing a de-synchronization between the depth and rgb images when employing timestamp synchronization.

What do I see:
The depth image and the rgb image do not align properly (in time). This can be observed while panning or moving the zed2 camera, which causes a clear misalignment between the synchronized rgb image and the depth map, while both have the exact same timestamp.
When storing the overlay of the depth on top of the rgb image, you can clearly see it lagging behind when the camera moves.

What do I expect:
the depth map is directly constructed from the (rectified) left rgb image. Hence, I would assume that the timestamp of the depth map is set to match the timestamp of the left image, hence I would expect the synchronized left rgb image and the depth map to properly align.

How to reproduce:
There are two ways to reproduce this behavior and I've added a minimal working example for both.
1: Using the RgbdSensorsSyncNodelet in combination with the RgbdSensorsDemuxNodelet as explained on 'https://www.stereolabs.com/docs/ros/zed-nodelets/'

• roslaunch zed_nodelet_example zed_sync_nodelet.launch camera_model:=zed2
• rosrun nodelet nodelet standalone zed_nodelets/RgbdSensorsDemuxNodelet __name:=demux_nodelet rgbd_sens:=/zed/zed_nodelet_sync/rgbd_sens
• rosrun zed_test zed_test (see the attached minimal working example - uncomment the lines near 'when using nodelets')

2: Using a stand-alone node that employs an image_transport::SubscriberFilter in combination with the message_filters::sync_policies::AproximateTime

• roslaunch zed_wrapper zed2.launch
• rosrun zed_test zed_test (see the attached minimal working example)

In short, I'm using the appoximateTime filter to synchronize the rgb and depth images from the "/zed2/zed_node/rgb/image_rect_color" and "/zed2/zed_node/depth/depth_registered" topics.
The callback of the synchronization shows identical timestamps for the rgb and depth input. However, the overlay shows a clear misalignment during camera movement, which means the depth map does not correspond to the rgb image (they are not properly synced).

If i'm overlooking something, please let me know. As far as I can tell, it is a bug ( or undocumented feature?) of the ros_wrapper.

This is really harming our robotics application, where the camera is continuously moving. Any help or fixes are appreciated.

Best regards,
Dennis)

minimal working example as ros node, where I've tried to make it as similar as possible to the synchronization nodelet.
zedtest.zip
example image showing the misalignment. The clock is a good example, where the 'occluded' areas are clearly misaligned to the actual clock in the rgb image.

[dennisVi]

I should have mentioned that I'm using an up-to-date master branch of the zed-ros-wrapper and have the ZED_SDK 3.2.2 for CUDA 10.2 installed.

[Myzhar]

Hi @dennisVi
this in an unexpected behavior. Thanks you for very good description of the problem and for the examples to replicate it.
I'm going to investigate on the causes.
I suppose this is related to https://github.com/stereolabs/zed-ros-wrapper/pull/557, but I thought that this problem was solved... maybe it's not the case

[dennisVi]

Some updates:

The desync only occurs for the ZED2 camera, not for the ZED mini.

We have both cameras at the office and we have created a dataset where both cameras were mounted on top of each other. This allows us to run the exact same pipeline and simply switch between the ZED2 and ZEDmini. We clearly observe a desync between the IMU data and the images.

Below is an example of an RTAB map output for both cameras, using the exact same data and pipeline, only interchanging the ZED2 and zedm svo and launch file. it is clear that the ZED2 cannot cope with strong turns (fast movement, at which point the desync becomes an issue).

ZED2:

ZED mini:

In the next example we walked slightly faster, showing even more desync moments.

ZED2:

ZED mini:

So the desync not only holds for the depth images, but also for IMU data. In its current state, the ZED2 is not suited for robotic applications.

I hope there will be a fix soon.

Best regards,
Dennis

[Myzhar]

Hi Dennis, we are investigating this issue at SDK level.
Is it possible to have the SVOs in order to be able to compare the data and debug them?
Please send an email to [email protected] to open a ticket about the problem

[Myzhar]

Hi @dennisVi
I just published a possible fix to the RGB/Depth sync problem.
It is available in the fix_629 branch.
I also added a tests folder in the zed-ros-examples package in the branch test_issue_629.
The test nodelet in the examples repository subscribes to RGB and Depth topics and publishes an overlay image to visually check for any desync.
There are a few options to enable the saving of single frames, you can activate them changing the launch file:
https://github.com/stereolabs/zed-ros-examples/blob/test_issue_629/tests/zed_sync_test/launch/zed_sync_nodelet.launch#L74-L83

Let me know if it works for you

[dennisVi]

Hi Myzhar,

Thanks for the fix. I'll be able to test it on Monday.

Best regards,
Dennis

[dennisVi]

Hi Myzhar and Walter,

I'm experiencing some issues with rtabmap when using the new branch.

It only shows these messages and does not seem to find a sync. Therefore I'm not able to check the results of the resulting rtab_map using my SVO files.

As I would expect, increasing the allowed delay does not help, neither does setting 'odom_sensor_sync' to 'true'.

I've also tried testing the depth delay as I did previously. However, the camera is currently at the office where we have only white walls without texture. I'm not able to reproduce the problem at this location (not even with the old branch), which is most likely due to lack of actual depth differences and textured surfaces (probably the AI refines the disparity for low-texture areas up to its visual edges). Also, my pc at the office is fixed, the usb cable is too short to go elsewhere and none of the usb3 extension cords seem to be working. I'm not able to test this further in the office. I would have to arrange for a notebook to record several SVO files at a different location and hope these files show what we need.

Perhaps good to ask, but the fix does not affect the SVO files to be made, right? Otherwise I'll have to capture two separate series with both branches.

Br,
Dennis

[Myzhar]

Hi, the fix does not affect the SVO. It's a fix made only on the ROS wrapper side to guarantee that RGB and Depth for the samer grab request are published with the same timestamp.

The problem that your are facing with TFs instead I think it is generated by a possible TF duplication or TF missing.
Can you run $ roswtf to see if it gives more info about the problem?

Finally a curiosity: your say increasing the allowed delay does not help... how are you increasing it?

Thank you
Walter

[dennisVi]

Thanks for the quick response.

In the rtabmap.yaml of the zed_rtabmap_example you can configure the max delay, which is what I meant to say

odom_sensor_sync: false
wait_for_transform_duration: 0.2
approx_sync: true

I've also ran roswtf and it outputs the following:

$ roswtf
Loaded plugin tf.tfwtf

No package or stack in the current directory

Static checks summary:

No errors or warnings

Beginning tests of your ROS graph. These may take a while...
analyzing graph...
... done analyzing graph
running graph rules...
... done running graph rules
running tf checks, this will take a second...
... tf checks complete

Online checks summary:

Found 1 warning(s).
Warnings are things that may be just fine, but are sometimes at fault

WARNING The following node subscriptions are unconnected:

• /zed_node/rtabmap:
  
  
  
  • /zed_node/global_pose
  
  
  • /zed_node/user_data_async
  
  
  • /zed_node/initialpose
  
  
  • /zed_node/gps/fix
  
  
  • /zed_node/imu
  
  
  • /zed_node/goal
  
  
  • /zed_node/goal_node
  
  
• /zed_node/zed_state_publisher:
  
  
  
  • /zed_node/joint_states
  
  

The exact same code runs successfully with the previous zed_wrapper branch.

[Myzhar]

Hi @dennisVi
I just tried to set to true the parameter odom_sensor_sync in our RTABmap example configuration and it works correctly for me:
https://github.com/stereolabs/zed-ros-examples/blob/master/examples/zed_rtabmap_example/params/rtabmap.yaml#L23

I receive the same warning as you get, but only when the node starts, but that's expected because the positional tracking is starting, while IMU data are just published.

[dennisVi]

Hi @Myzhar ,

I've tried the latest master and the 'test_629' branch of the examples, while setting the odom_sensor_sync to true, but it still does not help.
The errors persist even after 30 seconds, while other modules that read the RGB images work fine and show results after several seconds.

To ensure the problem was not in our code, I've simply ran the following example directly:
roslaunch zed_rtabmap_example zed_rtabmap.launch svo_file:=path_to_my_svofile.svo

It keeps spamming the same error message as before, even after minutes. Also, the example spawns two windows. The RTAB-map window only shows the pose, which does update over time. However, no point cloud is shown and the graph-view stays empty. The zed-rtabmap.rviz window only shows the pointcloud for the live view, but this is not propagated to the actual total model. See the attached screenshot:

We tested on two different computers, and both have the same issue.

Can you provide me with your full example, including the launch file that uses an svo input? Perhaps I can try to run your standalone test. I'll be able to look at it on Monday.

Best regards,
Dennis

[Myzhar]

Ok, wait... the problem is present only using the SVO, am I right?
I tested the example using a real camera connected to USB3

[Myzhar]

There are two main difference when using an SVO respect to using a live camera:

1. the SVO does not contain IMU data, but only fused camera orientation.
2. the sensor data do not run at 400Hz , but they are synchronized with the camera stream.
   The point 2 is really important because it can explain the warning that you are facing: sensors data pubblication runs in its own thread, in case of sensors/image sync the sensors data are emitted with the timestamp of the latest grabbed frame, but there is therefore no guarantee that they will be published at the same time. That's the cause of the warning.

I can work on this part to improve this behavior... but in your case, if you are trying to use IMU data you will not obtain a good result because they are fixed to zero (a not straightforward SDK modification is required)

[dennisVi]

Indeed I just tested with a live camera and that works fine. However, we have to work with the svo files that we recorded earlier and due to some unfortunate circumstances they are not easily captured anew.

I'm not completely sure if we actually need the IMU data, I'm assuming a fused camera orientation could be sufficient. I'll check with my colleague.

Btw. I appreciate your active support

[Myzhar]

It is a pleasure to fully satisfy the needs of our customers improving our products following your indications :+1:

[Myzhar]

@dennisVi I confirm that if you recorded the SVO with ZED Explorer the IMU data are missing for a bug that will be fixed for the next SDK release. Thank you for helping us to find it.

[JohanVer]

Hi, I recently encountered the very same problem (ZED camera 1). It seems that the depth image timestamp is just off by one frame. I tried the fix_629 branch but it seems to break the streaming completely. After starting the camera I get a few frames and it then stops streaming.
I think this bug makes the zed camera unusable for robotic scenarios in the current state.
Will there be a fixed master soon?

[Myzhar]

The desync issue has been fixed for ZED2 in fix_629.
We will test it deeper also for ZED to check if there is something wrong with it.

[JohanVer]

Thanks for the fast reply! Here are my configuration files if that helps: conf_data

[dennisVi]

@Myzhar Perhaps fix_629 does solve the desync issue, but it does not yet work for SVO files as other sensor data is not published at the same time in the SVO.
By not using IMU data we can perhaps avoid part of the problem, but we still need the other sensor data as well.

Can you give an estimate on when that will be fixed/supported? Until that time, we can no longer use any SVO files (also the ZEDmini SVO's cause issues in fix_629, while they worked before).

[JohanVer]

@dennisVi As I understood you correct you have prerecorded data and you want to fix that right? I have similar problems but it in my case I was able to correct my recorded data by just taking the previous depth frame for every rgb frame. I know that is just a dirty workaround but maybe it does the job until we have a fix.

[Myzhar]

@JohanVer I just tried the ZED with the fix_629 branch and it works correctly.
In the ZIP file that you attached I noticed that there is a launch file named zed_no_tf.launch, are you using it?
If you disable TF publishing you must be sure that another node is publishing the odom frame, otherwise the ZED node cannot reconstruct correctly the TF tree and before publishing data it waits for valid TF transforms

[Myzhar]

@dennisVi can you send me the SVO files (using a sharing service) to [email protected] so I can test them to understand the desync problem?

About IMU data recording, the ZED Explorer has been fixed and the new version will be added to the ZED SDK that will be released very soon. Meanwhile, you can use the svo recording example that correctly save all sensor data.

[dennisVi]

@Myzhar
I'm not allowed to share the SVO files, as they have been flagged confidential (restricted access area). So, the main issue is that they cannot be reproduced.

One of our partners did make recordings in a non-confidential setting. I'll have to see if those SVO files show the same issues and ask if they can be shared.

[Myzhar]

OK @dennisVi, I understand. Keep me updated about this problem so I can fix it better as soon as I have more information.

[dennisVi]

@Myzhar I have two SVO files that show the erroneous behavior using the ' fix_629' branch, one for the ZEDmini and one for the ZED2. Please send me a private message over email, such that I can share the download links with you. I do not want to post them on a public message board.

Br, Dennis

[Myzhar]

@dennisVi I cannot see your email in your profile. You can send the links to [email protected]

[dennisVi]

I've just emailed the download links.

Please keep me updated on the progress, as this is currently a blocking factor for our application.

Best regards,
Dennis

[Myzhar]

Hi @dennisVi
I applied many fixes to the branch fix_629 and now everything seems to work as expected in all the possible configurations.
I tested it with your SVO and RTABmap and I do not see TF errors anymore.
Let me know if you can test it. In case of your positive feedback, we will merge it into the master branch.

[dennisVi]

Hi @Myzhar
The first tests are succesfull. All SVO files are now working and the occupancy-grid map for the ZED2 now resembles that of the ZEDmini, i.e. no more strange deformations.

On Monday I'll run an additional check to see if the depth map and the RGB images are also correctly synced, by using a subriberFilter and overlaying the depth on the images. I'll need the camera for that, which is in the office.

So far, it looks like the fix is working properly. I'll keep you updated.

[Myzhar]

Thank you @dennisVi
Your feedback is really appreciated

[dennisVi]

Hi @Myzhar

The second test with the live camera was also successful. The depth map stays synchronized with the RGB images, even during motion of the live camera.

I can confirm that FIX 629 has resolved the issue. Thanks for the efforts and please let me know when it is merged to the master branch.

Br,
Dennis

[Myzhar]

@dennisVi merged