Zed-ros-wrapper: ZED frames latency and accurate time synchronization

[pavloblindnology]

I use ZED with ROS to do navigation stuff with IMU, navigation_stack, etc.
I have problems with accurate time stamping of ZED frames shot time.

If using zed-ros-wrapper approach, where the frame is time stamped by ros::Time::now() just before zed.grab, I get desynchronization (few frames) which leads to problems during rapid turns/movements.
I set camera_buffer_count_linux to 1 and it reduced the latency.
But still minor desynchronization is present. I suppose that this is due to the fact that frame grabbed by zed.grab is actually taken prior to this call, maybe during the previous zed.grab call.

I tried to use ZED internal time stamps, like this
zed.grab();
double frameAge = zed.getCurrentTimestamp() - zed.getCameraTimestamp();
ros::Time timestamp = ros::Time::now() - ros::Duration(frameAge * 1.e-9);

This gave me frames time stamped with approximately 1-frame latency, but it worked even worse than simple approach used in zed-ros-wrapper. And the time stamp latency reported with this approach doesn't depend on camera_buffer_count_linux value (always ~1 frame), though the latter clearly changes the latency.
According to description getCameraTimestamp it Returns the timestamp at the time the frame has been extracted from USB stream. (should be called after a grab()). So, it's not the time the frame was shot, but the time it was retrieved, which can differ.

So, question is how can I get the exact time the frame is shot by ZED?
Or, can I trigger the shutter of ZED from outside?

[nesnes]

The getTimestamp(TIME_REFERENCE_IMAGE) it the most accurate timestamp and is now in the Epoch format.
However, as the documentation mentions, it corresponds to the moment of the image extraction, not when it was actually read from the sensor.

If your synchronization requires this level of accuracy, I'll recommend you to calibrate the time offset between the IMU and the images using a tool like Kalibr.

[pavloblindnology]

Thank you for your answer.
I calibrate the time offset manually, which works ok, though it's inconvenient and the result depends on host machine.
Anyway, can you clarify the approach used by ZED a little?
Do I understand right that ZED buffers frames internally with the buffer length specified by camera_buffer_count_linux, and zed.grab call retrieves the oldest frame from the buffer and time stamps it with retrieval time of host machine, and after that it command the camera to shoot and store to the buffer the new one? Or ZED shoots frames independently of zed.grab calls?

Is it planned somewhere in the future to provide external triggering feature?

[klytech]

It seems that zed.grab() takes up to 110 milliseconds, and the rate is not a fixed number at all. The time varies up to 10 milliseconds which means the camera is not usable for visual inertial odometry, since most of the open source vio packages require synchronization between IMU and cameras.

Is there a way to reduce the latency? We are using Jetson TX2, with ZED SDK 2.2, USB 3.0 is used.

[pavloblindnology]

@klytech BTW, for me ZED's visual odometry works bad in real-life scenarios (outdoor, low FPS - 15 and resolution - up to 720p) - usually it's around 10% of all the time it keeps tracking. Can I know you experience?
And you probably know ZED Mini has internal vio.

[obraun-sl]

@pavloblindnology The behavior of grab is close to what you describe. Therefore it is strongly recommended to have zed.grab calls in a thread that can run as fast as possible.
Note that this behavior will change in the next release (v2.5) to have a behavior similar to what's happening on Windows (independant callback).

In Visual odometry, always prefer FPS to resolution. For example, It is better to use VGA @100Fps than 1080p@15fps for positionnal tracking. 720p60 is also a good compromise.