Cartographer_ros: Reliability evaluation of the output trajectory

[sunmk2006]

The output trajectory is primary function of cartographer,but what extent can I believe this trajectory?Are there some parameters in cartographer or algorithms that can evaluate the reliability of trajectories?
What's your suggestion?

[damonkohler]

Are you asking for something like covariance? Or are you asking how you can evaluate the quality of the trajectory compared to something like ground truth?

[sunmk2006]

Something like that，but how do we evaluate the quality of the trajectory without ground truth?
for example, does the matching algorithm or optimization algorithm have some output as a reference?

[damonkohler]

No, I don't think there's anything like that available. Evaluating the quality of a trajectory beyond subjective assessment of the x-rays is not something we developed a satisfactory strategy for.

[sunmk2006]

What x-rays have you mentioned many times?

[SirVer]

There is some ground truth data sets that can be used to judge the performance of Cartographer. See our paper about 2D examples. There is also work to look at the performance of Cartographer in googlecartographer/cartographer_ros#332.

[sunmk2006]

I still want to get cartogrpher's own evalution for some reason，even if it doesn't look so accurate or unwise,do you have any suggestions，i would appreciate it very much.
if MIT Stata center dare say their ground truth is 2cm accuracy #332 ,How many centimeters of accuracy can be achieved in cartographer 2D/3D model?

[SirVer]

Cartographer does not provide an evaluation of its own output - it is not a fully probabilistic system that gauges the estimate of its quality. The only thing you can do is compare it against some know ground truth. Our paper contains benchmarks against ground truth benchmarks for 2D and Cartographer fairs very well there.

In general, given enough loop closure and proper tuning, 2D SLAM error is around the voxel size. But this can never be generalized. For example looking at googlecartographer/cartographer_ros#332 you can see parts of buildings connected by hallways. There are no true loops, i.e. alternative ways to get from the left most wing to the right most wing - only one hallway. This means that slipping in the corridor or slight angle errors can lead to large global errors in these wings. While locally the SLAM will still be ~5cm accurate, the global position error (i.e. the lat/long of the walls in the wings) will be larger.

We never compared 3D SLAM against public datasets and published results. From my experience it works as well as 2D, but I have no hard data to proof this.

[sunmk2006]

Do you have some data that know ground truth for cartographer 3D model，or some public data sets, I want to do some tests on accuracy.

[SirVer]

No, as I said we did not do this for 3D yet. Should we ever get around to writing a paper about 3D, we will start investigating.

If you find any and do an investigation, please let us know, we are very interested in the results.

[sunmk2006]

• It's my pleasure to help you.

• What's more, I don't quite understand the accuracy of the trajectory test.The precision of the tracking(2cm accurate or 5cm accurate) is how to measure out,Is it the total length error of the trajectory or the average pose error of each point on the trajectory?

• Can I improve the trajectory accuracy if I adjust that parameter(the voxel size)?What parameters do I need to adjust if I don't pursue efficiency and focus only on precision?

[SirVer]

The accuracy metric (comparing to ground truth) is described in our paper.

I answered your last bullet point here as answer to bullet 5.

[sunmk2006]

hi,do you know the KITTI odometry dataset , i want to use it to do 3D cartographer accuracy tests , A few problems come up ：

1. the algorithm needs to use IMU information, but the IMU information is not found in this dataset.

2. the point clouds contain a sensor_msgs/PointCloud2 and timestamp per revolution,not one per UDP packet,Is it necessary to slice it and how to slice it?

3. After slicing, how does the output trajectory correspond to the ground truth in the dataset?

[SirVer]

Yes, I know KITTI - I was at the same institute as Andreas Geiger & Philip Lenz when they created Kitti and they are close friends.

See also #248 where @kentsommer looked into Kitti already.

1. IMU is provided in the raw data: http://www.cvlibs.net/datasets/kitti/raw_data.php
2. You have to write code for doing this. There are kitti raw -> ROS converters that are easy to adapt.
3. You have to find a rigid transform that minimizes the trajectory error. This is your frame conversion between kitti origin and cartographer origin. The remaining error is the Cartographer SLAM error. The kitti paper contains information how Andreas & Philip decided to compare trajectories and there is sample code on the kitti site to do it.

[sunmk2006]

For a point cloud per revolution, if i don't know the order in which the sensor receives the point cloud, how to slice, and give it a time tag like a UDP packet.

[SirVer]

The raw data contains the points in order. The firing pattern of the Velodyne is documented in its data sheet.

[sunmk2006]

From the document of Velodyne, I think that the time needed to send an UDP package should be equal，for example,vlp16 need 1306.37μs，HDL32E need 552.96μs,HDL64E S2 288μs and so on. Because my equipment hasn't arrived yet, i test it using a bag named b3-2016-04-05-15-51-36.bag from here,diff = this_stamp - last_stamp, Some examples are shown below, the values are not equal , Why is that?

[SirVer]

Because our deutsche museum data is YOLO timed - we used an approximation, since we had no hard real-time clock available to us.

[sunmk2006]

I don't have the concept of touching the knowledge of “YOLO timed",Can you give me some information about it?

[SirVer]

It is not a scientific term yet, I hope it catches on though: With Yolo-timing I mean to say that less thought, care and thoroughness went into time-stamping sensor data compared to extrinsic and intrinsic calibration.

[sunmk2006]

but if i do not care more about the time-stamping of per-UDP-packet point clouds,when it incorporates motion estimation that is closely related to the time stamp,Will the result be completely out of control? if not,why?

[SirVer]

Yes, the result will suffer. For our example bags we have tuned so that the timing jitter is not relevant for the final result, but if you want better results - i.e. smaller voxel size - you require better timing.

[sunmk2006]

How many parameters can be adjusted to make the final result independent of time jitter? Can you list a few?

[SirVer]

I do not get the question.

[sunmk2006]

I use kitti2bag tool to convert a raw data packet (2011_09_26_drive_0002_sync)into a bag file .The output frequency of the velodyne data is 10HZ, The following is the information of each frame，duration = this_stamp - last_stamp，Take second frames as an example ，The time interval with the last frame is about 0.103131s,it contain 12063 data points, Each UDP package is 384 data points in velodyne ，this frame contain 120603 / 384 approximately equal 314 udp package, The time per UDP package is 0.103131s / 314 approximately equal 328.4μs,but it is 288μs in HDL64E S2 , the situation is similar to other frames .I can't account for that，can you help me?

[wohe]

I'm not sure I got your last question. Could you explain what you observe and what you would have expected instead?

Keep in mind though that the timing of the laser firings is pretty accurate, so you would expect each UDP packet to contain information about a certain fixed duration. The 10 Hz rotation speed is something that the LIDAR tries to maintain, but is a lot less accurate. Rotation speed varies during each revolution, and different revolutions can take more or less time.

[sunmk2006]

The question I really want to ask is how to properly divide the laser data frames（which is 10 frames per second in kitti datasets） into UDP packets.From the data sheet of velodyn, i know each udp package should contain 384 points and the firing time of each udp packet will take 288μs in HDL64E S2, However, these two standards can not be correctly divided in the kitti bag file, Can you help me?

[SirVer]

Just cut the data arbitrarily in smaller pieces? You do not have to hit the UDP packet size exactly.

[sunmk2006]

In this case, the slice time also needs to be equally distributed，no longer limited to the firing capability of the equipment，yes?

[SirVer]

Yes. It is impossible to get to the original firing pattern from the data kitti released IMHO. If you want that, you need to reach out to andreas and ask for the original UDP packets (which would of course be very useful to others as well).

[sunmk2006]

ok,thank you very much!

[zhewen-hnu]

hi,do you know the KITTI odometry dataset , i want to use it to do 3D cartographer accuracy tests , A few problems come up ：

1. the algorithm needs to use IMU information, but the IMU information is not found in this dataset.

2. the point clouds contain a sensor_msgs/PointCloud2 and timestamp per revolution,not one per UDP packet,Is it necessary to slice it and how to slice it?

3. After slicing, how does the output trajectory correspond to the ground truth in the dataset?

Hi, sorry for disturbing. I try to use the kitti odometry datasets to run cartographer,but i can't get a proper result,not to mention comparing the output result with the ground truth.I try to use some tools like kittiplayer and kittihelper,but the result in rviz is wrong clearly,I don't know where the problem is,if you have got the proper output result,would you please give me a reference about it?
Thanks a lot!