Orb_slam2: Scale Drift Issue

[sunstarchan]

@raulmur

I tested some dataset, and I meet scale drift issue, especially when the car turns along the road.

do you know why scale is not consistent during straight road and curve road?
Any feedback is appreciated.

Thanks.

[joyousrabbit]

Hi @sunstarchan,
The scale issue is normal in Mono camera, because for mono camera (or said sfm), the reconstruction matrix can only be back to scale level.

Of course we have optimization tools such as SIM3 and BA, but there is no guarantee. Because the scale change, especially after tracking losing, has correct physical and numerical meaning (that's to say you can't not tell the new scale is right or wrong).

However, if you have some prior knowledge or be sure that scale will never change, you can use looping BA when it recognize a loop, as in ORB-SLAM2.

In conclusion, I guess that your new dataset doesn't has a loop, or the loop wan't detected correctly.

[sunstarchan]

Yes, the data doesn't have a loop. But I have GPS data for each frame. From my point of view, I can use this GPS data to correct the scale. My proposal is to keep the ratio of GPS and slam result of two KeyFrames.

For example, GPS distance of KF1 and KF2 is d1, two camera center distance is b1, GPS distance of KF2 and KF3 is d2, two camera center distance is b2. Then the scale of KF2 and KF3 is calculated by s = (b1 * d2) / (d1 * b2), I apply this scale to the baseline of KF2 and KF3. And the same time, I adjust 3D points corresponding to KF3 by using g2o::Sim3.

Do you think this is valid to correct the scale?

Thanks

[joyousrabbit]

@sunstarchan
Yes it's a good solution combined by GPS. Just take attention GPS has a error about 5 meters. Anyway, it should be OK for a car dataset.

In your case, the SIM3/BA doesn't have much sense, because the graph optimisation works generally for the case of looping. But you can still use it, it's better than nothing.

And, you needn't to do scale work. Calculate R as before, then set your GPS location transition as T; the scale is done intrisically. You can find an example at Tracking.cc:688 // Set median depth to 1.

Please report back if it works. It's very interresting to use GPS.

[yilun]

Hi @joyousrabbit , Im trying to use GPS to correct the scale drift as well.

I first tried to add GPS translation to last keyframe in Tracking and LocalMapping thread, but without success, I guess it was because the BA failed when there was a 'large' translation. Then I realized the map points need to be updated as well after BA and so the above two threads may not be the best place to correct the drift because it could crash the tracking. So, im trying to put it in loopclose thread by using SIM3 and BA to correct the accumulated scale drift, which is similar to what @sunstarchan was saying above.

Have not got success yet, but saw your reply today, why do you say SIM3/BA doesn't have much sense?

And Im working on a car dataset.

Thanks

[joyousrabbit]

@yilun
BA only takes sense when there is a loopclosing detected.

[ank700]

Hello @joyousrabbit @yilun @raulmur ,
Can you please explain, how exactly does scale drift occur?

I am thinking as follows (but not sure what is correct) - In case of a monocular setup, ORB does a initialization as follows, it detects features in first image and saves it as the reference keyframe. Now, after sufficient translation(parallax) a new keyframe is saved, the features of the reference frame are tracked and based on this and triangulation the camera pose and the map points are estimated in pixel coordinates(?).
The camera was initialized looking at a particular scene, and now it is translated while looking at the same scene. In this case, there should be no(or minimal) scale drift. Now, the camera is rotated and looks at a completely new scene. The 'estimated' scene depth is calculated again. Because of the depth calculation again and again, the scale drift error increases.

Any feedback is much appreciated.

Thanks

[joyousrabbit]

@ank700
If without close looping recorrect (update forward), you can image the case as following a right line. Then an integral error is accumulated monent after monent, because monent B's result is based on result A. Saying every step has a drift mistake (or whatever mistake, the data is data, "drift" is just a name) at 1%, then after several step, the mistake is accumulated.

So generally, for 3d reconstruction, you must find a cycle (or whatever) to link from monent to moment the current frame with older frames, to check if some error is accumulated.

I understand your question. But remember, the feature recovered from only one frame is not precise at all.

OK, let's take a real example.
In Frame 1, A B C is detected, and Dis_AB = 100cm, Dis_BC = 80cm, everything is OK.

In Frame 2, B C D is detected, where B C can be found in the model while D is a new one, however, from observsion of Frame 2, we think Dis_BC = 70 and Dis_CD = 100.
So we scale Frame 2, according the commun B C with Frame 1. The sacle is 80/70 = 8/7. Then Dis_BC = 80cm OK, Dis_CD = 800/7 = 114cm. But in fact, DIS_AB should be equal to DIs_CD, because they are two borders of the same square (did I say that? No, we could never know it in advance. The same for the program.) You see, a scale drift happens, by cause of the inter mistake in Frame 2, precisely, the mistake of observation of the propotion between Dis_BC and DIs_CD.

In Frame 3, we detect C D E, and Dis_CD = ..., Dis_DE = ... we can continue like this.

Finally, you will find the distance of Dis_YZ in frame N is very different with Dis_AB, even YZ and AB are both parts of a same grille who should have a same distance everywhere.

Is that clear?

[ank700]

Hi @joyousrabbit , great explanation. Now it is much clear to me.

So basically, the algorithm's internal scale should remain constant from frame to frame, theoretically.
If I change the way the scale is calculated when we see frame 2 and so on , such that Dis_BC = 80 for frame1 and DIS_BC=70 for frame2(assuming it to be correct) because the 2 viewpoints could be different in orientation also. But, this way the final pose error could be even larger, I think.
Where is this scale value (for frame 2) calculated in the code? I found invMedianDepth, but this is just for the scaling of pose and map points during initialization.

Is there any other way to correct the internal scale error internally? Using an IMU is definitely not a good option and because my usage is indoors GPS also makes no sense.

OR

In my case, I am using a height sensor to calculate scale and most of my motion is in a XZ plane, I will have to do some motion in Y axis only, after regular intervals based on keyframes created to update my computed scale value.

Thanks

[joyousrabbit]

@ank700
There is no code particular for scaling. It's calcualted by RANSAC the translation matrix. Then the scale is internal in the translation matrix (i.e. scale the new frames features to make them match the ones of frame1).

Seems your application like a robot. Are you recontructing 3d or just want some kind of obstracle avoiding? If just a robot application, we need only focus on local 3d instead of global 3d, so don't care about long time drift.

Or, just use something that you know its size in advance as reference. The size of the computer screen, the size/distance of one meter by robot, the size/distance between two cameras. If possible, use some markers to simplify the task. As human beings.

Or, because you have Y, maybe we can use Dif_Y as reference to correct the scale?

[ank700]

Hi @joyousrabbit , I am using orbslam for getting odometry. When I make some rotations, many time scale value changes drastically. So, there could be some variable in the code which could indicate that the scale has changed. The visualization in the map viewer window definitely changes.

Do you know specifically if something changes or can you tell me where and how is "scale the new features to make them match the ones of frame 1" done? I was going through the code trying to find something but did not find anything yet.
@raulmur , please if you can help me with this, it would be great.

Thanks

[joyousrabbit]

@ank700
maybe your tracking was lost when rotation? Then it reinitialized with new scale.

[ank700]

@joyousrabbit
Do you mean reinitialized or relocalise? Because when the tracking is lost, it always tries to re-localize itself based on the previously seen features.

[joyousrabbit]

@ank700
If re-localize failed, it will reinitialiaze.

[JoPaas]

Hello I am currently also trying to set the scale using odometry data. My Idea was to normalize the initial Translation to [m] (Initializer.cc line 915) but it didnt seem to make a difference. Then I tried adjusting the medianDepth / invMedianDepth in Tracking.cc. This changes the scale (using an arbitrary factor) but I fail to fully understand the correspondence to the odometry data I have available (Speed, Translation). Help or explaination would be highly appreciated!

Thank you

[JoPaas]

@joyousrabbit @sunstarchan
I think using the length of the translation between two Frames instead of the medianDepth leads to correct scale at initialization. The scale drift still occurs though, so the map (in my example KITTI seq 05) has different dimensions every time I build it.
I have yet to find out where I could pass the length of translations during mapping (other than at initialization) to prevent scale drift. Can you Point me in the right direction where to add to the code?
@sunstarchan were you able to incorporate GPS data? ..since I am trying something similar.

Thanks