Orb_slam2: purpose of usleep in local mapping and loop closing threads

[lokeshluckybisht]

Hello,

I want to know what is the significance of using usleep function at the end of threads for local mapping and loop closing?

Thanks

[AlejandroSilvestri]

@lokeshluckybisht

These threads process a queue of work. When the queue is empty, there's no point on repeatedly query the queue at max speed. So the thread sleeps, freeing some cpu time for other threads.

ORB-SLAM2 uses 4 threads, including one for viewer. OS uses other threads. It is good practice to sleep some ms when waiting for work to do.

[yiakwy]

@AlejandroSilvestri I suggested to use conditional variable to do that.

Instead of using usleep, it is better to use conditional variable to hung up the thread and notify it when a new keyframe is inserted into a thread safe queue.

std::unique_lock<std::mutex> lock(q_mutex_);
if (wait) {
  condVar.wait(lock, [*] () {
  return !accept || queue.empty();
  })
}

// processing new active frame computed from tracker

It is cheaper because, usleep requires more context transition.

B.T.W, I also have some concerns about local mapping thread. I cannot precisely tell the differences between local mapping and main thread MonocularInitialization.

We have triangulation in MonocularInitalization :

1. first, do matches
2. second, estimate current frame poses if we have 3d points already using PnP solver, otherwise estimate by computing fundamental or homogenous matrix
3. estimate depth using triangulation
4. bundle adjust with all points and poses
5. register them to map
6. set tracker to be initialised

While in local mapping:

1. find matches (should be already computed in main thread)
2. create new triangulated 3d points (Duplicated in main thread?)
3. map culling (remove bad points and ?)
4. do local bundle adjustment (current frame pose refinement) with local frames , i.e, vertices and non-neighbor poses are fixed

operations 1~3 in local mapping are really confusing. Could we directly use the result computed in the main thread? For example directly use the result computed from Monocular Initialization

[AlejandroSilvestri]

@yiakwy

Agreed, conditional variables are a lot better. There is a lot of room for code improvement, specially with C++17 (and 20), for example using for range, auto, structured bindings... And the extensive use of shared pointers.

Monocular initialization runs when there are zero mappoints. It uses homography and fundamental matrix in parallel. It doesn't use PnP. These aren't good ways to triangulate points: they are prone to error. Initial map obtained with initialization are subject to much improvement in the first bundle adjustment.

Local mapping operates with keyframes, not with frames. It runs when a new keyframe is added. It looks for new mappoints triangulating from keyframes pairs. These pairs aren't current and last frames, the ones matched in Tracking (main thread). This is the only point in code where mappoints are created, besides initialization. This triangulations have the benefit of having poses and BoW already computed, the algorithm is different and more efficient than the one in initialization.

Map culling is about reviewing old mappoints and keyframes and removing those redundant and less used. Map culling doesn't apply on the new keyframe nor recently added mappoints.

Many parts of orb.slam2 are similar but slightly different, highly tuned.

[yiakwy]

@AlejandroSilvestri Understand, very intuitive. Though I still have a different opinion. Considering the fact that:

1) Matching can be accurate only when two frames are close enough. I implemented a frame matcher always matching last frame based on OpticalFlow and Kalman predictors. This can be achieved very high matching accuracy and I can verify it.

2) Triangulation is accurate when we have enough stereo disparities, i.e. the we should match frames not close". Hence Implemented a Union Find algorithm to trace matched pixels so that I trace back from pixel.parent to find a match with enough stereo disparity.

3) For the moment, I treat a frame as "Key Frame" if it can be triangulated with the accepted accuracy (reprojection error 1e-4 ~ 1e-6)

I am going to do some experiments to see If I remove operation 1~3 , how many new points will be lost.

[AlejandroSilvestri]

@yiakwy

That's impressive. Reminds me of GSLAM. GSLAM uses KLT tracker (a tracker based on Lucas-Kanade optical flow) to track features with very high performance.

With an optical flow tracker you only can detect mappoints if they are continuously tracked, and that will reduce your mapping area.

Let me know about your findings. I'm very interested on fast alternatives to orb-slam2. I'm considering writing a whole new visual slam from scratch.

[yiakwy]

@raulmur @AlejandroSilvestri I want to take this opportunity to share my recent open source project for sparse slam using semantic information : svso

Github: https://github.com/yiakwy/SEMANTIC_VISUAL_SUPPORTED_ODEMETRY

I also attached a short report to my peers about key ideas behind the software system, algorithms and the work. I have been trying to build HDMap hybrid pipeline using both semantic information from images and spatial information from depth graph built either by epipolar geometry or observed 3d points from structural light in the past few years.

Let me know what you think about my work and I am glad if we could have further discussion and cooperation together.

[AlejandroSilvestri]

@yiakwy

Very interesting! If only I had time to try it!

Couple of questions: is it written in Python? Does it run in real time? How about performance?

[yiakwy]

@AlejandroSilvestri The project is a python/c++ project. Python is fast for POC purpose.

The major problem when I wrote the fist line of the codes is that, how to integrate semantic deep neural network into a slam system written in c++. As you can see I have developed a tensorflow based inference engine so that we are able to easily load models into our program.

As you may know mast jobs in the domain of "AI" for the moment are developed in python. Instead of write every network components from scratch, I have to use existing python libraries to test whether my freshly developed Semantic Feature Extractor (SFE) work.

The second problem is that even if I have done the most of jobs about SFE, integrate it in c++ is not that easy as I used to expect. There are many operations and preprocessing works for the input of Pyramid Network Component and SFE.

The third problems, I don't have GPU at the beginning and Google colab helps me in demonstration. Though writing c++ in colab is feasible but is a very unpleasant thing.

Finally, I made decision to start my work in two parallel pipelines:

• in python line, I could get incremental progress so that I can verify that the theory works with weak parallel and asynchronous support;
• in c++ pipleline, I could focus on overal system perforfance and dirty works (for example, eigen memory alignment).