Orb_slam2: How to use it on Android

[mafanwei]

I want to try it on Android.

[AlejandroSilvestri]

@mafanwei

You should compile it on Android. For starters, I believe the actual cmake won't work, you must write your own.

If you succeed, please let us know!

Last time I tried, many years ago, it was painfully slow, like 30 sec per frame. Of course it depends on your phone's processors.

[mafanwei]

This is a sad story.
I am not have enough time to solve compile problem.
So I didn't run it either.
Maybe i will have time to do it in the future.
If I success, I will tell you.

[AlejandroSilvestri]

@mafanwei

Running on phone is the holly grail of visual slam. The best visual slam available were looking for it for many years now.

[pouyaAB]

Checkout this repo.
https://github.com/Martin20150405/SLAM_AR_Android
While it didn't work in all cases for me. it does work sometimes and it is definitely faster than 30 sec/frame on a Pixel 2.

[netgfx]

If we host ORB-SLAM2 on server what kind of data set we need to send it and in what format? I assume Pose point cloud, but is there a specific format to broadcast?

[AlejandroSilvestri]

@netgfx

There's no established format.

You must decide how much processing the phone can do. For example, you let the phone apply ORB, send all keypoints and their descriptors to the server, and let the server compute the pose (localization, aka tracking) and do the mapping.

If you have enough power in your phone, you can let the phone do the localization, send the same ORB data but organized in grid, plus the obtained pose, and let the server do the mapping. And the server must send local mappoints to the phone. I believe a phone can cope with the "tracking only".

It is very important to reduce communication lag to a minimum, so it is important to choose the right protocol.

I should try the second approach (localization in the phone), using TCP/IP for minimum overhead, and an ad hoc format with protocol buffers or even better cap'n proto because there's no chance of compression in this data.

[netgfx]

Thanks for the explanation, some things are more clear now, is there some resource I can look into on how to construct the ORB data for the server to understand them? Like does it need only vector3 points or the whole 4x4 float matrix? Also from my research I found that sending streams of image data is also an option?

[AlejandroSilvestri]

@netgfx

Yes, video stream is the first option I didn't mention. It is the easiest one. My students did it many times. It worths trying although there are two drawbacks:

• video is compressed (no big problem, but not good for benchmarks nor evaluations)
• lag (depending on how you stream the video)

It works, and it is a great first step before sending keypoints.

Protocol buffers for orb-slam2 maps
Saved map viewer

[netgfx]

Thank you, much appreciated!