Arcore-android-sdk: Extracting Point Cloud

[Godric877]

Hi!

I have been working on an app to extract the 3D point cloud from ARCore and further process it to generate a 3D model using the texture from the camera photos. However, I am encountering some problems and have some doubts regarding ARCore. I would be grateful if you could clear them.

1) What is the coordinate system in which the Point Cloud acquired in a particular frame is represented? Is it a global coordinate system, so that the coordinate of a point will not change over time when it is observed from different frames? Or, is it a local coordinate system which will change coordinates every frame?

2) I was planning to place anchors at all the tracked points (ignoring the CPU overhead for now) so that ARCore continuously tracks these points and changes their coordinates as the Camera moves through space. Is this correct?

When should I be using anchors? Are the numerical coordinates of anchors supposed to change with every frame? Or, are they supposed to be in a world coordinate system and remain constant with time?

3) Do you suggest some better method to accumulate the point cloud information (the coordinates of the point cloud are in a single coordinate system)?

4) Will ARCore provide some functionality for 3D reconstruction in a future release?

[lvonasek]

Hi,

I am not a Google developer but I will answer you as I worked on the same - I am the developer of this app: https://play.google.com/store/apps/details?id=com.lvonasek.arcore3dscanner

1. Point cloud is in 3D. Zero position is the position where the ARcore session started and it is using meter units.

2. You can get camera pose by ARcore API but when it loses motion tracking then the coordinate system is not the same.

Anchors are connected to feature points (it has no coordinates at all but it is some kind of image description). Anchors can be used to get 3D position in different coordinate system (e.g. after restart mentioned in point 2).

1. ARcore is providing point cloud in one coordinate system as long as the motion tracking is continuous.

2. I tried to get answer on this on many places but without any result. I believe Google will provide it together when they add https://github.com/google-ar/arcore-android-sdk/issues/120 function.

There is deprecated Google's library for 3D reconstruction called Tango3DR, I am using it in my app and it is working. This library is created for devices with active depth sensor, doing reconstruction of the ARcore pointcloud is possible but you will receive not so good results.

[Godric877]

Hi,

Thank you so much for answering! You cleared a lot of my doubts, I will look into the library.

Could you also please point me towards how and where to start using the library?

[lvonasek]

Tango3DR library is very complicated and it was problem for me to make it working on Tango device (and even bigger to make it working on ARcore). I do not expect that you will make it working on ARcore. Maybe it would be better for you to wait until you get answer on your point 4.

Anyway here is the link:
https://github.com/googlearchive/tango-examples-c

More info here:
https://github.com/personalrobotics/OpenChisel

For your imagination, here is reconstruction by marching cubes:
https://youtu.be/UPuFItISE58

and here by Tango3DR:
https://youtu.be/L8IDeopw08Y

[Godric877]

Thank you for the pointers! :)

[inio]

Thanks for taking this one @lvonasek

[ommmid]

@lvonasek Hey thanks for your information. Do you have any suggestion on integrating 3d-model based tracking with ARCore? I have the 3d model of a room and its objects and I am wondering if i can match those points captured by Point Cloud in ARCore with the 3d model to have a better positional tracking. Any though would be appreciated.

[lvonasek]

@ommmid Point cloud is not enough for matching (you have no garantie that you will get the same points on next session and there is no easy way how to match the points). You need feature points (point cloud including information about colors around points -> anchors).

On Tango it was called ADF, I requested it here:
https://github.com/google-ar/arcore-android-sdk/issues/94

In company, I am working for, we are using optical markers (QR codes), user finds the marker and then we have coordinate system we want but let's say with every meter the pose error increases (10m far from the marker the error is e.g. 40cm)

[inio]

@ommmid As @lvonasek said, that's not currently possible. We have an open feature request though: #418.

[ommmid]

@lvonasek With QR codes, you have to have this QR in the camera field of view all the time, right? Or do you have any other method? I have the same project I think. I can use the QR code too but there is no guarantee that the user (a repairman) keep QR code in the FOV of camera. Do you any other method (non-optical) any sensor or any other ways that we can track the user's device?

[lvonasek]

@ommmid No, after start of ARcore I say to user that he has to find QRcode then I transform the current ARcore coordinate system into my scene coordinate system (since this moment I do not need QRcode)

The problem is when the ARcore loses motion tracking - the recovery really does not work so I have to restart ARcore and force user to find QRcode again.

[Godric877]

Do you know a way to perform mesh reconstruction given a 3D point cloud which can be implemented in android? I looked at PCL and CGAL but was unable to build them for android. Any help would be highly appreciated as I have been stuck on this for days.

[lvonasek]

Currently there is no easy way to do it. The most promising solution presented 6d.ai. They have a very powerful demo for iOS. I understand from talk with them, it is computing on GPU and currently they are not aware of a way how to do the same on Android. But they plan to bring it also on Android (I guess somewhen in 2019).

[Godric877]

Have you been able to build either PCL or CGAL for android? If so, can you help me through it?

[lvonasek]

You should use more Google:
https://github.com/bashbug/pcl-for-android
https://github.com/CGAL/cgal/wiki/Cross-Compilation-of-CGAL-for-Android

[Godric877]

I have already tried both of these, I encountered errors on both.

[lvonasek]

I found a link to the precompiled PCL in ARcore slack group:
https://drive.google.com/file/d/0B2NvEgKbIDm_R0JWclJNOUlFX2M/view

[inio]

You're welcome to continue this conversation here but I'm going to close this issue. In the future this type of discussion is more appropriate for StackOverflow.