Arcore-android-sdk: Feature Request: add confidence value to Plane detection

[kenyee]

• Android Studio: 3.4 Canary 6
• ARCore SDK for Android: 1.5.1
• Device manufacturer, model, and O/S: Samsung Note 8 running Android 8.0

Feature request is to add confidence value to tracked planes, similar to the confidence value we can get from the point cloud.

Use case: I'd like to autoselect the "floor" by asking users to point the phone at the floor instead of making them do the extra step of tapping on a plane. I do this by grabbing the first tracked horizontal plane, but when I do this and use the plane.centerPose to create an anchor, it randomly doesn't seem to be very accurate (floor is 10" below the actual floor). Tapping on a floor plane is totally accurate in comparison.

STEPS TO REPRODUCE THE ISSUE

1. add an onUpdate override to the ArFragment to grab the trackables and find a horizontal up plane like so:

class ArFragmentWithNotifications : ArFragment() {
    internal var planeFoundCallback: ArPlaneFoundCallback? = null
    private var notifiedPlaneFound = false

    override fun onUpdate(frameTime: FrameTime?) {
        super.onUpdate(frameTime)

        if (!notifiedPlaneFound) {
            this.arSceneView.arFrame?.let { frame ->
                checkTrackedPlanes(frame)
            }
        }
    }

    private fun checkTrackedPlanes(frame: Frame) {
        frame.getUpdatedTrackables(Plane::class.java).iterator().forEach { plane ->
            if (!notifiedPlaneFound &&
                (plane.trackingState == TrackingState.TRACKING) &&
                (plane.type == Plane.Type.HORIZONTAL_UPWARD_FACING)) {
                planeFoundCallback?.planeFound(plane)
                notifiedPlaneFound = true
            }
        }
    }
}

1. use plane.createAnchor(plane.centerPose) to create an anchor to the "floor" automatically
2. the plane is 10" below the floor maybe 30% of the time

WORKAROUNDS (IF ANY)

Ask users to tap on the floor plane to get the floor anchor.
Maybe ignore plane info for a few seconds until ArCore has had a chance to evaluate the room conditions better?

ADDITIONAL COMMENTS

I think a similar request is mentioned here:
https://github.com/google-ar/arcore-android-sdk/issues/395

[tangobravo]

I've got a similar concern - I update my floor estimate with the tracked plane with the lowest y value, but there are occasionally phantom planes reported below the actual floor surface. I suspect these are features that come from the ceiling being reflected in the floor surface (but may also just be mistaken feature tracks).

For larger tracking areas I can imagine somehow giving a larger weight to planes with a bigger extent, but the phantom planes are still occasionally reported with a significant size. Once the phantom planes are found it seems quite challenging to get them to be removed again from the set of reported planes. I've noticed this just from playing with the hello-ar sample.

Quite often the correct floor plane completely covers the phantom lower plane (so a hit test would always return the correct floor plane) but detecting this just from the provided data is challenging.

In a space-carving type of approach, it would be possible to discard the lower planes if it is completely occluded by other planes in all of the historical devices poses in the session. I don't really feel like writing that myself though! 😄

[inio]

That sounds like a pretty reasonable request. I've recorded it.