Arcore-android-sdk: Feature Request: use front-facing camera

[Viv3k]

Is this supported? If so, how? Kindly advice. Help is appreciated.

[nathanmartz]

ARCore only works on the rear/world facing camera, not the front facing camera.

[Viv3k]

Is there any reason/limitation why it is not supported on the front camera..?

[inio]

1. It doubles the calibration work
2. Front cameras often have higher distortion and more mechanical/optical variability compared to back cameras
3. The front camera is not as useful for the kinds of AR experiences we're focusing on right now.

[inio]

Duplicate of #128

[yperess]

While AR is not very useful with the front camera, many applications could require it. I've been working on an app that renders multiple objects using ARCore and would allow the user to capture a "selfie" at various points. On pixel devices I'm able to open the front camera at the same time as the rear so it's not an issue, but the s8 appears more limited. I have to stop my ARCore session, start a Camera2 session, take the picture, then restart the ARCore session. It would be nice if ARCore could allow getting frames from the front camera directly to make the experience more seamless.

[flankechen]

@yperess front and rear camera runing at the same time is somehow device and "rom" independent feature. I could have two cameras running well on one plus 6 with Unity webcam texture, but marking it in ARCore example crash, saying like "FATAL EXCEPTION: UnityHardwareCamera", "java.lang.RuntimeException: startPreview failed".
coudld you kindly point out which pixel you are using, and how you open front camera at the same time in ARCore?

[yperess]

@flankechen Wow it's been a long time. I can't directly speak for Unity but on my Pixel and Pixel2 I've been able to run ARCore on the rear camera, then open another TextureView and a Camera2 session for the front camera. The view will render frames from the front camera and will capture the user's reactions to what they're seeing in AR without needing to kill the ARCore session. You can then use the Frame from your ARCore session as well as the Camera information to augment your front camera feed (you just need to reverse the camera direction).

Best current solution:
On devices that only support one camera at a time we can pause the session, start the front camera, then resume the session when we're done.

The ideal solution:
Allow ARCore to continue the session, but provide frames using the front camera. This would require overloading the https://developers.google.com/ar/reference/java/com/google/ar/core/Session.html#setCameraTextureName(int) method to allow developers to supply a rear/front texture (in devices that only support 1 camera they can just be the same). ARCore would then allow the Session to be toggled between FRONT/REAR/BOTH modes, BOTH throwing an exception if the chip set doesn't support it.

In the case where a phone can only support one camera at a time, frames would continue coming into the provided front texture. If only one camera can run at a time AND the lack of enough anchor points ARCore could pause the session, this would be denoted by returning TrackingState.PAUSED for the Frame's Camera. Eventually drift will accumulate and the tracking state would have to be STOPPED.

The point here is that there's no guarantee that the session needs to be paused, if both cameras can run the session can continue regardless of what's on the front camera. Otherwise, it's possible that a quick selfie could allow the session to only pause, or even continue uninterrupted if the camera is far enough and has enough stable background feature points. Without ARCore handling this directly we could never achieve the ideal solution, at best we could always pause the session.