Arcore-android-sdk: Feature Request: Efficient CPU access to camera image pixel data

[twkx]

ARCore preview 2 provides a sample for computer vision, bud it only gives access to previous camera frame RGB data, involving OpenGL rendering are PBO reading.

With ARCore preview 1, I successfully got access to TangoImage object (via TangoUpdateCallback.onFrameAvailable() listener of Tango instance) by using a bunch of Java reflection on ARCore Session object. This provided me a full access to YUV camera frame data and life was so perfect. Now this hack is no more possible with preview 2, and the provided solution takes a lot of cpu time to finally get RGB data of the previous frame!!

I definitely don't understand why we can't have access to the camera image data ARCore is using for it's own analyse, and why we have to use complex OpenGL tricks to render the camera frame and get the exact same pixels ARCore is using in behind! What a waste of cpu time...

I really hope next ARCore release will provide a direct access to camera frame.

Thanks a lot.

[inio]

This is definitely something we'd like to offer, as a proper scaler-based CPU image would be much more efficient than the GPU-based copy from the texture. It's on our radar, but we can't offer a timeline just yet.

[twkx]

Terrific, I'm so glad to read this :)

Thanks for this quick reply.

[gitunit]

@twkx sounds like a good hack you did with the preview 1. any chance you will share it with us?

[twkx]

By the way, I also noticed that reading the ByteBuffer returned by glMapBufferRange() is really slow compared to others ByteBuffer created via allocateDirect().

This may be explained by the following:

_Mappings to the data stores of buffer objects may have nonstandard performance characteristics. For example, such mappings may be marked as uncacheable regions of memory, and in such cases reading from them may be very slow._

https://www.khronos.org/registry/OpenGL-Refpages/es3.0/html/glMapBufferRange.xhtml

This nearly kills all the performance gained by using PBO :(

[twkx]

@gitunit: here is the helper class I used for ARCore preview 1. As Tango has disappeared from preview 2, this code is now useless.

To use it:

• create an instance of TangoHelper right after Session.resume()
• register your callback after the first call to Session.update()
• unregister your callback just before the call to Session.pause()

I hope we'll be able to register this kind of listener in next ARCore release, without any hack ;)

Best,

TangoHelper.java.txt

[inio]

@twkx

By the way, I also noticed that reading the ByteBuffer returned by glMapBufferRange() is really slow compared to others ByteBuffer created via allocateDirect().

That's concerning. What device are you seeing this behavior on?

[twkx]

@inio I use a Samsung Galaxy S8 (SM-G950F)

Copying a ByteBuffer allocated with allocateDirect() into another ByteBuffer (with put()) takes 2ms in average (for a size of 1920 x 1080 x 4).

Copying the ByteBuffer (of same size) returned by TextureReader.acquireFrame() into another ByteBuffer takes around 13ms.

[inio]

@twkx Small update: We've been able to reproduce the disappointing buffer read performance on Exynos-based Samsung S8 devices. Thanks for pointing this out.

[wkl-coder]

In the future we can get Camera onFrameAvailable data?

[eddielong]

Having access to just the luminance or the RGBA isn't adequate for my use. I need access to the YUV frame as I need to use the y-plane for CV and also need to convert to jpeg. Going from RGBA to luminance on the GPU will be really slow and I simply don't have the UV data in the ARCore preview 2. I'm going to try to cook up a way of getting back all the planes using the current PBO 'hacks', not even sure if I can do this efficiently. Having onFrameAvailable/onPreviewFrame would be the ideal solution.

[inio]

The just-released ARCore SDK v1.1.0 includes CPU access to the same image used for motion tracking. Not a complete solution yet but making progress.

[inio]

Note: increasing the resolution of this image is tied in to #153.

[twkx]

@inio Thanks a lot for this release, it is just exactly what we needed!

[Godric877]

Is there a way of accessing the RGB values of a point whose 3D coordinates are known in ARCore?

[alienwings]

ARCore 1.2 release has solved this issue

[inio]

@alienwings Sorta. ARCore 1.2 provides a 640x480 image. We'd like to offer some configurability for higher resolutions before we call this one solved.

[inio]

ARCore 1.4 adds support for higher-res CPU images via Session.setCameraConfig (available configs are accessed using Session.getSupportedCameraConfigs()