Sceneform-android-sdk: How can I get CameraSource(or frame source) from Sceneform?

[hcjung2000]

Hi, I want to read QR Code from sceneform ArSceneView with BarcodeDetector.

but I can't get CameraSource.

Is there any method to get frame source? (or possible way to read QR Code in Sceneform session?)

Of course, I can get bitmap from surfaceview, but it is too slow.

[claywilkinson]

You can access the camera stream from ARCore's frame object. What I did was

Register the listener to be called every frame
sceneView.getScene().setOnUpdateListener(this::onSceneUpdate);

Then you in onSceneUpdate you can access the ARCore frame and copy the image by calling acquireCameraImage(). This example is the "computervision" code from ARCore. From the image object you should be able to get the buffer to feed to the barcode detector.

  public void onSceneUpdate(FrameTime frameTime) {
        if (session == null) {
            return;
        }

        Frame frame = sceneView.getArFrame();
        if (frame == null) {
            return;
        }
        // Copy the camera stream to a bitmap
        try (Image image = frame.acquireCameraImage()) {
            if (image.getFormat() != ImageFormat.YUV_420_888) {
                throw new IllegalArgumentException(
                        "Expected image in YUV_420_888 format, got format " + image.getFormat());
            }

            ByteBuffer processedImageBytesGrayscale =
                    edgeDetector.detect(
                            image.getWidth(),
                            image.getHeight(),
                            image.getPlanes()[0].getRowStride(),
                            image.getPlanes()[0].getBuffer());

            Bitmap bitmap = Bitmap.createBitmap(image.getWidth(), image.getHeight(),
                    Bitmap.Config.ALPHA_8);
            processedImageBytesGrayscale.rewind();
            bitmap.copyPixelsFromBuffer(processedImageBytesGrayscale);
        } catch (Exception e) {
            Log.e(TAG, "Exception copying image", e);
        }
}