Sceneform-android-sdk: SceneView and Camera preview

[galunecandrey]

Tell me please, how can I use SceneView with camera preview on background without AR session?

[JessHolle]

So I believe this is intended to be feasible via external textures -- see discussion on https://github.com/google-ar/sceneform-android-sdk/issues/169

That said, I've not actually tried to implement this as of yet.

If this approach does not work, then this gap needs to be resolved -- as 3D + camera background can't be limited to just ARCore devices.

[galunecandrey]

@JessHolle I hope that you will implement this functionality in the near future. Or add an example of such a functional implementation in github via Surfactexture, as in the video example

[Brianvdb]

An example on how to do this would be great.

[romainguy]

You can do this already with ExternalTexture indeed. ExternalTexture gives you a SurfaceTexture through getSurfaceTexture(). You can then pass this to the camera API's setPreviewDisplay(). Even better you could use the Camera2 API and pass the ExternalTexture's Surface instead of the SurfaceTexture.

[galunecandrey]

@romainguy This is all clear. It's unclear how to add a ExternalTexture to the SceneView as background...

[Brianvdb]

Yes its not clear how to do this as a background..

[dsternfeld7]

Create a custom material like this:

material {
    "name" : "Camera Background",
    "defines" : [
        "baseColor"
    ],
    "parameters" : [
        {
           "type" : "samplerExternal",
           "name" : "cameraTexture"
        }
    ],
    "requires" : [
        "position",
        "uv0"
    ],
    "shadingModel" : "unlit",
    "vertexDomain" : "device",
    "depthWrite" : false,
}

fragment {
    void material(inout MaterialInputs material) {
        prepareMaterial(material);

        vec2 uv = getUV0();
        material.baseColor = texture(materialParams_cameraTexture, uv);
        material.baseColor.rgb = inverseTonemapSRGB(material.baseColor.rgb);
    }
}

The key line in this material is "vertexDomain" : "device",. This makes it so that when the Renderable is rendered, the transformation of the node is ignored and it is rendered in clip/device space, which is normalized between -1 and 1 across the dimensions of the device. If you create a quad that is in clip coordinates, then it will render across the whole screen.

.obj file for a quad that renders across the entire clipping space:

o Plane
v -1.000000 -1.000000 0.000000
v 1.000000 -1.000000 0.000000
v -1.000000 1.000000 0.000000
v 1.000000 1.000000 0.000000
vt 1.0000 0.0000
vt 1.0000 1.0000
vt 0.0000 1.0000
vt 0.0000 0.0000
vn 0.0000 0.0000 1.0000
s off
f 1/1/1 2/2/1 4/3/1 3/4/1

Then later:

cameraTexture = new ExternalTexture();
modelRenderable.getMaterial().setExternalTexture("cameraTexture", cameraTexture);

String cameraId = cameraManager.getCameraIdList()[0];
CameraCharacteristics characteristics =
    cameraManager.getCameraCharacteristics(cameraId);

StreamConfigurationMap map =
    characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
Size previewSize = map.getOutputSizes(SurfaceTexture.class)[0];
cameraTexture
    .getSurfaceTexture()
    .setDefaultBufferSize(previewSize.getWidth(), previewSize.getHeight());

You'll then have to send capture requests via Android's Camera API to render the camera preview to the surface as you normally would. You may also need to set the priority of the renderable to ensure that it is rendered behind everything else.

There's a very important thing that still needs to be done to complete this solution. The .obj file I provided has UV coordinates for right side up portrait, the camera preview will face the wrong direction in other orientations. Additionally, the camera preview will appear stretched because of the difference between the device's aspect ratio and the camera's aspect ratio. To fix this, You'll want to actually use renderable.getMaterial() to get the material from the .sfb file, and then render it with procedurally generated geometry using RenderableDefinition that will have different UV coordinates depending on the orientation of the device and relative aspect ratios.

[EhabMH]

@dsternfeld7 hello please I have tried this solution and still doesn't work the 3dmodel is rendered however the background is black and not the camera preview, can you provide me with a tutorial or a code that works and I will follow it .

[subhafalcon]

@dsternfeld7 I have tried this solution but I am having some difficulties, is it possible to post the full source or a link to it? It would be very much appreciated!

[Hujunjob]

thank you very much i did it😁

[firetrap]

thank you very much i did it😁

@Hujunjob can you provide the source or some example? I'm trying to apply the texture but with no results.

[jpsarda]

Also wanting to show the camera preview in a sceneform scene, there is already good hints on this thread, but a piece of code from those who already succeeded would help...

[amarkovits]

please somebody provide some code