Libgdx: Android 8.1 device camera is not shown behind the game scene

[lucabee]

I use the following code to show android device camera as background of my game scene:
https://github.com/libgdx/libgdx/wiki/Integrating-libgdx-and-the-device-camera

On Android 8.1 devices, the camera preview is not shown behind the game scene, but appears on foreground.

LibGDX version: 1.9.8 and 1.9.9-SNAPSHOT

Please select the affected platforms

• [X] Android
• [ ] iOS (robovm)
• [ ] iOS (MOE)
• [ ] HTML/GWT
• [ ] Windows
• [ ] Linux
• [ ] MacOS

[EtK2000]

Did you add the permission to the manifest?
Is there anything in the log?

[lucabee]

I don't think it may depends on manifest permission, because I can see the camera preview correctly, but game elements are not on foreground.
The code works perfectly on android 7 and below versions.

[EtK2000]

Sadly I don't have any device over 7.1.1 so I can't test this.
Hopefully someone else will be able to replicate.

[lucabee]

I have replicated the issue also using emulator with API 27

[EtK2000]

I'll try that and see if I can find anything.
I'll also look into the changes in Android since 7.1.1 and see if anything could cause this.

[gfphoenix]

You can consider to use SurfaceTexture to receive the camera preview, the old way doesn't work in 8.0/8.1. I have the same problem a month ago. You should write a shader, to render the SurfaceTexture

[lucabee]

I have used SurfaceTexture to show camera preview and render it with shader (before of textures and sprites render) but device-camera still on top of the scene instead of background

[gfphoenix]

Maybe, I should make it clearly. There are lots of tricky issues. the camera setup steps are

1. open the camera you want to use
2. configure preview size
3. call
   mCamera.setPreviewTexture(getSurfaceTexture());
mCamera.startPreview();
   and you need to delete some old-style code, like
   mCamera.setPreviewDisplay(mHolder);
mCamera.setPreviewCallbackWithBuffer(xxx).

[lucabee]

OK, these steps are pretty clear. Maybe the problem is on Shader management to render SurfaceTexture.

[lucabee]

@gfphoenix I'm sorry but I'm not able to render camera preview using Shader. Could you send me any sample code?
Many thanks

[gfphoenix]

the shader code is:
`

String fragmentShader = "#ifdef GL_ES\n" //
+ "#extension GL_OES_EGL_image_external : require\n"
+ "#define LOWP lowp\n" //
+ "precision mediump float;\n" //
+ "#else\n" //
+ "#define LOWP \n" //
+ "#endif\n" //
+ "varying LOWP vec4 v_color;\n" //
+ "varying vec2 v_texCoords;\n" //
+ "uniform samplerExternalOES u_texture;\n" //
+ "void main()\n"//
+ "{\n" //
+ " gl_FragColor = v_color * texture2D(u_texture, v_texCoords);\n" //
+ "}";

`

Note, there are just two key line different between the above code and default shader of SpriteBatch: "#extension GL_OES_EGL_image_external : require\n" and "uniform samplerExternalOES u_texture;\n"
And you should change the default glTarget of the Texture when you bind to opengles. the code I do is following:
`
public class SurfaceTextureWrapper extends Texture {
private MyTextureDataV1 textureData;
private SurfaceTexture surfaceTexture;
private ShaderProgram shaderProgram;
private final float []v = new float[20];
boolean dataReady;

public SurfaceTextureWrapper(MyTextureDataV1 textureData) {
    super(textureData);
    this.textureData = textureData;
    glTarget = GL_TEXTURE_EXTERNAL_OES;
    this.setFilter(this.minFilter, this.magFilter);
    this.setWrap(this.uWrap, this.vWrap);

    this.surfaceTexture = new SurfaceTexture(glHandle);
    surfaceTexture.setOnFrameAvailableListener(new SurfaceTexture.OnFrameAvailableListener() {
        @Override
        public void onFrameAvailable(SurfaceTexture surfaceTexture) {
            setDataStatus(true);  // just set a flag
            System.out.println("Texture[OnFrameAvailable]");
        }
    });
    initUV();
    setColor(Color.WHITE);
    shaderProgram = new ShaderProgram(vertexShader, fragmentShader);
    if (!shaderProgram.isCompiled()) {
        throw new GdxRuntimeException("compile gdx failed:"+shaderProgram.getLog());
    }
}

public void reload(int unit) {
    glHandle = createGLHandle();
    load(unit);
}
public void load(TextureData data, int unit) {
    this.textureData = (MyTextureDataV1) data;
    load(unit);
}
void load(int unit) {
    if (surfaceTexture!=null) {
        bind(unit); // NOTE:  glTarget(GL_TEXTURE_EXTERNAL_OES) takes effect here
        surfaceTexture.updateTexImage();
    }
}
public void tryUpdate() {
       if (dataReady) {
        load();
        dataReady = false;
         }
}

`

After all these setup, you can use opengles to draw the surface texture to your screen or framebuffer when data is ready:

`

   public void draw(SpriteBatch batch, float parentAlpha) {
    if (config==null) return;

    batch.end();
    batch.setColor(getColor());
    batch.setShader(surfaceTextureWrapper.getShaderProgram());

    surfaceTextureWrapper.tryUpdate();
    if (config.shouldUpdateData()) {
        // drawC image to head buffer for object detection
        draw_square_head_to_buffer(batch);
    }

    // ### drawC camera image to the full screen
    if (config.shouldDrawCameraFullScreen()) {
        draw_background(batch);
    }
    // ### restore batch state
    batch.setShader(null);
    batch.begin();
    batch.setColor(batch.getColor());
}

`

[alfonsobee]

Hi, I have the same problems on Android 8.1.
I'm not an opengl programmer, is there any solution using only libgdx standard objects?

[gfphoenix]

@alfonsobee Sorry, I use a lower version of libgdx(0.9.9) and I don't use standard display object of libgdx. I write my own render object to use component-based design. Camera is a highly system dependent object, and there are lots of control details which you may need, so I don't think libgdx can do it very well to satisfy all needs.

[lucabee]

@EtK2000 did you find anything? How can we have support directly from libgdx dev? many thanks

[alfonsobee]

@gfphoenix Can you share something about your renderer object?

[EtK2000]

I'm having a few small issues with the code, I'll try fixing them and I'll get back to you when I have a proper, working example.

[lukaweb]

Hi, i found the same issue starting from android 8 version. I also updated libgdx version to latest but the issue still remains

[wslany]

Hi, did someone find a solution for this issue with the camera image overlayed over everything else in libgdx on Android 8 and above?

Wouldn't this issue also be show-stopping for augmented reality Android apps that use libgdx, e.g.:

• https://glud.github.io/trascendentAR/index_en.html / @Juankz
• https://github.com/google/helloargdx / @claywilkinson and @boonto

[boonto]

I have not done anything with it for some time, but I just checked and it seems to work on a 29 API emulator.

[wslany]

@boonto Patrick thanks, please could you elaborate what you mean? I tried with the latest libgdx version on an Android 9.0 Samsung Galaxy Note 8, and the camera image was still in front of everything else, despite working fine behind other stuff on older versions of Android. The emulator does not have a camera, right? Or do you mean the augmented reality part worked fine in the emulator? But how could you test augmented reality in the emulator? Sorry for bothering you ("ich stehe auf der Leitung ;-)"), but this is really a crippling bug for us we hope to fix.

[boonto]

The Android simulation has a camera feature with a virtual 3D scene, you can use AR with it.

I also tested it on my Android 10 phone and that worked with the base repository and with all (libGDX, ARCore, Gradle) versions updated.
The camera image is behind the AR objects.

[wslany]

@boonto thanks a lot, that's hopefully great news! I'll check it out. Hopefully, because we need it to work also with 2d applications using the camera. But since we also want to work with augmented reality, that's great for that case.

[mandelbrot2357]

We have finally managed to fix this issue for our project. The solution was one line of code to set the layout view order manually: https://github.com/Catrobat/Catroid/blob/e5370de1095cecf8fefb89d2088997d0a2431903/catroid/src/main/java/org/catrobat/catroid/stage/StageLifeCycleController.java#L111. setZOrderOnTop(true) ensures that the view will always be in front (of the camera preview).

[wslany]

We have finally managed to fix this issue for our project. The solution was one line of code to set the layout view order manually: https://github.com/Catrobat/Catroid/blob/e5370de1095cecf8fefb89d2088997d0a2431903/catroid/src/main/java/org/catrobat/catroid/stage/StageLifeCycleController.java#L111. setZOrderOnTop(true) ensures that the view will always be in front (of the camera preview).

@NathanSweet I think this issue can now be closed, as solutions for both 2d and 3d have been found and described above. Can @mandelbrot2357 's solution be included in libGDX's code? If so, please let us know. Thanks!!

[MrStahlfelge]

Closed, issue solved. @mandelbrot2357 if you think the fix should go into libdx' backend, please provide a PR.