Mediapipe: Arcore Sceneform dependencies in bazel

[suraj-0387]

How to add arcore sceneform dependencies in bazel mediapipe handtrackingproject

[jiuqiant]

If you plan to do your project with Android Studio, you probably want to take a look at the Mediapipe AAR instruction. MediaPipe hand tracking AAR can be imported into Android Studio. Then, you need to follow the ARCore instruction to add the dependency in Android Studio.

If you plan to do your project with Bazel, please take a look at the WORKSPACE file. We put several maven dependencies in the WORKSPACE file and you probably can do something similar to the sceneform lib.

[HebaMohamed]

Hello @jiuqiant thank you for the detailed steps but actually I already implemented them but failed to run arcore scene form with mediapipe ,
That due to the GL context used in the mediapipe surfacetexture and in arcore scene ,
Any idea how to properly integrate them please ?

[suraj-0387]

Hello @HebaMohamed Can you list down the dependencies that you have added.Also in gradle there is "apply plugin" of sceneform which we need to add.Do you have any idea about it as where to add it in bazel

[jiuqiant]

@mcclanahoochie, can you take a look at the GL context related question in https://github.com/google/mediapipe/issues/313#issuecomment-564343939? Thanks.

[mcclanahoochie]

Hi,
The goal here is to setup glcontext sharing.

You can set mediapipe to use an external EGLContext context (extracted from somewhere such as ARCore) in this way:

::mediapipe::Status SetExternalGlContextForGraph(
    CalculatorGraph* graph, EGLContext external_context) {
   auto gpu_resources = GpuResources::Create(external_context);
  return graph->SetGpuResources(std::move(gpu_resources));
}

The idea is you call this function with an existing EGLContext right after creating your graph.

A similar thing can be seen in the desktop webcam gpu demo when initializing the gpu

In java, there is SetParentGlContext

[suraj-0387]

I integrated mediapipe with arcore(sceneform) and it also displayed arcore plane detection and along with the mediapipe hand tracking but it failed after sometime.When I try to open the app for the second time it gives me following error and sometimes it does not display camera image but renders black and white surface.

    Process: com.example.handtrackinggradle, PID: 23620
     java.lang.RuntimeException: Error during attachToGLContext (see logcat for details)
    at android.graphics.SurfaceTexture.attachToGLContext(SurfaceTexture.java:289)

[mcclanahoochie]

Sounds like good progress.
Can you provide more logcat details?
How is the graph being handled in the app lifecycle? (i.e OnOpen, OnResume, OnPause,... etc.).
If the graph is ever closed or destroyed, the GL context setting needs to happen each time before running the graph again. Likewise to make sure the ARCore context is valid after resuming app.

[suraj-0387]

There are not big changes added to the mediapipe source code except for some few modifications which I will list down below.Currently we are using Samsung Galaxy A30 device to test the changes
added to mediapipe code

Below are the changes added to the mediapipe code
1) Changes to xml layout - Removed the FrameLayout from the activity_main xml file and added
arcore fragment instead.
2) SurfaceView which was created in a default mediapipe handtracking code was removed.
private SurfaceView previewDisplayView; //Removed
3) Surface holder callback which was added to previewDisplayView in mediapipe code is removed
and now it is added to ArsceneView of Arfragment.
4) Camera permissions for arcore in the manifest file is removed so that it takes the permission from
mediapipe code.
Just removed permission code line

I have also mentioned the issue in detail in the below link
https://github.com/google/mediapipe/issues/321

Logcat
Process: com.example.handtrackinggradle, PID: 23620
java.lang.RuntimeException: Error during attachToGLContext (see logcat for details)
at android.graphics.SurfaceTexture.attachToGLContext(SurfaceTexture.java:289)
at
com.google.mediapipe.components.ExternalTextureConverter$RenderThread.setSurfaceTextureAndAttachToGLContext(ExternalTextureConverter.java:187)
at com.google.mediapipe.components.ExternalTextureConverter.lambda$setSurfaceTextureAndAttachToGLContext$1$ExternalTextureConverter(ExternalTextureConverter.java:111)
at com.google.mediapipe.components.-$$Lambda$ExternalTextureConverter$5RBq2uHoPl06wf2nG84kcgcgP_U.run(Unknown Source:8)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:214)
at com.google.mediapipe.glutil.GlThread.run(GlThread.java:134)
2019-12-13 04:44:59.014 23620-23718/com.example.handtrackinggradle E/native: loge.h:15 DataSourceMetrics: kFirstGlCallback: 567.448154ms
2019-12-13 04:44:59.016 23620-23719/com.example.handtrackinggradle E/native: loge.h:15 CameraMetrics: FirstCaptureComplete: 440.307692ms

[HebaMohamed]

@suraj-0387 I just integrated the aar file and then added sceneform gradle dependancies as usual as illustrated here
and I got the same GLContext error too

I hope someone can help us

[mcclanahoochie]

as mentioned in https://github.com/google/mediapipe/issues/313#issuecomment-564726162

can you check you are correctly providing the arcore egl context to mediapipe (and not the default one)?

ex:

    processor =
        new FrameProcessor(
            this,
            <arcore egl context>,
            ...,
            ...

or

    eglManager = new EglManager(<arcore egl context>);
    processor =
        new FrameProcessor(
            this,
            eglManager.getNativeContext(),
           ...

[HebaMohamed]

@mcclanahoochie Thank you so much for you help, but there is no direct method to get sceneform gl context, so tried to get the current context after sceneform initialization like that
eglManager = new EglManager(((EGL10) EGLContext.getEGL()).eglGetCurrentContext());

and got that error in converter.setSurfaceTextureAndAttachToGLContext(...); and when trying to use converter.setSurfaceTexture() only, the error is gone but mediapipe will stop without any results received

E/GLConsumer: [SurfaceTexture-0-23241-0] attachToContext: GLConsumer is already attached to a context E/AndroidRuntime: FATAL EXCEPTION: ExternalTextureConverter Process: com.ml34.sceneformstestapp, PID: 23241 java.lang.RuntimeException: Error during attachToGLContext (see logcat for details) at android.graphics.SurfaceTexture.attachToGLContext(SurfaceTexture.java:289) at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.setSurfaceTextureAndAttachToGLContext(ExternalTextureConverter.java:187)

[jiuqiant]

Seems like you already can add ARCore Sceneform as a dependency in MediaPipe's WORKSPACE file. Close this issue for now and the discussion can be continued in https://github.com/google/mediapipe/issues/321 or https://github.com/google/mediapipe/issues/323

[suraj-0387]

@HebaMohamed Any update from your end

[brianm-sra]

as mentioned in #313 (comment)

        <arcore egl context>,

How can we get the EGLContext from ARCore, though ??

[boehm-e]

@suraj-0387, could you please provide a sample app where SceneForm and Mediapipe hand tracking coexist ?
Thank you :)

[mohdahk]

There are not big changes added to the mediapipe source code except for some few modifications which I will list down below.Currently we are using Samsung Galaxy A30 device to test the changes
added to mediapipe code

Below are the changes added to the mediapipe code

1. Changes to xml layout - Removed the FrameLayout from the activity_main xml file and added
   arcore fragment instead.
2. SurfaceView which was created in a default mediapipe handtracking code was removed.
   private SurfaceView previewDisplayView; //Removed
3. Surface holder callback which was added to previewDisplayView in mediapipe code is removed
   and now it is added to ArsceneView of Arfragment.
4. Camera permissions for arcore in the manifest file is removed so that it takes the permission from
   mediapipe code.
   Just removed permission code line

I have also mentioned the issue in detail in the below link

321

Logcat
Process: com.example.handtrackinggradle, PID: 23620
java.lang.RuntimeException: Error during attachToGLContext (see logcat for details)
at android.graphics.SurfaceTexture.attachToGLContext(SurfaceTexture.java:289)
at
com.google.mediapipe.components.ExternalTextureConverter$RenderThread.setSurfaceTextureAndAttachToGLContext(ExternalTextureConverter.java:187)
at com.google.mediapipe.components.ExternalTextureConverter.lambda$setSurfaceTextureAndAttachToGLContext$1$ExternalTextureConverter(ExternalTextureConverter.java:111)
at com.google.mediapipe.components.-$$Lambda$ExternalTextureConverter$5RBq2uHoPl06wf2nG84kcgcgP_U.run(Unknown Source:8)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:214)
at com.google.mediapipe.glutil.GlThread.run(GlThread.java:134)
2019-12-13 04:44:59.014 23620-23718/com.example.handtrackinggradle E/native: loge.h:15 DataSourceMetrics: kFirstGlCallback: 567.448154ms
2019-12-13 04:44:59.016 23620-23719/com.example.handtrackinggradle E/native: loge.h:15 CameraMetrics: FirstCaptureComplete: 440.307692ms

There are not big changes added to the mediapipe source code except for some few modifications which I will list down below.Currently we are using Samsung Galaxy A30 device to test the changes
added to mediapipe code

Below are the changes added to the mediapipe code

1. Changes to xml layout - Removed the FrameLayout from the activity_main xml file and added
   arcore fragment instead.
2. SurfaceView which was created in a default mediapipe handtracking code was removed.
   private SurfaceView previewDisplayView; //Removed
3. Surface holder callback which was added to previewDisplayView in mediapipe code is removed
   and now it is added to ArsceneView of Arfragment.
4. Camera permissions for arcore in the manifest file is removed so that it takes the permission from
   mediapipe code.
   Just removed permission code line

I have also mentioned the issue in detail in the below link

321

Logcat
Process: com.example.handtrackinggradle, PID: 23620
java.lang.RuntimeException: Error during attachToGLContext (see logcat for details)
at android.graphics.SurfaceTexture.attachToGLContext(SurfaceTexture.java:289)
at
com.google.mediapipe.components.ExternalTextureConverter$RenderThread.setSurfaceTextureAndAttachToGLContext(ExternalTextureConverter.java:187)
at com.google.mediapipe.components.ExternalTextureConverter.lambda$setSurfaceTextureAndAttachToGLContext$1$ExternalTextureConverter(ExternalTextureConverter.java:111)
at com.google.mediapipe.components.-$$Lambda$ExternalTextureConverter$5RBq2uHoPl06wf2nG84kcgcgP_U.run(Unknown Source:8)
at android.os.Handler.handleCallback(Handler.java:873)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loop(Looper.java:214)
at com.google.mediapipe.glutil.GlThread.run(GlThread.java:134)
2019-12-13 04:44:59.014 23620-23718/com.example.handtrackinggradle E/native: loge.h:15 DataSourceMetrics: kFirstGlCallback: 567.448154ms
2019-12-13 04:44:59.016 23620-23719/com.example.handtrackinggradle E/native: loge.h:15 CameraMetrics: FirstCaptureComplete: 440.307692ms

Hi @suraj-0387 Thanks for the description. i am looking for the same wherein i want to add SurfaceView to AR's Sceneview. Is it possible for you to share the code for the same ?