Arcore-android-sdk: How do we use sceneform along with ARCloud anchors?

[rohitagarwal3011]

Every time I try to host an anchor "session.hostCloudAnchor(anchor);" it shows up NotTrackingException. How can we host and resolve the Anchor that we get from arFragment.setOnTapArPlaneListener .

[rohitagarwal3011]

This is the code i am using,

     arFragment.setOnTapArPlaneListener(
    (HitResult hitResult, Plane plane, MotionEvent motionEvent) -> {
      Camera camera = arFragment.getArSceneView().getArFrame().getCamera();
      TrackingState cameraTrackingState = camera.getTrackingState();

      if (andyRenderable == null) {
        return;
      }

      if (plane.getType() != Type.HORIZONTAL_UPWARD_FACING) {
        return;
      }

      if (cameraTrackingState == TrackingState.TRACKING && session!= null) {
        // Create the Anchor.
        anchor = hitResult.createAnchor();
        try{
        session.hostCloudAnchor(anchor);
        }
        catch (NotTrackingException e)
        {
            e.printStackTrace();

        }
        setNewAnchor(anchor);
        appAnchorState = AppAnchorState.HOSTING;
        Toast.makeText(HelloSceneformActivity.this, "Now, hosting anchor", Toast.LENGTH_SHORT)
            .show();

        AnchorNode anchorNode = new AnchorNode(anchor);
        anchorNode.setParent(arFragment.getArSceneView().getScene());

        // Create the transformable andy and add it to the anchor.
        TransformableNode andy = new TransformableNode(arFragment.getTransformationSystem());
        andy.setParent(anchorNode);
        andy.setRenderable(andyRenderable);
        andy.select();

        checkUpdatedAnchor();
      }
    });``

[inio]

Where are you getting session? Are you creating it separately or calling arFragment.getArSceneView().getSession() (or I guess creating it and then using ArSceneView.setupSession())?

[rohitagarwal3011]

I am creating it seperately.

@Override
protected void onResume() {
    super.onResume();
    if(session==null)
    {
        //Toast.makeText(HelloSceneformActivity.this,"Session is null",Toast.LENGTH_SHORT).show();
        try {
            session = new Session(/* context= */ this);
        } catch (UnavailableArcoreNotInstalledException e) {
            e.printStackTrace();
        } catch (UnavailableApkTooOldException e) {
            e.printStackTrace();
        } catch (UnavailableSdkTooOldException e) {
            e.printStackTrace();
        }

        // Create default config and check if supported.
        Config config = new Config(session);
        if (!session.isSupported(config)) {
      //                showSnackbarMessage("This device does not support AR", true);
        }
        config.setCloudAnchorMode(Config.CloudAnchorMode.ENABLED);
        session.configure(config);
    }
    else {
       // Toast.makeText(HelloSceneformActivity.this,"Session is not 
       null",Toast.LENGTH_SHORT).show();
    }

    try {
        session.resume();
    } catch (Exception e) {
        e.printStackTrace();
    }

}

[inio]

I see you're not setting LATEST_CAMERA_IMAGE, which makes me wonder if you're using ArSceneView.setupSession(). I also see a catch-and-forget on session.resume(), which makes me think that method is failing.

You can only have one running Session at a time. If you let Sceneform try to start one yours definitely won't start. You need to give your configured session to sceneform using ArSceneView.setupSession().

[rohitagarwal3011]

Thanks. NotTrackingException is solved now but now when i try to host an anchor these are issues i am facing :

1. The 3D renderable object is not shown on the surface
2. The success or failure for the Hosting operation is not returned
3. I am getting this as error in the logs
   Mapped visual feature update failed when computing its residual and Jacobian.

Below is the whole code of the activity

 public class HelloSceneformActivity extends AppCompatActivity {
 private static final String TAG = HelloSceneformActivity.class.getSimpleName();

private ArFragment arFragment;
 private ModelRenderable andyRenderable;
private Session session;
private Anchor anchor;
// Lock needed for synchronization.
private final Object singleTapAnchorLock = new Object();
private TrackingState trackingState;
private Frame frame;

 @Override
 @SuppressWarnings({"AndroidApiChecker", "FutureReturnValueIgnored"})
 // CompletableFuture requires api level 24
 // FutureReturnValueIgnored is not valid
 protected void onCreate(Bundle savedInstanceState) {
  super.onCreate(savedInstanceState);

 setContentView(R.layout.activity_ux);

arFragment = (ArFragment) getSupportFragmentManager().findFragmentById(R.id.ux_fragment);

// When you build a Renderable, Sceneform loads its resources in the background while returning
// a CompletableFuture. Call thenAccept(), handle(), or check isDone() before calling get().
ModelRenderable.builder()
    .setSource(this, R.raw.andy)
    .build()
    .thenAccept(renderable -> andyRenderable = renderable)
    .exceptionally(
        throwable -> {
          Toast toast =
              Toast.makeText(this, "Unable to load andy renderable", Toast.LENGTH_LONG);
          toast.setGravity(Gravity.CENTER, 0, 0);
          toast.show();
          return null;
        });

arFragment.setOnTapArPlaneListener(
    (HitResult hitResult, Plane plane, MotionEvent motionEvent) -> {
      Camera camera = arFragment.getArSceneView().getArFrame().getCamera();
      TrackingState cameraTrackingState = camera.getTrackingState();

      if (andyRenderable == null) {
        return;
      }

      if (plane.getType() != Type.HORIZONTAL_UPWARD_FACING) {
        return;
      }

      if (cameraTrackingState == TrackingState.TRACKING && session!= null) {
        // Create the Anchor.
        anchor = hitResult.createAnchor();
        anchor = session.hostCloudAnchor(anchor);

        setNewAnchor(anchor);
        appAnchorState = AppAnchorState.HOSTING;
        Toast.makeText(HelloSceneformActivity.this, "Now, hosting anchor", Toast.LENGTH_SHORT)
            .show();

        AnchorNode anchorNode = new AnchorNode(anchor);
        anchorNode.setParent(arFragment.getArSceneView().getScene());

        // Create the transformable andy and add it to the anchor.
        TransformableNode andy = new TransformableNode(arFragment.getTransformationSystem());
        andy.setParent(anchorNode);
        andy.setRenderable(andyRenderable);
        andy.select();

          checkUpdatedAnchor();
      }
    });
   }

private Snackbar messageSnackbar;

@Override
protected void onResume() {
    super.onResume();
    if(session==null)
    {
        //Toast.makeText(HelloSceneformActivity.this,"Session is null",Toast.LENGTH_SHORT).show();
        try {
            session = new Session(/* context= */ this);
        } catch (UnavailableArcoreNotInstalledException e) {
            e.printStackTrace();
        } catch (UnavailableApkTooOldException e) {
            e.printStackTrace();
        } catch (UnavailableSdkTooOldException e) {
            e.printStackTrace();
        }

        // Create default config and check if supported.
        Config config = new Config(session);
        if (!session.isSupported(config)) {
     //                showSnackbarMessage("This device does not support AR", true);
        }
        config.setCloudAnchorMode(Config.CloudAnchorMode.ENABLED);
        config.setUpdateMode(Config.UpdateMode.LATEST_CAMERA_IMAGE);
        session.configure(config);
    }

    arFragment.getArSceneView().setupSession(session);
}

@Override
protected void onPause() {
    super.onPause();
    if(session!=null)
    {
        session.pause();
    }
}

private enum AppAnchorState {
    NONE,
    HOSTING,
    HOSTED
}

@GuardedBy("singleTapAnchorLock")
private AppAnchorState appAnchorState = AppAnchorState.NONE;

private void checkUpdatedAnchor() {
    synchronized (singleTapAnchorLock) {
        if (appAnchorState != AppAnchorState.HOSTING) {
            Toast.makeText(HelloSceneformActivity.this, "Hosting Canceled", Toast.LENGTH_SHORT)
                    .show();
            return;
        }
        Anchor.CloudAnchorState cloudState = anchor.getCloudAnchorState();
        if (cloudState.isError()) {
            Toast.makeText(this, "Error hosting anchor: " + cloudState,Toast.LENGTH_SHORT).show();
            appAnchorState = AppAnchorState.NONE;
        } else if (cloudState == Anchor.CloudAnchorState.SUCCESS) {
            Toast.makeText(
                    this, "Anchor hosted successfully! Cloud ID: " + anchor.getCloudAnchorId(),Toast.LENGTH_SHORT).show();
            appAnchorState = AppAnchorState.HOSTED;
        }
    }
}

/** Sets the new anchor in the scene. */
@GuardedBy("singleTapAnchorLock")
private void setNewAnchor(@Nullable Anchor newAnchor) {
    if (anchor != null) {
        anchor.detach();
    }
    anchor = newAnchor;
    appAnchorState = AppAnchorState.NONE;
}


}

[claywilkinson]

It looks like there is some code missing so I am not sure what's the problem. I was able to successfully convert the sample to use Sceneform in my personal fork. Feel free to compare your code to it.

I'll work on getting the converted sample added to the Sceneform SDK....

[rohitagarwal3011]

Thanks @claywilkinson for the code. Its working pretty well but while resolving the anchors i am facing issues in this line
anchorNode.getAnchor().detach();

in the function

private void setNewAnchor(Anchor newAnchor) {

    AnchorNode newAnchorNode = new AnchorNode(newAnchor);

    if (anchorNode != null) {
      anchorNode.getAnchor().detach();
      for (Node node : anchorNode.getChildren()) {
        node.setParent(newAnchorNode);
      }
      anchorNode.setParent(null);
    } else {
      Node andy = new Node();
      andy.setRenderable(andyRenderable);
      andy.setParent(newAnchorNode);
    }
    anchorNode = newAnchorNode;
    anchorNode.setParent(arFragment.getArSceneView().getScene());
  }   

It says
Attempt to invoke virtual method 'void com.google.ar.core.Anchor.detach()' on a null object reference

And it crashes the app.

[rohitagarwal3011]

And also in this function

synchronized void resolveCloudAnchor(String anchorId, CloudAnchorListener listener) {
    Preconditions.checkNotNull(session, "The session cannot be null.");
    Anchor newAnchor = session.resolveCloudAnchor(anchorId);
    pendingAnchors.put(newAnchor, listener);
  }

It throws a null pointer exception for session.

[claywilkinson]

The problem was not handling setNewAnchor(null) correctly. This is called when switching modes or cancelling. Update setNewAnchor to

  private void setNewAnchor(Anchor newAnchor) {
   AnchorNode newAnchorNode;

    if (newAnchor != null) {
      newAnchorNode = new AnchorNode(newAnchor);
      newAnchorNode.setParent(arFragment.getArSceneView().getScene());
    } else {
      newAnchorNode = null;
    }

    if (anchorNode != null) {
      anchorNode.getAnchor().detach();
      for (Node node : anchorNode.getChildren()) {
        node.setParent(newAnchorNode);
      }
      anchorNode.setParent(null);
    } else if (newAnchorNode != null) {
      Node andy = new Node();
      andy.setRenderable(andyRenderable);
      andy.setParent(newAnchorNode);
    }
    anchorNode = newAnchorNode;
  }

As for the resolveCloudAnchor() and the session being null, I'll have to look more closely, but it sounds like the session is not setup yet for some reason...

[rohitagarwal3011]

Okay.Thanks for the correction.
Is there are a way that we can attach multiple objects to a single anchor?

[claywilkinson]

Yes, you can create any number of Nodes and set the parent to the anchor node. The Sceneform scene is a tree, the root is the Scene object, and can have multiple child nodes.

[rohitagarwal3011]

Please correct me if I am wrong,
Scene object is the parent to an AnchorNode and we can add multiple nodes to an AnchorNode.
And then we can host the Anchor attached to the AnchorNode and also resolve all the nodes using that single Anchor.

[claywilkinson]

If i understand what you are asking, is that you want to have multiple objects per anchor, and have the same view for the host and the resolver? I think for that you need to share more information over between the devices.
Cloud Anchors only share the pose of the of the anchor. This means the position and rotation of the anchorNode is shared.
The other nodes that are children of the Anchor node also have a position and rotation relative to the anchor. That information is not shared by the ARCore API. You would have to write the code to share that independently. You can use Anchor.getCloudAnchorId() to get a key for the anchor to associate your additional data.

[rohitagarwal3011]

Okay. So you mean to say if store the Pose of all the child nodes and then retrieve them then all the objects attached to that AnchorNode can be resolved?
Does ARCore provide the necessary information for each node that they can be stored and restored across devices?

[claywilkinson]

you can get the rotation and position of all the nodes. It is a little more involved that can be easily answered here. You could look at the Just A Line demo app, specifically how the FirebaseAnchorData class is used.

[rohitagarwal3011]

Okay thanks. I will definitely go through it and get back to you in case of any help.
Just one last doubt,
Is there a way in which we can store the whole information of AnchorNode and other nodes in our own database instead of CloudAnchor especially because of 24 hours limitation?

[claywilkinson]

At this time there is no way to privately store cloud anchors.