Ar.js: Marker based and Location based elements does not work together

[umutto]

Describe the bug
When using geo based and marker based AR in the same scene only either one of them work. Problem seems to be the camera entity or I am missing something.

When <a-entity camera/> is used as advised on arjs docs for multiple marker support, geo based elements are not displayed. <a-entity camera gps-camera rotation-reader/> does not work either.

When <a-camera gps-camera rotation-reader/> is used as advised on geoarjs docs, then barcode or pattern marker is found but the overlaying a-element can't be seen (I was able to see it at some weird lucky angles, maybe the problem is due to camera being moved?). Acts as same when <a-camera camera gps-camera rotation-reader/> is used or when 2 different camera elements used as <a-camera camera gps-camera rotation-reader/><a-entity camera/>

To Reproduce
I have prepared a simple example to reproduce, change the camera elements to see different cases:

````html

    <a-marker preset='hiro'>
        <a-sphere position='0 0.5 0' material='opacity: 0.5; side: double;color:blue;'>
        </a-sphere>
    </a-marker>

    <!-- Location based elements are displayed, hiro marker is found but element is not displayed -->
    <!-- <a-camera  gps-camera rotation-reader/> -->

    <!-- Location based elements are displayed, hiro marker is found but element is not displayed -->
    <!-- <a-camera camera gps-camera rotation-reader/>
    <a-entity camera/> -->

    <!-- No gps based elements displayed, hiro marker is found and it's element is displayed -->
    <a-entity camera gps-camera rotation-reader/>
</a-scene>

````

Expected behavior
As my understanding of ARjs 2.0, both cases should be able to work at tandem. When a marker is read, it should display an element and when the device orientation changes to a nearby geo-entity-place it should display it's element. (sphere and a box on the example code)

Desktop (please complete the following information):

• OS: Windows 10
• Browser Chrome
• Version 79.0.3945.117

Smartphone (please complete the following information):

• Device: Pixel 3a
• OS: Android 10
• Browser: Chrome for Android
• Version: 79.0.3945.116

Thank you very much!

[nicolocarpignoli]

Hi,
thanks, that's actually a bug.

I gave a look, and two problems turned out:

1) a-camera and a-entity camera cannot be used together (so have two of them), and we have to find a way to make a-camera behave like a-entity camera (or vice versa)

2) By removing this piece of code, location-based feature breaks. To not have it, in some cases (e.g. local machine) breaks marker-based:

 <script>
        THREEx.ArToolkitContext.baseURL = 'https://raw.githack.com/jeromeetienne/ar.js/master/three.js/'
    </script>

[nicolocarpignoli]

@umutto feel free to give it a look and to open a PR on this :)

[umutto]

I would like to but after looking into it I'm a bit lost. I'm not very familiar with three.js and 3D calculations so I don't think I would be a great help.

Saying that I think the reason is in marker based recognition, camera is set to static (in <0, 0, 0>), never moves and marker object positions are updated accordingly. This is also written in the documentation here (a-marker-camera).

Now, there is a changeMatrixMode parameter that is set to modelViewMatrix as default (where camera is static). But changing it to cameraTransformMatrix (camera is movable but markers are static) breaks it.

So I've looked into offsetting the markers to kinda move together when gps-camera moves by adding the following lines here.

js if (document.querySelector('[gps-camera]')) { var cameraPosition = document.querySelector('[gps-camera]').object3D.position; markerObject3D.position.add(cameraPosition) }

which fixes the positioning, but due to my inexperience in 3D calculations I couldn't offset the camera rotation that happen here. Also this kinda felt a bit hackish way.

I think we should adapt geoAR.js to static camera or fix the cameraTransformMatrix code and adapt AR.js marker tracking to movable camera (latter sounds more clean imho). But doing which is above my experience.

[umutto]

Actually now that I had a second look, setting changeMatrixMode to cameraTransformMatrix does not break marker recognition (only some parts does not work but main function works, broken parts are markers object3d stays visible, it flickers, does not emit events etc..). As expected marker stays static and camera moves. But camera's position doesn't change as I assumed but rather camera.getWorldDirection() changes (now that I think of it is obvious.)

I'm not sure how to combine compass heading and this camera rotation but solution seems to be that. Sadly I won't have much time to work on this couple of weeks, but I'll try to take a look after that.

[nicolocarpignoli]

Thank you. I will be busy for nft development of arjs, so our best bet for now is you.😊

[andypotato]

Saying that I think the reason is in marker based recognition, camera is set to static (in <0, 0, 0>), never moves and marker object positions are updated accordingly. This is also written in the documentation here (a-marker-camera).

I'd also love to see this feature implemented, so I had a quick look at the source code. In my opinion with the current implementation this is unfortunately impossible. As @umutto points out, the camera behavior in both "modes" is fundamentally different.

• In "Location" mode a 3D scene with fixed position elements is rendered and the user moves the camera through the scene. The component is also checking for the look-controls attribute of the attached object and will only init on an <a-camera> but not on an <a-entity camera>.

• In "Marker" mode the camera position is fixed but instead the objects are moved, rotated and scaled to align with the marker position relative to the user position. This is also why multiple markers can work as the objects are moved around the fixed position camera.

The approach I would suggest is to render both Three.js scenes and then overlay them on the same canvas. Means there are two active cameras, one handling markers, one handling location based objects. Both renderers then render into the same canvas resulting in an overlayed picture.

@nicolocarpignoli @umutto Thoughts? Comments?

[nicolocarpignoli]

@andypotato it's a brilliant idea. Maybe the only that can work without have to rewrite one of the two features.

[umutto]

@andypotato Yeah good observation and idea! The two features are fairly separated, I don't think they use any common code so it should be most straight forward / quick way to implement it. I doubt it but wonder if that would effect the performance though.

Sadly I will not have time to check it out until next month, but would gladly help/test after then.

[nicolocarpignoli]

added on new repo, we can continue there: https://github.com/AR-js-org/AR.js/issues/29