Model-viewer: Marker-based AR

[dakom]

The AR capabilities are great but require special instructions and aren't widely available out of the box.

Would be nice to have an option for supplying marker-based AR which doesn't require anything other than getUserMedia() (works on most modern browsers I believe).

Perhaps using OpenCV with the Aruco module enabled? At a glance it looks to me like this would work, though I'm not sure if camera calibration is necessary or if it can somehow be gotten by looking up runtime info available to the browser, in a database that can give back camera stats?

[cdata]

We have not really talked about it as a team yet, but speaking for myself I like this idea a lot.

I have seen a couple of libraries over the years that help with marker-based AR, both high level and low. What is the state of the art?

[dakom]

Cool! Personally I've only used Vuforia in Unity - but I'll have a go at tinkering with OpenCV+Aruco in the next night or two and report here if I get anything going in JS/WASM :D

Not sure where it fits on the current state of the art - I'm curious about that too since approaches I found by searching for this took the same route it seems but haven't been updated in around a year or so

[cdata]

@dakom FWIW this is the most recent high-level library I am aware of that offers marker-based tracking: https://github.com/jeromeetienne/AR.js

[mikkoh]

Based on playing around with AR.js last week I think AR.js is actually based on this Emscripten build.

https://github.com/artoolkit/jsartoolkit5

To keep this lib as light as possible likely you'd want to load a feature like this on the fly not bundled into the actual JS cause Emscripten builds are MASSIVE.

From what I can see AR.js just gives you an abstraction on top of jsartoolkit5 + Three.JS. Although it looks like jsartoolkit5 Also has it's own abstraction so 🤷‍♂️

[dakom]

I got opencv.js to compile with aruco, and there's a more stripped down / updated demo found here which I think already has a build: https://github.com/ganwenyao/opencv_js/tree/master/docs

Though the details of all the opencv settings are beyond me, haven't used it before :\

Also I ran into trouble trying to import the opencv.js via webpack... seems like it really wants to be included as a global o_O

[dakom]

FYI I got something basically working, here's the repo: https://github.com/dakom/web-menorah

Feel free to grab the opencv* builds from the site/ folder if it helps. I just added aruco to the build from the repo mentioned above. Also "borrowed" the camera and parameter settings from there.

There's still a bug and someone tells me it's not working on iOS, but it's working pretty well as a proof of concept for me with desktop and android.

The long wait time to init the wasm is kinda rough, and I'm not sure how camera calibration should fit into a normal flow... but it's a start :)

[mithru]

Is this in the pipeline? Any idea when we can expect this? This is currently possible with newer iOS devices if .reality files are used. Would be super cool to have it work consistently across platforms.

[cdata]

@mithru that's interesting, I didn't know that iOS had added marker support. Do you have any references to docs or videos on this?

[mithru]

Sorry @cdata . I missed your last comment.

see the video at 03:54
https://developer.apple.com/videos/play/wwdc2019/609/

The bottom left option is "Image". Using reality files on ios-src seems to allow a fair bit of fun on iOS.

I made a few experiences like this and can quickly set up a demo model viewer page with this if needed.