Arfoundation-samples: ARFoundation Image Tracking trackingState bug is back

[cbanwell]

It seems the bug regards the trackingState of an image never getting set to trackinState.None when the image is moved from the camera's view is back at least for Unity 2019.2.0f1, and this time it doesn't work on either Android orIOS.

It was reported as fixed for 2019.1.1 and preview 2, but is back and worse with 2019.2.0f1

[anhkhoa45]

I'm facing the same problem.

Unity: 2019.2.0f1

ARFoundation: preview3 2.2.0

ARKit XR plugin: preview4 2.2.0

[cbanwell]

Also not working correctly with Arfoundation 2.1 and Unity 2019.1.16f1
Using :
AR Foundation 2.1.3
ARCore XR Plugin 2.1.1
ARKit XR Plugin 2.1.1

Image gets tracked but is never added to the eventArgs.removed list, nor does it's tracking state ever changed to None, it only ever gets tracked and limited.

I've pretty much tried all suggestions regards package versions in the other threads pertaining to this bug and although they give differing results, none of them act correctly.

It should be pretty straightforward: If the camera can't see the image anymore then it's trackingState should equal None, not Limited or Tracked. or at least should be added to the removed list - neither of which happen.

I can sort of get the required functionality if I trap the limited state and disable the spawned gameobject when that is true, but this means the user must have the camera square on to the image and the lighting/image conditions must be 100% perfect. In the real world this isn't the case!!

[tdmowrer]

The method of reporting the tracking state for images was changed in ARCore XR Plugin 2.2.0-preview.2. The only mention of that package in this thread refers to version 2.1.1. Try updating to 2.2.0-preview.2.

[dorkbot]

I don't think there is a Unity 2019.1.16f1.

I'm using:
Unity 2019.1.14f1
AR Foundation 2.1.3
ARCore XR Plugin 2.2.0 preview.2
ARKit XR Plugin 2.1.1

And image tracking works perfectly. My app is in Google Play and the Apple App Store. You can see them here -> http://www.superherorobot.com/projects/crispr-3d
*to turn on image tracking, go into "Settings" (during planar AR mode) and turn on "Card Mode".

[dorkbot]

please note that these versions are not in the current live version of my app. I have a version that will be live in the next 24 hours, version 1.5.3, this version has the below tools:
Unity 2019.1.14f1
AR Foundation 2.1.3
ARCore XR Plugin 2.2.0 preview.2
ARKit XR Plugin 2.1.1

[cbanwell]

Sadly I'm not worried about the ARCore functionality it's the ARKit functionality I need and that also doesn't work as intended. TrackingState.None is never available, only Tracking and Limited.

And sorry I meant Unity version 2019.1.6f1 not 16f1.

As I described before the desired functionality can be achieved, if you disable the spawned gameobject when trackingState is limited, but this causes problems without perfect image and lighting conditions, especially on older iOS devices. As the state will change to limited quite often even when the device has a perfect view of the image to be tracked.

[cbanwell]

I've just run a test using the suggested AR Foundation 2.1.3
and ARCore XR Plugin 2.2.0 preview.2 on an android phone, and the same is happening, trackingState.None is never set, so it seems it's still a problem even on android.

I've just read the release notes for ARCore XR Plugin 2.2.0 preview.2, and it states that trackingState will change to trackingState.Limited even when the image can't be seen. So it seems we are stuck with this functionality for now.

So for anybody reading this, as it stands at the present time, there is no support on either platform for trackingState.None.

[3d03]

devs, any update for this issue?

[simar88]

Also on iOS i found this issue today. The TrackingState.None is never setted.
Any news about it?

[tdmowrer]

I've just run a test using the suggested AR Foundation 2.1.3
and ARCore XR Plugin 2.2.0 preview.2 on an android phone, and the same is happening, trackingState.None is never set, so it seems it's still a problem even on android.

I've just read the release notes for ARCore XR Plugin 2.2.0 preview.2, and it states that trackingState will change to trackingState.Limited even when the image can't be seen. So it seems we are stuck with this functionality for now.

So for anybody reading this, as it stands at the present time, there is no support on either platform for trackingState.None.

This is how ARCore's image tracking states map to ARFoundation's:
AR_AUGMENTED_IMAGE_TRACKING_METHOD_NOT_TRACKING => TrackingState.None
AR_AUGMENTED_IMAGE_TRACKING_METHOD_LAST_KNOWN_POSE => TrackingState.Limited
AR_AUGMENTED_IMAGE_TRACKING_METHOD_FULL_TRACKING => TrackingState.Tracking

Under what conditions would you expect "none" vs "limited"?

[FDDdan]

I have this problem too, I'm working on a custom script that is trying to replicate the same behavior as Vuforia, using separate prefabs for each image and disabling prefab when the image is out of sight.

On ARCore it is managable and we can use the Limited trackingstate the same as None, because Tracking is called only when the image is visible and Limited only when it is not visible. But on ARKit it's completely broken from our experience, Tracking is called only on the first frame when the image is detected and then it it switched to Limited and it stays that way even if the image is visible or not.

UPDATE: Setting "Max Number Of Moving Targets" more than 0 fixed it. It seems that AR Foundation isn't checking if the image is Tracking or not if set to 0, it just instantiate it and set the tracking state to Limited, setting to Tracking only on the frame when it's updating it's position, I think it is behaving similar to a regular reference point.

Using:
Unity 2019.2
AR Foundation 2.1.3
ARCore 2.2.0 preview 2
ARKit 2.1.1

[macklinb]

@FDDdan Thanks for this! I was having the exact same problem. On ARKit, the TrackingState is always Limited as soon as the image is detected, only switching to Tracking at intermittent, seemingly random intervals (and only for a frame or two). My app hides the content when not Tracking, and I was very confused as to why the content was flickering on and off.

As you suggested, setting ARTrackedImageManager's "Max Number of Moving Targets" to more than 0 fixes this. I don't need to track moving images in my app, but it seems like this is the only workaround.

I'm using:

• Unity 2019.2.10f1 (on Mac)
• AR Foundation / AR Subsystems / ARCore XR Plugin / ARKit XR Plugin (all v3.0.0 preview-4)
• iPad 6th gen (AR7F2X/A) - if that matters

[rvegele]

Hi, same issue here, getting 'Tracking' or 'Limited' but never 'None'

[macklinb]

As far as I know (and as mentioned above), the trackingState is never set to TrackingState.None as ARKit/ARCore still has knowledge about the trackable's approximate location using sensor data relative to the last known position, even if it is not being visually tracked. In this case, the state is set to TrackingState.Limited. This can be seen when tracking an image, then covering the camera - any visuals created will persist and adjust to rotations and slight movements of the device (to a certain degree), as the session tries its hardest to maintain the pose. The only time that the trackingState is set to TrackingState.None and ARTrackedImagesChangedEventArgs.removed is filled is when the session is destroyed or the app is closed (perhaps also if the device goes to sleep). There might be other instances, but in my experience none of these occur during a regular session.

Right now, there's no real harm in designing with this in mind. TrackingState.Tracked is when the image is within view and is being visually tracked with confidence, and TrackingState.Limited is when the image cannot be visually tracked confidently, and is relying only on sensor data to keep the pose updated (which it can seemingly do endlessly). Using the ARTrackedImageManager, the ARTrackedImage does not exist before the image is first tracked, and its trackingState is set to TrackingState.Tracked on the same frame it is created, so there's never really an instance where None is used. In testing, without looking at the image again to regain visual tracking, nothing I did caused the trackingState to switch to TrackingState.None, regardless of the ARSession.notTrackingReason.

Your use case should decide what to do when tracking is Limited, but for the most part you should consider TrackingState.Limited as "not being visually tracked". You could add a timer to see how long this state persists, or a check to see if the estimated position is off screen in order to consider the image as "untracked", even though the session will continue to look for it and update its pose. Perhaps if there was some threshold or confidence level of the sensor data relative to the last pose, the state could then be set to None if it is really off the mark - but I'm pretty sure ARCore nor ARKit report anything like this.

The confusion behind these three states is down to having a generic API where one subsystem might support 3 tracking states (ARCore) and another might only support 2 (ARKit). I haven't tested it much, but I believe that ARCore also uses None in a similar sense, and as long as it functions similarly, there shouldn't be any inconsistencies in the way that an app should be designed between platforms.

I really think that this issue is by design, and if so perhaps the documentation should be updated to clarify this?

[rvegele]

As far as I know (and as mentioned above), the trackingState is never set to TrackingState.None as ARKit/ARCore still has knowledge about the trackable's approximate location using sensor data relative to the last known position, even if it is not being visually tracked. In this case, the state is set to TrackingState.Limited. This can be seen when tracking an image, then covering the camera - any visuals created will persist and adjust to rotations and slight movements of the device (to a certain degree), as the session tries its hardest to maintain the pose. The only time that the trackingState is set to TrackingState.None and ARTrackedImagesChangedEventArgs.removed is filled is when the session is destroyed or the app is closed (perhaps also if the device goes to sleep). There might be other instances, but in my experience none of these occur during a regular session.

Right now, there's no real harm in designing with this in mind. TrackingState.Tracked is when the image is within view and is being visually tracked with confidence, and TrackingState.Limited is when the image cannot be visually tracked confidently, and is relying only on sensor data to keep the pose updated (which it can seemingly do endlessly). Using the ARTrackedImageManager, the ARTrackedImage does not exist before the image is first tracked, and its trackingState is set to TrackingState.Tracked on the same frame it is created, so there's never really an instance where None is used. In testing, without looking at the image again to regain visual tracking, nothing I did caused the trackingState to switch to TrackingState.None, regardless of the ARSession.notTrackingReason.

Your use case should decide what to do when tracking is Limited, but for the most part you should consider TrackingState.Limited as "not being visually tracked". You could add a timer to see how long this state persists, or a check to see if the estimated position is off screen in order to consider the image as "untracked", even though the session will continue to look for it and update its pose. Perhaps if there was some threshold or confidence level of the sensor data relative to the last pose, the state could then be set to None if it is really off the mark - but I'm pretty sure ARCore nor ARKit report anything like this.

The confusion behind these three states is down to having a generic API where one subsystem might support 3 tracking states (ARCore) and another might only support 2 (ARKit). I haven't tested it much, but I believe that ARCore also uses None in a similar sense, and as long as it functions similarly, there shouldn't be any inconsistencies in the way that an app should be designed between platforms.

I really think that this issue is by design, and if so perhaps the documentation should be updated to clarify this?

Thanks for a great reply @macklinb this really helps in choosing a further design path instead trying to debug. Yes, perhaps the documentation could reflect the differences in these states for different platforms.

[mdurand42]

Update: We confirmed with Google today that the ARCore API assumes that augmented images are static in the environment so once they are recognized successfully they will always appear in the list of anchors until the session is reset. This means that AR Foundation will never report that a tracked images is removed on ARCore-enabled devices nor will a tracked image's tracking state be reported as None.

I will mark this issue is resolved since the issue around ARKit tracked images was explained in the thread above.