Xamarin.forms: [Android] Using AutomationId prevents TalkBack screenreader accessibility

[tomgilder]

Description

When using the AutomationId property on Android, apps end up almost completely inaccessible to users who use Android's TalkBack screenreader.

Xamarin.Forms "borrows" the ContentDescription property on Android for Automation IDs. These IDs polute Android's TalkBack accessibility tree, making apps almost impossible to navigate.

This means you can support test automation or accessibility, not both. Our app needs to support both 😃

I can't find any easy way of working around this restriction, short of walking the Android view tree and setting ContentDescription to null. Being unable to update AutomationId once set also adds to the difficulty of working around this issue.

Potential solutions

1. Use another system for setting AutomationIDs

Probably a non-starter for backwards compatibility?

2. Monitor AccessibilityManager to see if accessibility is enabled

Assuming there’s no alternative property to use for setting IDs, one solution would be for Xamarin.Forms to use Android's AccessibilityManager and check the IsTouchExplorationEnabled property.

This is true when the TalkBack screen-reader is being used, and Forms could ignore AutomationId and not set ContentDescription. Forms should then also use an ITouchExplorationStateChangeListener to monitor changes to this state.

Steps to Reproduce

1. Create any element with an AutomationId set
2. Go to that element on an Android device with TalkBack enabled

Expected Behavior

Element is ignored, or its actual content is read.

Actual Behavior

Element's AutomationId is read to the user.

Basic Information

• Version with issue: 2.5.0.121934
• Last known good version: none. Always happened since AutomationId supported.

[nexxuno]

Hello,
I'm having this problem too, this is how I workaround it while waiting for an improved xamarin version.

In xaml production code:
rnational:BMapTranslate goLabel}" MinimumWidthRequest="150" Text="{international:BMapTranslate goLabel}" ...

In Xamarin Coded UI testing:
app.Tap(c => c.Button(tr.GetTranslation("goLabel")));

In other words I'm using the same translator class and resource files I use in production as helpers in the UItests. I know it's not optimal but it works for now...

[codingL3gend]

@tomgilder

i was able to find a work around to this issue by creating a customrenderer and respective custom component to allow for overriding the native android method(s) that get triggered when accessibility events are fired. you will need to create some bindable properties on your custom component that you can access in the custom renderer to allow for setting the content description value to what you want but that's simple enough.

this method gets triggered in the control/custom renderer whenever an accessibility event is fired

public override bool OnRequestSendAccessibilityEvent(Android.Views.View child, AccessibilityEvent e)
  {
         if (AccessibilityHandler.IsAccessibilityEnabled(_context) && child != null)
         {
                if (!string.IsNullOrEmpty(_automationId) && _automationId.Equals(child.ContentDescription))
                {
                    child.ContentDescription = $"{_automationName} {_helpText}";
                }
         }

         return base.OnRequestSendAccessibilityEvent(child, e);
}

then you can set the contentDescription value of the control/custom renderer back to what the automationId value was originally when the control/custom renderer is detached from the view.

protected override void OnDetachedFromWindow()
  {
        base.OnDetachedFromWindow();

        if (!string.IsNullOrEmpty(_automationId))
        {
            Control.ContentDescription = _automationId;
        }
  }

helper class

public static class AccessibilityHandler
    {
        public static bool IsAccessibilityEnabled(Context context)
        {
            var accessibility = (AccessibilityManager)context.GetSystemService(MainActivity.AccessibilityService);

            return accessibility?.GetEnabledAccessibilityServiceList(Android.AccessibilityServices.FeedbackFlags.Spoken)?.Count > 0;
        }
    }

[hminaya]

@jimbobbennett have you run into this before? any thoughts?

[claudiosanchez]

@brminnick and @davidortinau are aware of this.

[claudiosanchez]

@codingL3gend How many renderers did you have to create? Have you had any additional issues with the approach you suggested?

[codingL3gend]

@claudiosanchez i think your question may have been meant to be directed toward me. We already had renderers created for other reasons outside of needed ada and automation support; so I just leveraged those implementations. However for any controls where we need this support that we don't have renderers for already, we will create new ones for those. So far we haven't had any issues with this approach thus far and it is working as expected with the controls we have tested it against.

[claudiosanchez]

@codingL3gend You are correct. I fixed the mention :). We implemented your workaround and so far is working like a charm for a Button Renderer, and your answer is on point. I wanted to know what had been the experience with this workaround so far. We are going to propose this solution to the development team 💯. cc: @brminnick

[codingL3gend]

@claudiosanchez glad to hear that I could be of assistance with the workaround. currently our application is still in the development phase and have only gone through local testing so we don't know if any major issues will arise from it if any. We've also ran it against some automated testing and its performing as expected. Hopefully the implementation will help others in the future and perhaps help with an overall platform solution soon enough.

[claudiosanchez]

@codingL3gend Happy to report we have implemented a button solution based on your guidance. The hope is to submit back to the code base at some point. What other renderers did you have implement to support both Accessibility and UI Test Automation?

[codingL3gend]

@claudiosanchez that's great news to hear; happy I could help in some capacity with providing a solution. the other renderers implemented were around the default xamarin forms controls built on top of the native controls such as Label, Switch, DateEntry, LinkButton, and Entry. these are the more common controls that are being used; although custom controls have been created for ListView, ScrollView, and TableView. those renderers haven't needed any specific Accessibility or UI Test Automation configuration.

[codingL3gend]

@claudiosanchez also want to add a quick touch point. we noticed that on android when adding automationId to elements on the xamarin forms side (not sure if this is a result of our custom renderers or just how the controls would get created in general) the android resulting control would be wrapped in an additional ViewGroup. meaning that instead of say a button being wrapped in a single ViewGroup parent container and then the resulting Button element as its child, it is instead 2 Parent ViewGroup containers and then the resulting Button element as the inner most child. as a result the direct Parent ViewGroup of the Button then gets its ContentDescription value assigned by default to the child's or element's (i.e Button) AutomationId property plus _Container such as Button_Container. this in turn causes these container elements to be read when the top most parent element is clicked on a View when the Accessibility tree is triggered. even using the FastRenderers (though in beta) does not flatten the elements enough to not include the direct ViewGroup parent of the Button element. this is the next challenge that will be tackled from a workaround standpoint to get either those elements to not read the ContentDescription of Button_Container or ideally to not have the additional ViewGroup to be created at all. if there is a known workaround for this, i'd appreciate the direction.

hopefully that all made sense. I can provide a screen shot if needed.

[geeetarguy]

@codingL3gend @claudiosanchez Did you ever find a fix for this? I'm experiencing the same thing with a ListView reported here: https://github.com/xamarin/Xamarin.Forms/issues/6758

[codingL3gend]

@geeetarguy we did find a fix for this based on the implementation I posted above. however when we updated the version of our Android plugins the implementation began to run into issues. Updating to 4.0 to take advantage of Fast Renderers may help your situation. Also we had to fall back to an implementation that I had as a worst case scenario which is to have two different builds where one would have automation turned on and one would not using compiler flags. hope this helps.

[jfversluis]

I’m wondering it this would be enough to fix it: https://github.com/xamarin/Xamarin.Forms/commit/ba65be103935db3ee85f5b586eccd7ad6db1c767

Simply skipping over the assigning of the AutomationId whenever the AccessibilityManager.IsTouchExplorationEnabled is true. This means you cannot have accessibility on when doing automated tests, but I guess that is the implication of the implementation.

@samhouts thought? Also in light of #5237

[codingL3gend]

@jfversluis that's not a bad idea because we have had to separate our implementations out to have a build for automation vs a build for accessibility. however for situations where accessibility has to tested through automation as well this will not suffice.

[ryl]

I've has some luck using an AccessibilityDelegate to provide text to TalkBack without disrupting ContentDescription.

[jfversluis]

@ryl, sorry for the late reply. Could you define "some luck"? Can we translate it into a fully usable solution perhaps?

[ryl]

@jfversluis Take a look at the UI test I created in this pull request (https://github.com/xamarin/Xamarin.Forms/pull/5237/files#diff-420d1f5f1ff2e21615eb786f50ef8328). Using the AccessibilityDelegate, I am able to find a variety of elements using their AutomationId, but their Name + HelpText is still read out correctly by TalkBack. My original bug report was only focused on buttons, so I'm not sure how well it applies across the entire board of elements. It might need some (or a lot of) refinement. It seems like it might be a good starting point, though.

The nice thing about the approach: ContentDescription and AutomationId continue to work like they do now, but the addition of an AccessibiltiyDelegate allows TalkBack to read from a source other than ContentDescription.

[jfversluis]

It looks like AccessibilityDelegate is available on the View object, so I guess it would be available on all visual elements and we can use this to create a solution for this while retaining both the AutomationId functionality as well as gaining accessibility?

[ryl]

That's my impression so far. The UI test works with button, switch, and image so far. I imagine it might work for others, but I haven't had the opportunity to try out ALL of them.

[codingL3gend]

i'd be willing to look further into this as well. I did an initial implementation using the AccessiblityDelegate but it didn't overwrite the speech the way i wanted it to. Also with changes in android and how it does accessiblity after updates it causes issues with my initial solution outlined above; which resulted in going with the worst case route of having a build for automation and one for accessibilty utilizing compiler directives. this approach seems as if it would be more viable in the long term to maintain

[IeuanWalker]

Any update on this? this has prevented us from implementing the UITest

[ryl]

I haven't heard anything, though I'm also curious if any progress has been made.

[codingL3gend]

@IeuanWalker have you tried using compiler directives in the interim to do both?

[IeuanWalker]

@codingL3gend how do i use compiler directives to only add AutomationId to the ContentDescription whilst running the uitests, but add only the Automation.HelpText every other time?

[codingL3gend]

@IeuanWalker it would depend also on if you have any custom controls as well because you can set the contentdescription value in the native android project based on what the compiler directive is set. this would allow you to add a custom property to your controls that you can check for on the native android side and set the content description to that value when the directive is set. the compiler directives could be set either via command line builds or in the project settings during development.

[IeuanWalker]

@codingL3gend ye we use quite a few custom controls, but most of them don't go down to the native platform. I'll take a look at using compiler directives tomorrow or next week.

Is there not a way to override it globally?
Or an actually fix?

[codingL3gend]

@IeuanWalker i did have a solution that i proposed toward the top of the thread. however this had issues when forms and android updated a few months back. therefore we had to switch to the worst case scenario that i feared would be the only solution in the end and that was having two different builds via the use of the compiler directives. only way to perhaps fix it would be for their to be a different value internally that can be mapped and checked on the native side for android that wouldn't override the contentdescription or allow for a different property to be set on the control itself that automation frameworks and see and use to reliably access the control without the need for xpaths etc.

[ryl]

The AccessibilityDelegate has is own ContentDescription that overrides the element's ContentDescription when read by a screen reader. However, since the element's original ContentDescription remains untouched, AutomationId continues to work. I had a UI test that demonstrated that. Have you had any luck with that approach?

[codingL3gend]

@ryl i haven't looked into that approach yet