Adaptivecards: Accessibility issues

[arpitgupta19]

Platform

iOS

Author or host

Host

Version of SDK

1.2

Issue

Voiceover and screen reader is not working correctly on adaptive cards.
We are sending correct json object but voiceover is not navigating in a serial order.
Also, plenty of UI elements are not accessible.

In sample json given below, voiceover navigates to first container inside body, then moves to actions and then comes back to factsets in the body

{
  "body": [
    {
      "items": [
        {
          "size": "large",
          "weight": "bolder",
          "type": "TextBlock",
          "text": "Would you like to book this shuttle?"
        }
      ],
      "type": "Container"
    },
    {
      "items": [
        {
          "facts": [
            {
              "title": "Pick-up location:",
              "value": "Building 11"
            },
            {
              "title": "Drop-off location:",
              "value": "Building 33"
            },
            {
              "title": "Passenger count:",
              "value": "1"
            },
            {
              "title": "Accessibility options:",
              "value": "(none)"
            }
          ],
          "id": "BookingFactSet",
          "type": "FactSet"
        }
      ],
      "type": "Container"
    }
  ],
  "type": "AdaptiveCard",
  "actions": [
    {
      "title": "Book now",
      "data": {
        "CardType": "ShuttleBookingCard",
        "Command": "BookShuttle"
      },
      "type": "Action.Submit"
    },
    {
      "title": "Pick-up",
      "card": {
        "type": "AdaptiveCard",
        "actions": [
          {
            "title": "Save",
            "data": {
              "CardType": "ShuttleBookingCard",
              "Command": "ChangeFromLocation"
            },
            "type": "Action.Submit"
          }
        ],
        "version": "1.0"
      },
      "type": "Action.ShowCard"
    },
    {
      "title": "Drop-off",
      "card": {
        "type": "AdaptiveCard",
        "actions": [
          {
            "title": "Save",
            "data": {
              "CardType": "ShuttleBookingCard",
              "Command": "ChangeToLocation"
            },
            "type": "Action.Submit"
          }
        ],
        "version": "1.0"
      },
      "type": "Action.ShowCard"
    },
    {
      "title": "Options",
      "card": {
        "type": "AdaptiveCard",
        "actions": [
          {
            "title": "Save",
            "data": {
              "CardType": "ShuttleBookingCard",
              "Command": "ChangeOptions"
            },
            "type": "Action.Submit"
          }
        ],
        "version": "1.0"
      },
      "type": "Action.ShowCard"
    }
  ],
  "version": "1.0"
}

[shalinijoshi19]

Thanks for reporting @arpitgupta19. We are doing an all-around Accessibility push for 1.3(including team training + testing by a contract team + bug fixes) ; We'll make sure this set of issues gets resolved as part of this push. Are you targeting a specific date for the fixes to be available at your end? FYI that 1.3 is some ways out (likely targeting end of this year for a Beta attempt)

[arpitgupta19]

@shalinijoshi19 There is a strong push on accessibility fixes on our end and the adpative cards issues have become high priority items.
We have been asked to get these issues fixed asap.

[shalinijoshi19]

@arpitgupta19, just to be clear you are from the Teams team correct? Meanwhile, understood and @nesalang is already looking into this one but just setting your expectations on this that we are aware of a few major Accessibility related issues in the platform and a whole bunch that we aren't even aware exist. The next few months for us a team, will be dedicated to discovering, triaging, fixing these issues across the board across all platforms. If there's low hanging fruit we'll certainly try our best to get these fixed and work with you to backport for consumption in 1.2 and others just roll out all up as part of our vNext release (1.3). Am also following up offline around this. Thanks again

[arpitgupta19]

@shalinijoshi19 yes, I am from teams mobile team working on iOS platform.
Justin Jacob can give you more info regarding the prioritisation.

[jwoo-msft]

@arpitgupta19 @shalinijoshi19 for visibility
i ran the provided json, and tested the accessibility scenario.

how i tested it:

1. turned on the voice over from ios setting.
2. from ios sample app in the repo, ran the json,
3. then swiped up with two fingers on a ios device to start voice over that reads each items in the screen sequentially. Speak the entire screen, from the top: Swipe up with two fingers.

what i found:
voice over started and ended in expected order. from textblock to factset's top most title to its value, then to next fact, and so on. Then to the buttons starting from the left and to the end.

is your experience different from what i just described?

Thanks

[shalinijoshi19]

Have not heard back from Teams around repro steps here. Removing from 1.2.3 investigations.

[shalinijoshi19]

@arpitgupta19 assigning to you to help with @nesalang's questions

[VikrantSingh01]

@arpitgupta19 : Can you pls. get this updated.

[shalinijoshi19]

@VikrantSingh01 heads up this is likely to get cut from our list (we are tracking ZBB for next week) since we haven't heard back from @arpitgupta19

[arpitgupta19]

@shalinijoshi19 @nesalang
I am seeing these issues in sdk version 1.2.
In which sdk version did you verify the changes to be working fine.

[shalinijoshi19]

@VikrantSingh01 this needed to be pushed out because of the lack of repro + long turnaround time here. This is next on our radar.

[VikrantSingh01]

@nesalang : Once you get some time to verify these accessibility issues. Pls. let us know. You are also copied on the CSEO thread from where you can check the reproducer cases. CC - @arpitgupta19

[jwoo-msft]

@VikrantSingh01
Please see the video pointed by this link you have to download it first, and play it on mac or quicktime player.
This video shows that voice over is working as expected. Sample app used in the video is based on the release 1.2.5

[arpitgupta19]

@nesalang @shalinijoshi19
Thanks for verifying this.
We will update to 1.2.5 and verify the issues on our side.

[jwoo-msft]

@arpitgupta19 thanks for confirming.
feel free to open it once you've verified it after 1.2.5 sdk integration

[spencer-microsoft]

@nesalang I tested with 1.2.5 integration and voiceover is still getting stuck in adaptive cards. Is there support for including an accessibility summary when creating the json payload for an adaptive card so the card as a whole can be summarized for voiceover?

[jwoo-msft]

@spencer-microsoft Can you elaborate on what is "1.2.5 integration"? thanks.