Botframework-webchat: Accessibility Issue - Screen reader reads adaptive card content as "attachment"

Created on 13 Jan 2021  ·  20Comments  ·  Source: microsoft/BotFramework-WebChat

We are using webchat-es5.js for our Virtual assistant and our customers are using NVDA screen reader to access it. The screen reads the normal card and user messages properly. But for Adaptive card, it reads as "Bullet A Card" , "one attachment on date time" instead of reading the content. When checked the DOM, the adaptive card has aria-roledescription="attachment" which triggers screen reader to read it as attachment.

Screenshots

image

Version

webchat es5 version 4.11.0
https://cdn.botframework.com/botframework-webchat/4.11.0/webchat-es5.js
Integrating the CDN version js directly on the page using script tag (No Iframe). Desktop Chrome Browser.

<meta name="botframework-directlinespeech:version" content="4.11.0">
<meta name="botframework-webchat:bundle:variant" content="full-es5">
<meta name="botframework-webchat:bundle:version" content="4.11.0">
<meta name="botframework-webchat:core:version" content="4.11.0">
<meta name="botframework-webchat:ui:version" content="4.11.0">

Describe the bug

We are using webchat-es5.js for our Virtual assistant and our customers are using NVDA screen reader to access it. The Virtual Assistant is web app bot developed using VA template that uses QnA/LUIS cognitive services. The screen reader reads normal card text and user input text. But if Bot response is adaptive card, then it reads as "Bullet A Card", "one attachment received on date time" instead of reading the card content. The adaptive card has a message and few buttons.

Steps to reproduce

  1. Deploy Web App Bot with adaptive cards.
  2. Start NVDA screen reader or JAWS 2021 screen reader.
  3. Open the web app bot on browser.
  4. Interact with the bot till the adaptive card response arrives.
  5. Observe how the screen reader reads the bot response.

Expected behavior

The screen reader should read the adaptive card content along with control texts (Button labels). The DOM should render adaptive card response in div with attribute aria-roledescription="message" instead of aria-roledescription="attachment"

Please advice if any other fix to make the screen reader to read the adaptive card content.
[Bug]

Accessibility Bot Services Bug customer-replied-to customer-reported
Source
picture venkatx5

All 20 comments

When using NVDA and JAWS, what browsers are you using them in combination with?

e.g. For Edge, the recommended screenreader is Windows Narrator.

https://microsoft.github.io/BotFramework-WebChat/01.getting-started/a.full-bundle/ When I test an adaptive card (e.g. try the command: card sports) on Chrome+NVDA / Edge+Narrator, I see the intended behavior which is for messages with attachments to first read the message text, if applicable, then indicate an attachment, then read the content of the card.

The reasoning for aria-roledescription="attachment" is because the card is sent in the message activity under the attachment prop.

Could you describe the mal-behavior you are experiencing because the content is an attachment? If possible, it would be great to have a video.

@tracyboehrer FYI, I reassigned this to myself to help load balance compulim. Please let me know if this causes Bot Service reporting problems. Thank you.

corinagum picture corinagum on 13 Jan 2021

Used Google chrome Version 87.0.4280.141 (Official Build) (64-bit)
Tested in Edge with Narrator, It reads the button text in adaptive card. But it doesn't read the text before the buttons.

venkatx5 picture venkatx5 on 15 Jan 2021

I'm unable to produce a repro. Please provide your Adaptive Card json with PII removed and I will test it out.

corinagum picture corinagum on 15 Jan 2021

Here the adaptive card json. Thanks. 


{
  "type": "message",
  "serviceUrl": "http://localhost:52061",
  "channelId": "emulator",
  "from": {
    "id": "c66*******************581",
    "name": "Bot",
    "role": "bot"
  },
  "conversation": {
    "id": "5*****1|livechat"
  },
  "recipient": {
    "id": "7e7ca***********a2",
    "role": "user"
  },
  "speak": "Please select one of the questions below to learn more or type in another question you may have.",
  "inputHint": "expectingInput",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.adaptive",
      "content": {
        "type": "AdaptiveCard",
        "version": "1.0",
        "body": [
          {
            "type": "TextBlock",
            "text": "Please select one of the questions below to learn more or type in another question you may have.",
            "speak": "Please select one of the questions below to learn more or type in another question you may have."
          }
        ],
        "actions": [
          {
            "type": "Action.Submit",
            "id": "How can I contact ABC?",
            "data": "How can I contact ABC?",
            "title": "How can I contact ABC?"
          },
          {
            "type": "Action.Submit",
            "id": "Why doesn't anyone answer the phone?",
            "data": "Why doesn't anyone answer the phone?",
            "title": "Why doesn't anyone answer the phone?"
          },
          {
            "type": "Action.Submit",
            "id": "I'm experiencing a technical issue; how do I contact the help desk?",
            "data": "I'm experiencing a technical issue; how do I contact the help desk?",
            "title": "I'm experiencing a technical issue; how do I contact the help desk?"
          },
          {
            "type": "Action.Submit",
            "id": "Return to main menu",
            "data": "Return to main menu",
            "title": "Return to main menu"
          }
        ]
      }
    }
  ],
  "replyToId": "0ee3d090-****-****-****-698db52d3009",
  "id": "2101b***-****-11eb-9dfe-698db52d3009",
  "localTimestamp": "2021-01-18T12:03:51+05:30",
  "timestamp": "2021-01-18T06:33:51.572Z",
  "locale": "en-US"
}
venkatx5 picture venkatx5 on 18 Jan 2021

Looks like you have a mal-formed Adaptive Card. I got following to read as expected with AT:

{
  "$schema": "https://microsoft.github.io/AdaptiveCards/schemas/adaptive-card.json",
  "type": "AdaptiveCard",
  "version": "1.0",
  "speak": "Please select one of the questions below to learn more or type in another question you may have.",
  "body": [
    {
      "text": "Please select one of the questions below to learn more or type in another question you may have.",
      "version": "1.0",
      "type": "TextBlock"
    }
  ],
  "actions": [
    {
      "type": "Action.Submit",
      "id": "How can I contact ABC?",
      "data": "How can I contact ABC?",
      "title": "How can I contact ABC?"
    },
    {
      "type": "Action.Submit",
      "id": "Why doesn't anyone answer the phone?",
      "data": "Why doesn't anyone answer the phone?",
      "title": "Why doesn't anyone answer the phone?"
    },
    {
      "type": "Action.Submit",
      "id": "I'm experiencing a technical issue; how do I contact the help desk?",
      "data": "I'm experiencing a technical issue; how do I contact the help desk?",
      "title": "I'm experiencing a technical issue; how do I contact the help desk?"
    },
    {
      "type": "Action.Submit",
      "id": "Return to main menu",
      "data": "Return to main menu",
      "title": "Return to main menu"
    }
  ]
}
corinagum picture corinagum on 19 Jan 2021
😕1

The json generated by the code and we extracted it from emulator. Please advice what we have to fix?
If possible, please keep this issue open as it's still exist.

venkatx5 picture venkatx5 on 20 Jan 2021

Updated json. Please review and advice..


{
  "type": "message",
  "serviceUrl": "http://localhost:51252",
  "channelId": "emulator",
  "from": {
    "id": "e4*************************91",
    "name": "Bot",
    "role": "bot"
  },
  "conversation": {
    "id": "e4**********************6f53a|livechat"
  },
  "recipient": {
    "id": "17*************************f7735",
    "role": "user"
  },
  "inputHint": "expectingInput",
  "attachments": [
    {
      "contentType": "application/vnd.microsoft.card.adaptive",
      "content": {
        "type": "AdaptiveCard",
        "version": "1.0",
        "body": [
          {
            "type": "TextBlock",
            "id": "firstresponse",
            "text": "I am sorry, I am not able to understand your question. Can you please rephrase your question or click on �Return to main menu� button to go to main menu and navigate the categories to find a question that I can answer. If I can�t answer your question, I can..."
          },
          {
            "type": "TextBlock",
            "id": "secondresponse",
            "text": "I am sorry, I am not able to understand your question. Can you please rephrase your question or click on �Return to main menu� button to go to main menu and navigate the categories to find a question that I can answer. If I can�t answer your question, I can connect you with a live agent. Our agents are available on 9:00 AM to 5:00 PM CST. Please check during these hours.",
            "isVisible": false
          },
          {
            "id": "showmorebtns",
            "type": "ActionSet",
            "actions": [
              {
                "type": "Action.ToggleVisibility",
                "targetElements": [
                  {
                    "elementId": "firstresponse",
                    "isVisible": false
                  },
                  {
                    "elementId": "secondresponse",
                    "isVisible": true
                  },
                  {
                    "elementId": "showmorebtns",
                    "isVisible": false
                  },
                  {
                    "elementId": "hidemorebtns",
                    "isVisible": true
                  }
                ],
                "title": "Read more"
              }
            ]
          },
          {
            "id": "hidemorebtns",
            "type": "ActionSet",
            "actions": [
              {
                "type": "Action.ToggleVisibility",
                "targetElements": [
                  {
                    "elementId": "secondresponse",
                    "isVisible": false
                  },
                  {
                    "elementId": "showmorebtns",
                    "isVisible": true
                  },
                  {
                    "elementId": "firstresponse",
                    "isVisible": true
                  },
                  {
                    "elementId": "hidemorebtns",
                    "isVisible": false
                  }
                ],
                "title": "Read less"
              }
            ],
            "isVisible": false
          }
        ],
        "actions": [
          {
            "type": "Action.Submit",
            "id": "return",
            "data": "Return to main menu",
            "title": "Return to main menu"
          }
        ]
      }
    }
  ],
  "replyToId": "36***********************f53a",
  "id": "6b2*************************f53a",
  "localTimestamp": "2021-01-20T10:16:11+05:30",
  "timestamp": "2021-01-20T04:46:11.203Z",
  "locale": "en-US"
}
venkatx5 picture venkatx5 on 20 Jan 2021

@v-kydela or @stevkan could you guys triage this item? Thanks!

corinagum picture corinagum on 20 Jan 2021

@corinagum - We can assign this in tomorrow's meeting

v-kydela picture v-kydela on 20 Jan 2021
👍1

@corinagum

I can reproduce the issue. The behavior I observe is that NVDA is able to read all the text in the Adaptive Card correctly when hovering over the text, but when the mouse is still and NVDA reads the messages automatically as they appear, NVDA just says "bullet A..." as @venkatx5 reported.

I'm not sure how Web Chat interfaces with the screen reader, but Web Chat could supposedly be modified to provide more useful information to the screen reader in the case of Adaptive Cards, or we could just change the role description to "message" as @venkatx5 suggested.

What would you like me to do now?

v-kydela picture v-kydela on 21 Jan 2021

@v-kydela thanks for the help, I obviously missed something. Would you mind recording a video of the behavior so I can verify on my side? After that we'll reassign this to me for further investigation. (Both hovering and live reading) Thanks again for the help.

corinagum picture corinagum on 22 Jan 2021

@corinagum - I just did it in Emulator so it's easier to repro, but this is the same behavior that's in Web Chat:

https://user-images.githubusercontent.com/41968495/105538272-03074000-5ca8-11eb-82a2-5ebfd2570394.mp4

v-kydela picture v-kydela on 22 Jan 2021
👍1

(just in case can you also include the json you used, @v-kydela?)

Thanks for the repro! I'm working on another P0 accessibility bug today but I will look at this issue again next week.

corinagum picture corinagum on 22 Jan 2021

@corinagum - I did not modify @venkatx5's JSON:

{
  "type": "AdaptiveCard",
  "version": "1.0",
  "body": [
    {
      "type": "TextBlock",
      "id": "firstresponse",
      "text": "I am sorry, I am not able to understand your question. Can you please rephrase your question or click on �Return to main menu� button to go to main menu and navigate the categories to find a question that I can answer. If I can�t answer your question, I can..."
    },
    {
      "type": "TextBlock",
      "id": "secondresponse",
      "text": "I am sorry, I am not able to understand your question. Can you please rephrase your question or click on �Return to main menu� button to go to main menu and navigate the categories to find a question that I can answer. If I can�t answer your question, I can connect you with a live agent. Our agents are available on 9:00 AM to 5:00 PM CST. Please check during these hours.",
      "isVisible": false
    },
    {
      "id": "showmorebtns",
      "type": "ActionSet",
      "actions": [
        {
          "type": "Action.ToggleVisibility",
          "targetElements": [
            {
              "elementId": "firstresponse",
              "isVisible": false
            },
            {
              "elementId": "secondresponse",
              "isVisible": true
            },
            {
              "elementId": "showmorebtns",
              "isVisible": false
            },
            {
              "elementId": "hidemorebtns",
              "isVisible": true
            }
          ],
          "title": "Read more"
        }
      ]
    },
    {
      "id": "hidemorebtns",
      "type": "ActionSet",
      "actions": [
        {
          "type": "Action.ToggleVisibility",
          "targetElements": [
            {
              "elementId": "secondresponse",
              "isVisible": false
            },
            {
              "elementId": "showmorebtns",
              "isVisible": true
            },
            {
              "elementId": "firstresponse",
              "isVisible": true
            },
            {
              "elementId": "hidemorebtns",
              "isVisible": false
            }
          ],
          "title": "Read less"
        }
      ],
      "isVisible": false
    }
  ],
  "actions": [
    {
      "type": "Action.Submit",
      "id": "return",
      "data": "Return to main menu",
      "title": "Return to main menu"
    }
  ]
}
v-kydela picture v-kydela on 22 Jan 2021
👍1

@venkatx5 is this still an issue blocking you?

srinaath picture srinaath on 4 Feb 2021

@venkatx5 is this still an issue blocking you?

The issue is still persist. Is there any fix deployed? Should I change the webchat js version?

venkatx5 picture venkatx5 on 7 Feb 2021

Problem/fix for this issue still needs to be investigated.

corinagum picture corinagum on 17 Feb 2021

@venkatx5 I'm not sure how this is a Web Chat bug. You say you got the card from emulator, but when you test it in emulator, this card doesn't work, correct? Since emulator uses WC, this would be expected in your case. On the bot, have you confirmed that this card is not malformed? Please try sending the json I sent above and see what happens: https://github.com/microsoft/BotFramework-WebChat/issues/3667#issuecomment-763158435

Again, when I test other Adaptive Cards, the AT is working as expected. When I test the code you provided above, it does not.

For example, please visit the following link and type 'card sports' and send: sample

You will see that the card is read aloud.

I am assigning @amal-khalaf for sanity check. However @venkatx5 please also try out the above.

corinagum picture corinagum on 18 Feb 2021

For the speak property to work, it need to be placed in the card, i.e. under content. See highlighted in the AC schema below.

image

I spotted few problems in your card attached here:

  • There are two speak property:

    • The first one is in the activity, not in the card

    • This will not be narrated by screen readers via Web Chat, because the content can potentially contains SSML

    • The second one is in the card inside the TextBlock element

    • The AC schema for TextBlock element said it should not have speak property

Right now, for AC inside Web Chat, we use the AC's speak property for screen reader, which IMO, is not a correct behavior but a workaround for AC alt text.

So we tested on Chrome + NVDA (version 2020-02) with your original card and it narrate: "Bot said. Bullet. A card. How can I contact ABC? ... Return to main menu. Sent at February 18th, 2021 at 12:34 PM." This is expected behavior.

In order for screen reader to speak the card text body, you need to put a speak property inside the root of the card, but not inside one of the TextBlock in the card.

Another thing: we use different DOM elements for narrating when the activity arrives. The DOM elements you pointed in your screenshot, is not used for live reading on the screen reader. However, if the user navigate the transcript, then it will narrate the DOM elements you referred. This is by design and is a workaround for browser bugs. You can read more about the accessibility design in this documentation.

@v-kydela when using screen reader, never touch the mouse.

compulim picture compulim on 19 Feb 2021

@compulim Thanks for the suggestion. We'll try the fix and let you know the outcome.

venkatx5 picture venkatx5 on 23 Feb 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

GewoonMaarten picture GewoonMaarten  ·  3Comments
naveen-vijay picture naveen-vijay  ·  4Comments
compulim picture compulim  ·  3Comments
Kellym-Kainos picture Kellym-Kainos  ·  4Comments
prashanthsridhar picture prashanthsridhar  ·  3Comments