Accessibility-insights-web: Landmarks requirements confusing for target page with no (or multiple) main landmarks

Created on 21 Apr 2020 · 13Comments · Source: microsoft/accessibility-insights-web

Describe the bug

The "How to test" info for the Landmarks > Primary content requirement reads:

The visual helper for this requirement highlights the page's main landmark.

In the target page, examine the main landmark to verify that it contains all of the page's primary content.

Record your results:
a. If you find any failures, select Fail, then add them as failure instances.
b. Otherwise, select Pass.

The Getting Started text for landmarks includes this paragraph:

Landmarks are not required, but if you use them, you must use them correctly. Also, if you add Landmarks, you must have a main Landmark.

...however, it's not clear to me that this is up to date (I'd have expected it to fail an automated check for "contains landmarks but no main landmark", but it doesn't, maybe it's one of the best-practice landmark rules now?), and it's easy to miss this while going through the Primary content requirement, and the "How to test" text seems to be written assuming that there will always be a main landmark. I was assessing a page without landmarks today and was confused for a while about whether I was supposed to mark this "pass" or "fail".

Same feedback applies to the "No repeating content" requirement's text.

I think it would be nice to change the How to test text to something that makes it more clear whether the user should mark pass or fail for target pages that don't have exactly 1 main landmark. Leaving the exact text change for @ferBonnin / @LiLoDavis to suggest.

If we want these to be marked pass, it would probably be ideal for the requirement to auto-pass if there is no main element, similar to how the Landmark roles requirement auto-passes if there are no landmarks found.

While we're here it would also be nice to fix the inconsistent use of <Markup> tags with the term "main" in these requirements.

To Reproduce
Steps to reproduce the behavior:

Create a local test html file with the following content (can't use codepen because its visual wrappers have landmarks in them):
html <html lang="en"> <head> <title>Test page</title> </head> <body> <h1>Content</h1> <p>Test content.</p> </body> </html>
Start an AIWeb assessment against this page (will need to enable local file usage if not already enabled)
Observe that automated checks all pass
Go to "Landmark" test
Observe confusing text/behavior of "Primary content" and "No repeating content" requirements
(note: the example html above has no landmarks at all, but the same repro steps apply to a page with some non-main landmarks but no main landmarks, or to a page with multiple main landmarks)

Expected behavior

It should be clear to a user without any background in landmarks usage what they need to do to assess a page with:

No landmarks
Landmarks, but no main landmarks
Multiple main landmarks

Context (please complete the following information)

OS Name & Version: any
AI-Web Version & Environment: 2.16.0 Production
Browser Version: any
Target Page: any page without exactly 1 main landmark, see repro above

Are you willing to submit a PR?

yes

Did you search for similar existing issues?

yes

bug resolved

Source

dbjorge

Most helpful comment

Per conversation with Dan, let's go with option 1 and change the text from:
"If no matching/failing instances are found, this requirement will automatically be marked as pass" to
"If no landmarks are found, this requirement will automatically be marked as pass"

ferBonnin on 4 May 2020

👍2

All 13 comments

This issue has been marked as ready for team triage; we will triage it in our weekly review and update the issue. Thank you for contributing to Accessibility Insights!

msft-github-bot on 21 Apr 2020

id have expected it to fail an automated check for "contains landmarks but no main landmark", but it doesn't, maybe it's one of the best-practice landmark rules now?

@dbjorge there isnt a rule that checks for that. the closer think is landmark-one-main but its a best practice.

now for clarifying the how to test in Primary content and no repeating content:
@LiLoDavis should we add some step that says something around: if you don't have any landmarks then you can mark this req as pass, if you have landmarks but not a main landmark then its a fail?

to make it easier to the user we could automatically pass these two requirements if there are no landmarks at all?

ferBonnin on 21 Apr 2020

Yes, I'll work on the text.

LiLoDavis on 21 Apr 2020

Here are two versions: one with automatic pass and one without.

Alternative 1: Automatic pass when page has no landmarks of any type

2 Primary content. The main landmark must contain all of the page's primary content.
The visual helper for this requirement highlights the page's main landmark.
_Note: If no matching/failing instances are found, this requirement will automatically be marked as pass._

Examine the target page to verify that all of the following are true:
a. The page has exactly one main landmark, and
b. The main landmark contains all of the page's primary content.
Record your results:
a. If you find any failures, select Fail, then add them as failure instances.
b. Otherwise, select Pass.

3 No repeating content. The main landmark must not contain any blocks of content that repeat across pages.
The visual helper for this requirement highlights the page's main landmark.
_Note: If no matching/failing instances are found, this requirement will automatically be marked as pass._

Examine the target page to verify that all of the following are true:
a. The page has exactly one main landmark, and
b. The main landmark does not contain any blocks of content that repeat across pages (such as site-wide navigation links).
Record your results:
a. If you find any failures, select Fail, then add them as failure instances.
b. Otherwise, select Pass.

Alternative 2: No automatic pass

2 Primary content. The main landmark must contain all of the page's primary content.
The visual helper for this requirement highlights the page's main landmark.

Examine the target page to verify that one of the following is true:
a. The page has exactly one main landmark, and the main landmark contains all of the page's primary content, or
b. The page has no landmarks of any type.
Record your results:
a. If you find any failures, select Fail, then add them as failure instances.
b. Otherwise, select Pass.

3 No repeating content. The main landmark must not contain any blocks of content that repeat across pages.
The visual helper for this requirement highlights the page's main landmark.

Examine the target page to verify that one of the following is true:
a. The page has exactly one main landmark, and the main landmark does not contain any blocks of content that repeat across pages (such as site-wide navigation links), or
b. The page has no landmarks of any type.
Record your results:
c. If you find any failures, select Fail, then add them as failure instances.
d. Otherwise, select Pass.

LiLoDavis on 21 Apr 2020

I thought a page was specifically allowed to have multiple main landmarks if iframes are involved, there is at most one per iframe, and they have distinct labels?

dbjorge on 23 Apr 2020

Oh, yeah, you're right. Forgot about that case. Below I added exceptions for nested main landmarks, but I didn't mention the label requirement, which I see as a separate issue. Also, I didn't say that the nested pages have to meet the requirements, which could be quite burdensome to verify. @ferBonnin do you agree with this approach?

Alternative 1: Automatic pass when page has no landmarks of any type

Examine the target page to verify that all of the following are true:
a. The page has exactly one main landmark, and
b. The main landmark contains all of the page's primary content.
Exception: If a page has nested document or application roles (typically applied to iframe or frame elements), each nested document or application may _also_ have one main landmark.
Record your results:
a. If you find any failures, select Fail, then add them as failure instances.
b. Otherwise, select Pass.

Examine the target page to verify that all of the following are true:
a. The page has exactly one main landmark, and
b. The main landmark does not contain any blocks of content that repeat across pages (such as site-wide navigation links).
Exception: If a page has nested document or application roles (typically applied to iframe or frame elements), each nested document or application may _also_ have one main landmark.
Record your results:
a. If you find any failures, select Fail, then add them as failure instances.
b. Otherwise, select Pass.

Alternative 2: No automatic pass

2 Primary content. The main landmark must contain all of the page's primary content.
The visual helper for this requirement highlights the page's main landmark.

Examine the target page to verify that one of the following is true:
a. The page has no landmarks of any type, or
b. The page has exactly one main landmark, and the main landmark contains all of the page's primary content.
Exception: If a page has nested document or application roles (typically applied to iframe or frame elements), each nested document or application may _also_ have one main landmark.
Record your results:
a. If you find any failures, select Fail, then add them as failure instances.
b. Otherwise, select Pass.

3 No repeating content. The main landmark must not contain any blocks of content that repeat across pages.
The visual helper for this requirement highlights the page's main landmark.

Examine the target page to verify that one of the following is true:
a. The page has no landmarks of any type, or
b. The page has exactly one main landmark, and the main landmark does not contain any blocks of content that repeat across pages (such as site-wide navigation links).
Exception: If a page has nested document or application roles (typically applied to iframe or frame elements), each nested document or application may _also_ have one main landmark.
Record your results:
c. If you find any failures, select Fail, then add them as failure instances.
d. Otherwise, select Pass.

LiLoDavis on 23 Apr 2020

❤1

Per conversation with Peter lets go with opt 1 (automatic pass).

ferBonnin on 27 Apr 2020

I think the proposed option 1 text is a little confusing in the case where there exist some non-main landmarks but no main landmarks; I think what would happen in that state would be:

The text will say "If no matching/failing instances are found, this requirement will automatically be marked as pass."
There will be no listed matching/failing instances
...but the requirement will not be automatically marked as passed

There aren't any other requirements right now that have both auto-pass behavior and require the user manually enter instances like alt 1 proposes; everything is either an assisted list of instance that auto-passes if there are no matching instances, or non-assisted. There aren't currently any cases where we calculate auto-pass behavior based on a list of instances that we don't show the user in an instance list.

Personally, my preference would be:

Primary content: have the user verify only that every main landmark contains all of the primary content for its corresponding document (make it an assisted test that lists main landmarks as instances)
No repeating content: have the user verify only that every main landmark contains no repeating content (make it an assisted test that lists main landmarks as instances)
Automated checks: add a new check that verifies that if any landmarks are present, the page must pass the axe-core landmark-one-main rule

(or alternative 2, if we want to save on impl cost)

dbjorge on 4 May 2020

(@ferBonnin / @LiLoDavis , see above)

dbjorge on 4 May 2020

ferBonnin on 4 May 2020

👍2

@LiLoDavis @ferBonnin The text updates and new auto-pass behavior are now in canary, it'd be great if you could double check them to close out the bug :)

Per discussion with Fer we also implemented similar auto-pass behavior for Semantics > CSS content and Sequence > CSS positioning; would be good to double check that too (the only text change for those was adding the standard auto-pass note in the standard spot after the visual helper description).

dbjorge on 18 May 2020

@dbjorge @ferBonnin, these look good to me: