Wcag: Challenges: CSS with media features can have huge number of combinations

Created on 13 Jul 2020 · 22Comments · Source: w3c/wcag

The CSS below only has 21 lines and is much simpler than production CSS, but has a huge number of media feature combinations, some of which cause WCAG failures on specific device types, or with certain user preferences.

Assuming it takes 10 minutes to manually check each possible permutation of media features then there are:

4 * 65,536 combinations to test = 262,144 combinations
10 minutes * 262,144 = 2,621,440 minutes
2,621,440 minutes = 1820 days to test a single page (working 24 x 7 x 365)

<style>
body { color: black; background-color: rgba(255,255,255, 0.5); }
/* these have 4 combinations on current hardware */
@media (monochrome) { body { background-color: transparent; }  }
@media (color)) { body { color: red; } /* any color display */ }
@media (min-color: 8) { body { color: #24ba13; } /* display is 24-bit or better */ }
@media (min-color-index: 256) { body { color: pink; } /* display uses 8-bit palette */ }
/* these 16 are either true/false = 2**16 combinations = 65,536 combinations */
@media (prefers-contrast: high) { color: black; background-color: white; }
@media (prefers-reduced-transparency) { body { color: rgb( 127,127,127); } }
@media (prefers-reduced-motion) { body { color: rgb( 70,0,0); } }
@media (prefers-color-scheme: dark) { body { color: rgb(40,40,40); background-color: black; } }
@media (prefers-reduced-data) { body { color: rgb( 0,255,0); } }
@media (forced-colors) { body { color:rgb(200,200,200); } /* not supported yet - will be soon in Chrome/Edge/Opera https://bugs.chromium.org/p/chromium/issues/detail?id=970285 */ }
@media (inverted-colors) { body { color: rgb(127,127,127); background-color: rgb(140,140,140); }  }
@media (orientation: portrait) { body { background-color: rgb(255,255,255); }  }
@media (min-resolution: 150dpi) { body { background-color: rgb(10,10,10); }  }
@media (update:slow) { body { color: #eee; } /* e-ink reader or low power mode */ }
@media (light-level: washed) { body { color: rgba( 127, 127, 127, 0.5 ); } /* bright environment */ }
@media (hover) { a:hover { color: #777; } }
@media (pointer) { a:hover { color: red; } }
@media (color-gamut: rec2020) { body { background-image: url('high-gamut-background.png'); }  }
@media (dynamic-range: high) { body { background-image: url('high-range-background.png'); }  }
@media (max-device-width: 499px) { a:link,a:visited { text-decoration: none; } }
</style>
<body>Hello World</body>

Challenges with Conformance

Source

dd8

All 22 comments

Hi: I think you're referencing CSS Media Queries 5 and related CSS color picking specs? Assuming I've understood this correctly, this would not be a concern for WCAG beyond the selections the content provider has made for the default presentation. These media queries were created to facilitate personalizing content to user preferences, as APA discussed with CSS during TPAC 2016. So, really a feature, not a bug.

sajkaj on 15 Jul 2020

I don't agree this should be closed. The issue here was about the CSS author doing stuff with media queries that cause problems for the user.

The user preferences media queries can be misused by authors - the examples below show how a stylesheet author can create accessibility problems by detecting and over-riding the user's preferences:

1) This media query in an author style sheet removes all images on Safari macOS when Inverted Colors is checked in the macOS Accessibility control panel:

@media (inverted-colors) { img { display:none; /* author: images look ugly when inverted - hide them */ }

2) This media query in an author style sheet disables high contrast mode (forced-colors is the standardized version of -ms-high-contrast)

@media (forced-colors) { body { forced-color-adjust: none; /* author: high contrast colors look ugly - ignore them */ }

3) This media query produces low contrast graphics when the user requests prefers-contrast:low (good) but also when the user requests prefers-contrast:high (bad) because low and high both evaluate to true in the MQ boolean context. This is a subtle authoring error:

@media (prefers-contrast) { img { opacity: 50%; /* author: reduce contrast if user wants low contrast */ }

https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-contrast

Display characteristics can also cause device specific accessibility problems.

4) It's possible to get accessibility naming errors that depend on device configuration - the code below displays a picture of loaf of bread on narrow screens (e.g. iPhone in portrait mode) but a picture of a Copenhagen neighbourhood on on wider screens (e.g. iPhone in landscape mode) so the accessible name is wrong on narrow screens:

<picture>
 <source media="(min-width: 1000px)" srcset="nyhavn-true-color.png">
 <source media="(min-width: 320px)" srcset="bread-true-color.png">
 <img src="nyhavn.jpeg" alt="Nyhavn, a Copenhagen neighbourhood">
</picture>

You can change the media queries used in the picture example above to produce the right name on modern displays, but the wrong accessible name on old 256 color displays that use indexed color:

<source media="not(color-index)" srcset="nyhavn-true-color.png">
<source media="(color-index)" srcset="bread-256-color.png">

If any device or preference specifc media queries are combined with CSS opacity: or RGBA colors with A < 1.0 or graphics with an alpha channel you get a huge number of potential color combinations.

dd8 on 15 Jul 2020

👍1

Originally in 2.1's development, I did suggest that the various permutations of a page (also relating to things like browser-sniffing etc) should be audited. This was softened in https://www.w3.org/TR/WCAG21/#cc2 to just cover "variation of the page [...] for various screen sizes".

Agree in principle (though there may be some nuances, like forced high contrast modes that don't allow author control) that variations beyond just viewport-size-related ones should be covered.

patrickhlauke on 15 Jul 2020

as for the fundamental "this can lead to a huge number of variations" ... yes, accessibility testing can be hard and laborious. generally, we'd ask clients that we're auditing sites for if there are any particular adaptations they've done/included in their styles, to at least be able to narrow down the testing.

i'd say there's also the idea that accessibility testing simply can't catch all possible variations in testing. unless a client explicitly states which adaptations they've made, and are then made aware of the implication (in terms of time and money) of validating EACH possible combination, auditing must be understood as a way to validate the "common" scenarios. and that an auditor, unless explicitly requested to do so, is not liable if a particular gnarly combination of device metrics/user preferences/system settings/environmental factors (e.g. network condition, light sensor reading, etc) leads to an inaccessible outcome that was not found.

patrickhlauke on 15 Jul 2020

Yes, although there are two types of testability issues at play here:

1) we can't test this because the client won't give us the time/budget to test all combinations
2) we can't test this because testing all combinations will take years / decades / centuries / aeons

The latter issue goes to the heart of testability in WCAG - as the number of ways authors can combine things in breaking ways increases, testability decreases.

dd8 on 15 Jul 2020

point 2 is basically true even without dragging in CSS. sites may do very specific things like IP-based custom content, sniffing specific browser/OS versions, etc. and anything involving the user entering some information (e.g. in a humble text input) may result in a myriad of different sets of data/responses being shown.

testing is, even today, simply a very selective / representative "slice". the more layers/combinations are possible, the thinner that slice will be. this is probably something that needs to be made clear to clients first and foremost, and something that needs to be clearly aknowledged in the challenges document AND in documents like https://www.w3.org/TR/WCAG-EM/

patrickhlauke on 15 Jul 2020

👍1

Regardless, I still don't see a WCAG issue here. No W3C specification attempts to police against misuse. As noted above, these CSS features are in the Media Queries 5 specification to meet an identified need. If a content provider misuses CSS to create content that violates WCAG, that single presentation constitutes the violation. The permutations Media Queries supports on behalf of users is not subject to WCAG, except perhaps to support a simple consistent mechanism to undo a suboptimal setting when trialing different combinations. If you believe better guidance on not misusing these Media Queries features is needed, that should be taken up with the CSS-WG, perhaps via the CSS Accessibility Task Force.

sajkaj on 16 Jul 2020

i thought the point of this discussion here was that this should be aknowledged in the "challenges" document (and, i'd add, in WCAG-EM). not that this is a "WCAG issue" that @dd8 is asking WCAG to address with an SC or similar...

patrickhlauke on 16 Jul 2020

@patrickhlauke, Challenges is about challenges with asserting conformance in the current binary, all or nothing conformance model, especially with respect to very large, complex, and highly dynamic sites. As the Abstract clearly says, it is intended as input toward a more useable and useful conformance approach in next gen WCAG.

sajkaj on 16 Jul 2020

so how is explicitly mentioning the various shapes/variations/differences that a site can take (based on user agent, specific settings or circumstances of the user's browser/OS, etc) NOT a challenge in asserting conformance since it increases the complexity of highly dynamic sites even further?

patrickhlauke on 16 Jul 2020

and no, this is not about "misuse". this is about the myriad of potential variations that these various technologies can lead to, often with unintended/unexpected consequences, and how they make it almost impossible to make a definitive pass/fail statement that covers ALL possible variations - since they simply realistically can't all be tested - and how even the most extensive auditing process can only hope to cover a very specific "slice" of complexity, a particular constrained view into a site and its various potential adaptations,

patrickhlauke on 16 Jul 2020

i'd move to reopen this issue, as to me it makes a valid point (and a much larger point beyond just CSS MQs)

patrickhlauke on 16 Jul 2020

OK, agree to reopen for further discussion, though I still don't agree on relevance. If we were to base conformance on all possible user environments we'd have to look at all aapi, all AT as well. Can you point to were any current (or historic) WCAG conformance includes user configuration choices?

sajkaj on 16 Jul 2020

Can you point to were any current (or historic) WCAG conformance includes user configuration choices?"

this is not about user configuration choices. a user can't help what their user agent string is, what dimensions their display are, what their light sensor is currently picking up, what the color gamut of their display is, what their IP address is, whether their display has a certain DPI density, whether their primary input is a touch screen or a pointer, etc. these are all factors that can be used by sites to change what they serve to the user. and yes, THEN there are user configuration choices for their whole OS, like preferring (or needing) a dark theme, or needing reduced-motion.

If we were to base conformance on all possible user environments we'd have to look at all aapi, all AT as well

but that's the point. since we can't, we need to aknowledge that whatever conformance statement made may not reflect all possible variables that can influence how a site changes/adapts to a myriad of different factors. nobody is saying auditors must be able to test all of these possible iterations, but just that a conformance claim MAY not cover every possible variation, and that this doesn't come as a surprise.

patrickhlauke on 16 Jul 2020

let's take a simpler example of what some sites used to do: a site detects that there's a touchscreen, and makes the silly assumption that it then doesn't need keyboard-capabilities anymore. auditor only tests on a non-touchscreen device, and the claim in the end says that yes, the site works fine for keyboard and passes 2.1.1. but does it?

yes, the auditor should have tested on a device with a touchscreen (e.g. a desktop with an additional external touchscreen, or a two-in-one laptop with both a touchscreen and trackpad/mouse/keyboard). but they didn't. so who's at fault? is the conformance claim wrong? i'd say yes, because it doesn't explicitly at least mention that it's only accurate for a particular set of circumstances and that there MAY be things that weren't even tested.

in short, who's at fault? at a high level, i'd say it's the conformance claim/process itself which does not even aknowledge/consider that the site may adapt to different devices/environments differently.

patrickhlauke on 16 Jul 2020

OK, that does make sense and may indeed have a place in Challenges. Care to suggest a few paragraphs? Meanwhile, feel free to color me dense! :)

sajkaj on 16 Jul 2020

not dense, just that ... modern websites are a lot more complex, and can be exponentially more complex, as they adapt/respond to different factors. and that just snowballs into an impossible set of variations that no auditor can hope to (easily) test.

i'll have a think about a possible para or two.

patrickhlauke on 16 Jul 2020

Thanks for helping clarify the, @patrickhlauke. And, I'll look forward to your text to get us started on addressing this. You've pointed me to thinking about it as being about the limit of shared responsibility, i.e. who/what spec/tech is responsible for what in the stack that is supposed to all work together for an accessible user experience. That's definitely a useful conversation.

sajkaj on 16 Jul 2020

i think it's actually potentially gotten so complex, that even site owners themselves aren't aware of the possible combinations/variations that their site may end up in. they may look at, say, the various CSS media queries in isolation, but not realise that they can combine in unexpected ways and lead them to some inaccessible (or just plain broken) combinations.

patrickhlauke on 16 Jul 2020

👍1

I'd agree this is relevant for the conformance challenges doc, it is similar to challenge 2 in the doc.

It contributes towards: How do we scope a conformance statement from a user-agent coverage point of view?

Silver is taking a different approach to 'accessibility supported', so this is relevant.

As Pat said, when testing we are always taking a 'slice', over time we've adapted our slice to include certain checks in a few user-agents. Generally we use particular device checks that we find catch the most issues over the most devices combinations. But it is a slice.

Secondarily, there is potentially an SC in not mis-using personalisation support.

alastc on 17 Jul 2020

Secondarily, there is potentially an SC in not mis-using personalisation support.

rather than misusing, i'd probably say there's the inverse - having a (AAA, i'd say) SC about heeding personalisation preference hints, or at the very least not standing in their way/trying to counteract them

(but again, for clarity, this issue here isn't just about personalisation, but about all the different metrics/properties/factors that let a site serve different content/styling/behavior)

patrickhlauke on 17 Jul 2020

In my experiments for Silver I came to the conclusion we might need something like:

ESSENTIAL METHODS?
SUPPLEMENTAL METHODS?
COMBINATIONS?
BONUS METHODS?

Why these? Based on a test with luminosity contrast (taking WCAG2.1) it was clear we at least need:

(essential?!)

Luminosity contrast Text & Images of Text
Luminosity contrast Non-Text
Luminosity contrast Focus Indicator

BUT, What about

(Bonus?!)

Luminosity scheme can be overridden (own CSS, stylus etc.)
Luminosity scheme can be changed by switch (created by author)

For TESTING AND METHODS we might need the following methods types?!

VISUALLY
CODE
VISUALLY&CODE COMBI???
PERSONALIZATION

So, back to this issue, personalization should be specifically mentioned in Silver and have a status on its own when testing and doing conformance.

If so, we might be able to cover these questions / testing procedures more easily.

Cheers!

(ps. experiment can be found here, three TABS present...: https://docs.google.com/spreadsheets/d/1yTpbJFsaadIbCigV15YK6bWB4xk_MoOgREjGMHHMVgQ/edit#gid=1983238719 )