Lighthouse-ci: Which accessibility audits relate to certain WCAG levels?

[JackMidd-Amido]

Hey,

I am using lighthouse-ci to do an accessibility audit in our pipelines and we have a requirement to meet WCAG 2.1 Level AA rules.
I am aware that Lighthouse and Lighthouse-CI do not verify full compliance with the WCAG requirements, but I would like to know which audits in lighthouse-ci relate to each WCAG level.

I have got a list of all the audits that are available in Lighthouse-CI which can be seen here

I have also found in lighthouse core github repo that there is a list of gatherers that map to A and AA levels. However, these do not directly map to the ones that lighthouse-ci has available.

I have also looked in axe-core (which I believe Lighthouse utilises) and have a list of the rules for each level, but again I cannot see an easy way to map these to lighthouse-ci audits.

Has anyone got a definitive list of lighthouse-ci audits and their mapping to WCAG levels? Have I missed something?

Thanks,
Jack

[patrickhulce]

Thanks for filing @JackMidd-Amido! This is really a Lighthouse core question. Lighthouse CI just runs the default Lighthouse configuration for whatever Lighthouse version is used (5.5.0 for v0.3.x and v6.0.0 for v0.4.x).

See https://github.com/GoogleChrome/lighthouse/issues/9507 and https://github.com/GoogleChrome/lighthouse/issues/10202 for previous answers. You've already found most of the information we have available. There is no easy table that exists to do the mapping, but the information is found in the documentation linked in each audit.

The relationship of each audit to WCAG guidelines is covered in the Learn More link that accompanies each. Example below.

[JackMidd-Amido]

Thanks for the reply, @patrickhulce

I was hoping for that easy mapping table to be honest :), but saying that I understand the complexity and am happy to do additional reading/research around your response to get the list of audits I require.

Thanks again, awesome tool I will continue to use on all future projects!

[JackMidd-Amido]

I realised that each of the audits that can be seen in lighthouse-ci config has the same ID as those on Axe rules list: https://dequeuniversity.com/rules/axe/3.5 (should have realised this a lot sooner...)

Realising this allowed me to search each audit by appending the audit ID to the rules list URL. E.g. https://dequeuniversity.com/rules/axe/3.5/accesskeys
As Patrick showed in his screenshot, there is a section on the right that shows the requirements (I.e. what WCAG level it relates to).

Using this, I mapped each audit to requirements as documented in Axe:

'accessibility': {
      title: str_(UIStrings.a11yCategoryTitle),
      description: str_(UIStrings.a11yCategoryDescription),
      manualDescription: str_(UIStrings.a11yCategoryManualDescription),
      // Audit weights are meant to match the aXe scoring system of
      // minor, serious, and critical.
      // See the audits listed at dequeuniversity.com/rules/axe/3.2.
      // Click on an audit and check the right hand column to see its severity.
      auditRefs: [
        {id: 'accesskeys', weight: 3, group: 'a11y-navigation'}, // https://dequeuniversity.com/rules/axe/3.5/accesskeys - Best Practice
        {id: 'aria-allowed-attr', weight: 10, group: 'a11y-aria'}, // https://dequeuniversity.com/rules/axe/3.5/aria-allowed-attr - WCAG A
        {id: 'aria-required-attr', weight: 10, group: 'a11y-aria'}, // https://dequeuniversity.com/rules/axe/3.5/aria-required-attr - WCAG A
        {id: 'aria-required-children', weight: 10, group: 'a11y-aria'}, // https://dequeuniversity.com/rules/axe/3.5/aria-required-children - WCAG A
        {id: 'aria-required-parent', weight: 10, group: 'a11y-aria'}, // https://dequeuniversity.com/rules/axe/3.5/aria-required-parent - WCAG A
        {id: 'aria-roles', weight: 10, group: 'a11y-aria'}, // https://dequeuniversity.com/rules/axe/3.5/aria-roles - WCAG A
        {id: 'aria-valid-attr-value', weight: 10, group: 'a11y-aria'}, // https://dequeuniversity.com/rules/axe/3.5/aria-valid-attr-value - WCAG A
        {id: 'aria-valid-attr', weight: 10, group: 'a11y-aria'}, // https://dequeuniversity.com/rules/axe/3.5/aria-valid-attr - WCAG A 
        {id: 'audio-caption', weight: 10, group: 'a11y-audio-video'}, // https://dequeuniversity.com/rules/axe/3.5/audio-caption - Section 508 must
        {id: 'button-name', weight: 10, group: 'a11y-names-labels'}, // https://dequeuniversity.com/rules/axe/3.5/button-name - Section 508 must
        {id: 'bypass', weight: 3, group: 'a11y-navigation'}, // https://dequeuniversity.com/rules/axe/3.5/bypassWCAG - Section 508 must
        {id: 'color-contrast', weight: 3, group: 'a11y-color-contrast'}, // https://dequeuniversity.com/rules/axe/3.5/color-contrast - WCAG AA
        {id: 'definition-list', weight: 3, group: 'a11y-tables-lists'}, // https://dequeuniversity.com/rules/axe/3.5/definition-list - WCAG A
        {id: 'dlitem', weight: 3, group: 'a11y-tables-lists'}, // https://dequeuniversity.com/rules/axe/3.5/dlitem - WCAG A
        {id: 'document-title', weight: 3, group: 'a11y-names-labels'}, // https://dequeuniversity.com/rules/axe/3.5/document-title - WCAG A
        {id: 'duplicate-id', weight: 1, group: 'a11y-best-practices'}, // https://dequeuniversity.com/rules/axe/3.5/duplicate-id - WCAG A
        {id: 'frame-title', weight: 3, group: 'a11y-names-labels'}, // https://dequeuniversity.com/rules/axe/3.5/frame-title - Section 508 must
        {id: 'html-has-lang', weight: 3, group: 'a11y-language'}, // https://dequeuniversity.com/rules/axe/3.5/html-has-lang - WCAG A
        {id: 'html-lang-valid', weight: 3, group: 'a11y-language'}, // https://dequeuniversity.com/rules/axe/3.5/html-lang-valid - WCAG A 
        {id: 'image-alt', weight: 10, group: 'a11y-names-labels'}, // https://dequeuniversity.com/rules/axe/3.5/image-alt - Section 508 must
        {id: 'input-image-alt', weight: 10, group: 'a11y-names-labels'}, // https://dequeuniversity.com/rules/axe/3.5/input-image-alt - Section 509 must
        {id: 'label', weight: 10, group: 'a11y-names-labels'}, // https://dequeuniversity.com/rules/axe/3.5/label - Section 508 must
        {id: 'layout-table', weight: 3, group: 'a11y-tables-lists'}, // https://dequeuniversity.com/rules/axe/3.5/layout-table - WCAG A
        {id: 'link-name', weight: 3, group: 'a11y-names-labels'}, // https://dequeuniversity.com/rules/axe/3.5/link-name - Section 508 must
        {id: 'list', weight: 3, group: 'a11y-tables-lists'}, // https://dequeuniversity.com/rules/axe/3.5/list - WCAG A
        {id: 'listitem', weight: 3, group: 'a11y-tables-lists'}, // https://dequeuniversity.com/rules/axe/3.5/listitem - WCAG A
        {id: 'meta-refresh', weight: 10, group: 'a11y-best-practices'}, // https://dequeuniversity.com/rules/axe/3.5/meta-refresh - WCAG AA
        {id: 'meta-viewport', weight: 10, group: 'a11y-best-practices'}, // https://dequeuniversity.com/rules/axe/3.5/meta-viewport - Best Practice
        {id: 'object-alt', weight: 3, group: 'a11y-names-labels'}, // https://dequeuniversity.com/rules/axe/3.5/object-alt - Section 508 must
        {id: 'tabindex', weight: 3, group: 'a11y-navigation'}, // https://dequeuniversity.com/rules/axe/3.5/tabindex - Best Practice
        {id: 'td-headers-attr', weight: 3, group: 'a11y-tables-lists'}, // https://dequeuniversity.com/rules/axe/3.5/td-headers-attr - Section 508 must
        {id: 'th-has-data-cells', weight: 3, group: 'a11y-tables-lists'}, // https://dequeuniversity.com/rules/axe/3.5/th-has-data-cells - Section 508 must
        {id: 'valid-lang', weight: 3, group: 'a11y-language'}, // https://dequeuniversity.com/rules/axe/3.5/valid-lang - WCAG AA
        {id: 'video-caption', weight: 10, group: 'a11y-audio-video'}, // https://dequeuniversity.com/rules/axe/3.5/video-caption - WCAG AA
        {id: 'video-description', weight: 10, group: 'a11y-audio-video'}, // not in Axe 3.5, latest is 3.3 - https://dequeuniversity.com/rules/axe/3.3/video-description - WCAG AA
        // Manual audits
        {id: 'logical-tab-order', weight: 0},
        {id: 'focusable-controls', weight: 0},
        {id: 'interactive-element-affordance', weight: 0},
        {id: 'managed-focus', weight: 0},
        {id: 'focus-traps', weight: 0},
        {id: 'custom-controls-labels', weight: 0},
        {id: 'custom-controls-roles', weight: 0},
        {id: 'visual-order-follows-dom', weight: 0},
        {id: 'offscreen-content-hidden', weight: 0},
        {id: 'heading-levels', weight: 0},
        {id: 'use-landmarks', weight: 0},
      ],
    }

This has helped me with my requirements. Hopefully it will help someone else down the road.

Jack

[patrickhulce]

Many thanks for sharing your findings @JackMidd-Amido! I'm sure someone else will benefit from it in the future :)