Storybook: Addon-a11y: Manual doesn't work

[donaldpipowitch]

Describe the bug
if you add

FullApp.parameters = {
  a11y: {
    manual: true
  }
};

to this example https://github.com/storybookjs/storybook/blob/next/examples/cra-kitchen-sink/src/stories/App.stories.js#L16 you'll see that the a11y tests are run even though "manual: true" is set.

To Reproduce
Steps to reproduce the behavior: See above.

Expected behavior
We should be in the manual state where no aXe test is run automatically.

Screenshots
-

Code snippets
I tested a bunch of stuff in order to debug this and fix this on my own, but I didn't came far.

There might be two problems contributing to this...?
https://github.com/storybookjs/storybook/blob/4867e539f68478f878838d270ce34f8c1e86809f/addons/a11y/src/components/A11YPanel.tsx#L54-L64
It once looked like this triggers _initial_, _manual_ and later _initial_ again...?

It also looks that in some places (for example I think in the following code snippet - but I couldn't find all places) the EVENTS.REQUEST is triggered without having a check for _manual_?

https://github.com/storybookjs/storybook/blob/4867e539f68478f878838d270ce34f8c1e86809f/addons/a11y/src/components/A11yContext.tsx#L67-L85

System:
next branch.

Environment Info:

System:
OS: macOS 10.15.6
CPU: (16) x64 Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Binaries:
Node: 10.16.0 - ~/.nvm/versions/node/v10.16.0/bin/node
Yarn: 1.22.4 - ~/.yarn/bin/yarn
npm: 6.14.4 - ~/.nvm/versions/node/v10.16.0/bin/npm
Browsers:
Chrome: 84.0.4147.105
Safari: 13.1.2

Additional context
-

[shilman]

cc @tooppaaa @tmeasday

[tooppaaa]

I saw that manual was not working when working on a11y as well.

I'm curious about your use case, I've got a lot of stories, relying on this addon a lot and, never used manual.

The fix isn't that hard but it's a good occasion to me to challenge this feature now we know someone uses it :)

[donaldpipowitch]

I'm curious about your use case, I've got a lot of stories, relying on this addon a lot and, never used manual.

My use case was _big_ stories (documenting pages with complex tables and filters for example). During v5 when I navigated though big stories the a11y check was blocking the UI thread quite heavily which made using Storybook very hard.
So I didn't want to run the checks automatically, but keep the functionality for manual runs.

I only had a quick look at v6 so far, but it looked like the a11y checks became faster. Or are they run now in web workers even and don't block the UI thread/the navigation anymore...? If that is the case the manual option could maybe even be removed? But if the checks are still blocking, I'd like to keep the option.

[tooppaaa]

Of course, performance is a real concern ! Can you give a try on your large stories ?

[donaldpipowitch]

I just tried it. Navigation between stories seems to be very fast - but I think because the a11y check runs too early (the _big_ DOM was not rendered yet, that's why the check is fast and also navigation is fast; also I get back incomplete results because of this).

When I completely refresh such a story the a11y check is run correctly and if you try to scroll inside the story while the check is run I can see that it is still blocking the UI. Now the results also seem to be complete.

tl;dr:

• Big stories still need long time to test for the a11y check and that would block the UI.
• The a11y check is run too early when you navigate between long stories. (You get different results, if you make a hard refresh and the big story is the very first one which will be rendered/checked.)

[tooppaaa]

Thanks a lot for your feedback, very helpful !
I'll try to work on this but that might not be until next week. Want to help on this @donaldpipowitch ?

@shilman not blocker for 6.0, can that be a patch some time later ?

[shilman]

@tooppaaa not a blocker and i think @tmeasday also had thoughts on addon-a11y implementation

[donaldpipowitch]

@tooppaaa I'll try what I can (definitely test it at least). But maybe not within the next two weeks (vacation ❤️).

[shilman]

¡Ay Caramba!! I just released https://github.com/storybookjs/storybook/releases/tag/v6.0.13 containing PR #12003 that references this issue. Upgrade today to try it out!

Closing this issue. Please re-open if you think there's still more to do.