Storybook: A11y Addon error option

[evanmwillhite]

Is your feature request related to a problem? Please describe.
Can we pass some extra options in the a11y addon to allow for errors to be passed actively to the component viewer (like an eslint error)?

Describe the solution you'd like
The axe tooling already provides the detail/content we would need, and it would be awesome to have errors "stop" development (see screenshot below) based on the level (Serious, Moderate, etc.) - options something like:

error: true,
errorLevel: 'Serious'

Describe alternatives you've considered
CLI tests (duplicate efforts), storyshots testing (don't really need jest, etc. just for this)

Are you able to assist bring the feature to reality?
Myself or the team I work with may be able to help but wasn't sure if this was a desired/possible feature.

Additional context

[shilman]

cc @CodeByAlex

[CodeByAlex]

I like this a lot and while the addon wasn’t originally intended to stop development, I could see how this could be a neat feature to make the errors more well know to the dev if they choose to opt in. Would you and your team want to tackle this addition @evanmwillhite?

[evanmwillhite]

Yay! Thanks for the feedback, and yes, we can definitely poke at it from our end @CodeByAlex. I'll see what progress we can make this week.

[CodeByAlex]

@evanmwillhite it might be cool if you could show some type of error state but still display the component. One of the cool features about the a11y addon is that it points out the failed a11y rule on the component so having a hybrid approach could be neat - not showing the component at all may make it difficult to diagnose. @domyen do you have any ideas about this from the design perspective?

[evanmwillhite]

@CodeByAlex I love that idea, and I think I saw something like that mentioned in the official roadmap too. I think we may still focus on the approach above from our end, as we are wanting to enforce accessibility standards at the same level as coding standards. But we'll definitely make that an "opt-in" thing and not default. Also, the work we do to interact with the component viewer will likely help inform what you've mentioned as well.

[evanmwillhite]

We're still actively working on this - hoping to make progress in the next few weeks.

[tmeasday]

I don't want to comment on the design side of things, but just thinking out loud about the question of how you would render an arbitrary error in the preview:

An error is rendered by the StoryRenderer in one of two main cases:

1. The Storybook itself is misconfigured and the store has been loaded up with an error state. [This is a global error and probably not appropriate].

2. The story itself throws an error or misbehaves during rendering.

I suspect leveraging 2. makes sense here. Naively what you could do is add a story decorator that looks something like:

const a11yErrorDecorator = (storyFn) => isBroken(storyFn) ? throw new Error(...) : storyFn();

I guess the problem here is that the storyFn() actually needs to be rendered to the screen before we can actually tell if the story is broken (is that correct)? This will be a problem no matter what if you want to replace the story with an error (you'll "need" to show a flash of the story before rendering the error).

With that in mind to achieve the above you'd want to do something like:

const a11yErrorDecorator = (storyFn) => wasBrokenOnLastRender ? throw new Error(...) : storyFn();

And you'd want to:

1. Set wasBrokenOnLastRender via the usual way a11y is storing story info.
2. Ensure the Story re-renders if it was broken, after a11y has run. You can the force-rerender event to trigger this.

You'd want to be careful to avoid an infinite loop of course.

[evanmwillhite]

Thank you so much for this guidance @tmeasday!