Vscode: Notebook cell results fully unaccessible

[isidorn]

Refs: https://github.com/microsoft/vscode/issues/93742

1. Open a notebook file, run any query
2. Get results -> you can not focus on them and they are fully not accessible 🐛

I suggest that the down key would navigate to the results before navigating to the next cell.
I understand you are using a list of editors and that is the reason why the down navigates to the next editor but we need to think about how to make the results navigable to.

Adding label important since this makes the Notebooks not accessible.

[rebornix]

cc @misolori

[rebornix]

I suggest that the down key would navigate to the results before navigating to the next cell.

@isidorn Can you elaborate a bit more on this one? Usually the results are readonly, not sure how we are going to enable cursor focus on them.

If we allow users to tab through the editor and the results, can we call it accessible? For example, say the results are a list of links, users can navigate out of the editor and then navigate through the links in the output container. The workflow is then similar to the Interactive Playground.

[isidorn]

@rebornix even if the resuls are readonly you can make them focusable. For example the extensions view is also a list, and each element is "readonly" but each element can be focused onto.

Yes if you allow to tab through the editor and the results that is a good first step towards accessibility.
Interactive Playground imho is not a good experience since nothing gets read out in the results.

To recap here are the things which need to be done:
1) Make the results output focusable (either by down arrow or by tab)
2) Once the results are focusable users should be able to focus down into the results (by pressing down similar to how Tree allows users to move focus into the elements) and jump with focus from result to result

I am not sure how the extensions are providing results in what form. Can you maybe fill me in on that?
Alternative to 2) is that there is just one ariaLabel for the whole results area however that would not bring any structure and would be hard to use, though might be a good first step.

fyi @jrieken

[rebornix]

I am not sure how the extensions are providing results in what form. Can you maybe fill me in on that?

The output can come with multiple mime types, simple ones like plain text, images, markdown; richer ones like HTML, SVG, etc. Not all mime types are accessible, and it's not something we can push the extension authors to do, it would be a huge overhead if @jrieken needs to make the GitHub html rendering fully accessible.

However we can ask extensions to provide accessible mime types. For example, the GitHub notebook should always return at least two mimetypes: markdown and html, the markdown mime type output will be rendered by the core in the main thread, since it's using our own markdown renderer, we can make it accessible, just like Hover. When there are screen reader attached, we should always choose accessible mime types to render as a fancy interactive plot might not be helpful from accessibility's perspective.

[isidorn]

Yes I like both those points. Extensions should be aware of accessiblity and should try to return accessible mime types if possible.
When we see a screen reader is attached we pick accessible mime types by default.

[MarcoZehe]

I am all for implementing this "choose accessible mime type" approach for notebooks when AT is detected. This proves to be another great tool in VS Code, and we should make it's as accessible as possible.

[zersiax]

I am very interested in this one. Currently, working with notebooks in most forms is tricky at best. Making this work in VS Code would open doors to many users who currently want to use these accessibly, but can't.
I myself I am unfortunately completely unskilled in the use of Jupyter notebooks, and can therefore not really advise how this would best be done.
I aim to remedy this in the ocming days to better advise about possible usage patterns that would work well.

[isidorn]

@roblourens @ctmayn looking at the PR this fixes the Focus of the note book output which is great! Thanks!
However nothing about the second part of our discussion is captured by the PR. Namely how we should show results in plain text once screen reader is detected. More in the two comments by me and @rebornix above
https://github.com/microsoft/vscode/issues/93896#issuecomment-610223080
https://github.com/microsoft/vscode/issues/93896#issuecomment-610525042

Should I open a new issue to capture this or should we reopen this one?

[roblourens]

Not sure whether that's the right thing to do. We can't know which mimetype is most accessible, and html like in the case of the github extension will be just as accessible as a webpage with a simple list, like you can navigate by paragraph or list element and the screenreader will read it and so on.

Also if we pick a different mimetype by default then it will just block the extension from actually providing a html output designed with accessibility in mind, right?

I guess on one hand some html outputs will be incomprehensible to a screenreader, and on the other hand, some plain text versions will be inferior. Maybe the renderer can tag some output as 'accessible'.

[isidorn]

@roblourens I like that idea of a renderer being able to tag an output as accessible
@jrieken how is the renderer connected, is it via some API. Can we add some accessibility fields there as well?

[roblourens]

Talked about this with Peng, and we don't really have a conclusion. It feels awkward to me to have API to say "this is accessible" when we still don't really understand how some output types could be made to be accessible. Let's wait and see how things develop.

[rebornix]

Currently behavior: if the workbench is in accessibility mode / screen reader attached, text/markdown will have higher priority than text/html, which is our heuristics and might not be true. Tagging an output mimetype as "accessible" is appealing but I think it should not be done by the extension itself, otherwise it's just another property for changing mimetype display order.

In GH Notebook, it's fairly easy to make the text/markdown output to be accessible: adding self explanatory aria labels for links and users can easily tab through the results (also no "loads more ... ").