Nvda: Implement a mirror driver for comunicating with the display driver (OSM) in an installed version of NVDA.

[Adriani90]

Is your feature request related to a problem? Please describe.

There are often problems when trying to make an application accessible, especially when the application is very old and/or it is explicitely needed in corporate environments.
Also handling mouse cursor with NVDA is often unreliable in different applications.
Another problem are situations where multiple screens are used and/or multiple braille displays need to be connected to NVDA. A mirror driver would open up a lot of possibilities for new features and would also make NVDA more flexible.

Describe the solution you'd like

A mirror driver according to
https://docs.microsoft.com/en-us/windows-hardware/drivers/display/mirror-drivers
which communicates with the display driver, gathering the correct information of elements on the display and mirroring them in a virtual buffer or anything like that.
I would propose to implement this as dependency or as submodule in NVDA so that it can be improved independently. It might be useful for other use cases as well, not only for NVDA but also for braille devices like canute etc. But starting the discussion here is a good beginning I think.

Describe alternatives you've considered

none

Additional context

Especially for people working in governamental organizations where application often are outdated, they have to struggle alot with applications which are not accessible. They often lose their jobs or are not satisfied with their jobs because they cannot perform the same tasks as good as their sigted coleagues.
Also modern applications (especially in financial environment), accessibility is considered too late and it is sometimes not possible to open the whole source code of an application and change it entirely. I am of the opinion that we should raise awareness of developers with regard to accessibility, but we also need to keep in mind the cost factors which are involved especially in already developed applications which are already comonly used and are a standard for many companies.
But as I said, I think this would open up a very big variety of new approaches for new and innovative features.

[josephsl]

Hi,

Three important issues:

1. With the mirror driver, one must restart the computer in order to finish installing NVDA. Do we need this?
2. Although it is important to remind ourselves that people would install NVDA on their computers, some may prefer to use NVDA as a portable screen reader, and some folks may ask NVDA to implement this in portable copy as well. We then run into the same situation as portable JAWS. Do we want this experience?
3. Modern apps (particularly universal apps in Windows 10) are written in such a way that mirror drivers cannot be used. As some app developers are migrating their apps to be based on modern technologies (particularly using web technologies), the case for requiring mirror drivers may shrink.

Another thought to consider: there is so much a screen reader can do when it comes to gathering information from an app. What makes a bigger difference is mindset of app developers to make their apps accessible. This was shown last year with a United States Supreme Court refusing to take on a case on web accessibility (Domino's Pizza). Therefore I would advise rather than just throwing in mirror display drivers, outreach to app developers should be done at the same time.

Thanks.

[OzancanKaratas]

There is an option can be provided in the NVDA's Tools menu or General section of Settings. In this way, NVDA runs a DPInst.exe copy when the “Install display mirror driver” button was clicked. I think this installation process should be placed in the Tools menu such as COM Registration Fixing tool.

[DrSooom]

This is an obsolete technology, which already significantly changed how the Windows GUI looked like on Windows Vista in 2007 (basic theme/design instead of Aero Glass). And it's not part of Windows 8. Furthermore the windows-own Magnifier requires Aero Glass on Windows 7. Thus using NVDA with an active mirror display driver and the Windows-own Magnifier (in full-screen mode) will not work at the same time. Read also this Microsoft Support article.

So: A clear no. And Windows 7 and 8/8.1 machines should not be used after 2023 due to security reasons. (Note: Companies can purchase an additional ESU for Windows 7 until January 2023.)

[Adriani90]

1. With the mirror driver, one must restart the computer in order to finish installing NVDA. Do we need this?

I guess this is not a big problem. If we tell the user at the finish install promt something like "NVDA has been successfully installed. In order to use the mirror driver, a restart is needed."

2. Although it is important to remind ourselves that people would install NVDA on their computers, some may prefer to use NVDA as a portable screen reader, and some folks may ask NVDA to implement this in portable copy as well. We then run into the same situation as portable JAWS. Do we want this experience?

It is actually not possible to implement the mirror driver in a portable version because the driver needs to add and change registry entries. Thus, the driver is simply not available for portable versions. This is also now the case with other features like speaking on protected screens and other features which are not working in the portable version of NVDA.

3. Modern apps (particularly universal apps in Windows 10) are written in such a way that mirror drivers cannot be used. As some app developers are migrating their apps to be based on modern technologies (particularly using web technologies), the case for requiring mirror drivers may shrink.

This might be right for mainstream apps, but not for apps used in corporate environment. There are still lots of apps which are not using web content and this will be the case at least for the next 5 years in my view. And as I mentioned, there are still lots of organizations which are not updating their apps as fast an users would do privately. So the question is, do we want to continue endless discussions with developers to rewrite their old apps which will not happen (i.e. SAP, My PBX, Cisco etc.)? In my opinion we should try to achieve as much flexibility as possible while continuing to show developers what it means to create a really accessible application. But in case of SAP, rewriting the whole application environment would cost millions of US Dollars. No one would be wiling to do that. But there are many blind users out there who would benefit from making such apps more accessible through such a mirror driver.

Another thought to consider: there is so much a screen reader can do when it comes to gathering information from an app. What makes a bigger difference is mindset of app developers to make their apps accessible. This was shown last year with a United States Supreme Court refusing to take on a case on web accessibility (Domino's Pizza). Therefore I would advise rather than just throwing in mirror display drivers, outreach to app developers should be done at the same time.

I do not say that we should stop raising awarenessfor accessibility. More than that, I would rather say implementing such a mirror driver would open up many possibilities which would lead to much more contact points between developrers of assistive technologies like NV Access and developers of normal apps. We could tell them, "look, your app is not accessible and it is too late to rewrite the whole app environment. But let's try to work together to make it accessible through the mirror driver at least". I am sure that the app developer would certainly consider accessibility much earlier when developing a new app after such discussions.

The driver could be implemented as an optional setting in advanced settings. And in case of scripts or addons which are relying on the mirror driver, they could be loaded from the scratchpad directory.

[Adriani90]

@DrSooom mirror drivers are also working on Windows 10, Jaws use off screen model still on Windows 10 and you can adapt them as you want. API hooking should still remain the higher priority here. What I am talking about is just to create more flexibility and room for more features for people who really need it.

[Adriani90]

And if you make it optional, well users who don't need it can just turn it off.

[josephsl]

Hi,

Imagine someone installing NVDA who is very new to computers. How can you describe what a "mirror driver" is?

As for enterprise-level apps, I still believe it is up to app developers to change their mindset. It isn't that there isn't a good technology for app accessibility - it is culture that should be discussed.

I get a feeling that this is a duplicate of another issue. If so, I propose closing this one unless the duplicated one isn't offering any additional info.

Thanks.

[josephsl]

Hi,

As for portable copy: one can interact with elevated screens if one runs the portable NVDA in admin mode (not advised).

Thanks.

[Adriani90]

In any case, restarting the computer after an installation is not really something that would be a show stoper here.

Regarding mindset, I agree with you. However, we can only change the mindset of developers if we work together with them. just telling them it is not possible to make you app accessible because you do not expose anything to the APIs is just too less. Actually OSM is not a bad technology. And we could discuss about more possibilities with the app developer on how to achieve more accessibility for their apps while providing more flexibility in professional environments.

In my view, implementing such a driver does not mean that we stop talking to developers of mainstream apps or what ever.

We could also distribute such a driver like the controller.dll. The app developer himself could add the required OSM informations to the driver and distribute it with the application itself. I think it does not really have to be part of NVDA.

Any kind of implementation of such a driver would just open new windows instead of waiting for the ideal world to come.

Von: Joseph Lee [e-mail]
Gesendet: Donnerstag, 23. Januar 2020 20:16
An: nvaccess/nvda [e-mail]
Cc: Adriani90 [e-mail]; Author [e-mail]
Betreff: Re: [nvaccess/nvda] Implement a mirror driver for comunicating with the display driver (OSM) in an installed version of NVDA. (#10715)

Hi,

Imagine someone installing NVDA who is very new to computers. How can you describe what a "mirror driver" is?

As for enterprise-level apps, I still believe it is up to app developers to change their mindset. It isn't that there isn't a good technology for app accessibility - it is culture that should be discussed.

I get a feeling that this is a duplicate of another issue. If so, I propose closing this one unless the duplicated one isn't offering any additional info.

Thanks.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/nvaccess/nvda/issues/10715?email_source=notifications&email_token=AGVCP4MUYRCH2GRMBBZ2I7DQ7HUGJA5CNFSM4KKXTQK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJYQFRQ#issuecomment-577831622 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AGVCP4IM4Z6PUZJPGYHUKNTQ7HUGJANCNFSM4KKXTQKQ . https://github.com/notifications/beacon/AGVCP4PY3W3CGON45WHYWTTQ7HUGJA5CNFSM4KKXTQK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJYQFRQ.gif

[josephsl]

Hi,

My issue with OSM is teaching and assumptions: by adding OSM, people may think off-screen model is superior to accessibility API's when the other route may offer superior mindset, performance, and future-proofing.

Thanks.

[dingpengyu]

Hi,

My issue with OSM is teaching and assumptions: by adding OSM, people may think off-screen model is superior to accessibility API's when the other route may offer superior mindset, performance, and future-proofing.

Thanks.
hello josephsl
I personally think that NVDA should support OSM.
UIA and iaccessible are not everything,NVDA should have stronger screen capture capabilities.
Compatibility and continuation is my summary above.
thanks

[dingpengyu]

hi
At present, NVDA developers have been avoiding the technology of graphics acquisition, mainly because developers believe that driving acquisition is a kind of obsolete technology, which will affect the promotion and development of barrier-free API technology. However, traditional win32 applications that do not currently have accessibility APIs still account for the majority. Especially in Asia. It may be that this technology may be completely phased out in the future, as developers say, but it is currently impossible to phase out. So there must be a lot of technology. I still support this ISSUE, because the ideal is good, and the actual user experience is more important. After all, with this win10 technology, permissions may be restricted in the near future. But jaws still adjusted through technology. This technology has been implemented on win10. So it can be proved that this technology is still suitable for the current development environment of screen readers.

[michaelDCurran]

NVDA already hooks GDI calls within each process, in order to create an off-screen model. A part from being able to restructure the code to be OS-wide rather than process-specific, I do not believe we would get any extra information with a mirror driver over our current approach. If there are things that our off-screen model cannot track that another Screen Reader can, then the issue is most likely in our existing displayModel code and can be fixed. For example, we only track GDI. We have made no effort yet to track APIs such as Direct2D.
Mirror drivers do not work on Windows 8 and higher: https://docs.microsoft.com/en-us/windows-hardware/drivers/display/mirror-drivers
Mirror drivers also cannot work with certain display presentation settings in Windows 7.
Microsoft does have a replacement in Windows 8 and higher called the GDI Accessibility Driver. But it is my understanding that that API only gives access to GDI calls, which we are already doing on a per-process level anyway.
By all means, if someone in the community wants to try and improve our existing displayModel code, then please go ahead. But we will not be implementing a Mirror Driver.

[Brian1Gaff]

But as has been said many many times since the start of nvda, this does mean
that gradually the portable version of nvda will lose functionality. One of
the good ideas I noticed that Dolphin uses is to make the driver available
stand alone and free, so that it can be used if detected by their software.
I'm thinking its a pity that the industry as a whole does not specify a
driver like this already in Windows to cope with these issues. I know Jaws
has one also and it loads even if Jaws is not running.
Its a shame because back in the old days everyone had to have a screen
scraping display chain driver in order to give you a chance of
accessibility.
Brian

[email protected]
Sent via blueyonder.
Please address personal E-mail to:-
[email protected], putting 'Brian Gaff'
in the display name field.

Newsgroup monitored: alt.comp.blind-users

[Adriani90]

Mick thanks for your comments. This clarifies the current state for me now. We have some people in Germany who definitely would like to contribute to improve the display model code. Let's see if they come up with some proposals over the next months. I am in contact with them and will forward this information.

[Brian1Gaff]

In the first instance, eyes you need to restart, but as it loads every time
no matter what you do thereafter its probably quite minor. A bigger fear is
if multiple drivers of this kind from different manufacturers have to chain
together in different orders, it can make for a pretty unusable computer
which then needs to be started in safe mode and the offending drivers
stopped from running before a reboot. This used to happen a lot back in the
old days where Dolphin and FS each had their own one.
A real pain for companies trying to sell screenreaders.
Also so embedded in the operating system load these are, there is already
resistance to open source software on corporate machines, what would this
look like to an admin?

Another thing. It really would be great if Microsoft themselves put in a
driver which was certified by them which could fill in these gaps that, lets
face it every screenreader could use. Maybe it already exists as part of
Narrator?
Brian

[email protected]
Sent via blueyonder.
Please address personal E-mail to:-
[email protected], putting 'Brian Gaff'
in the display name field.

Newsgroup monitored: alt.comp.blind-users

[Brian1Gaff]

Yes and many of us probably don't care about windows 7 having been abandoned
as it in practice makes little difference until the design of crucial
software demands we change. That is and has to be realised as a common trait
in blind computer users lives.
Brian

[email protected]
Sent via blueyonder.
Please address personal E-mail to:-
[email protected], putting 'Brian Gaff'
in the display name field.

Newsgroup monitored: alt.comp.blind-users

[Brian1Gaff]

I think the old one was way back before the github days, and maybe the world
has now moved on, who knows. I think myself that it should be off by default
with the advice that if you are running an installed copy and there are
issues with software you have to use, you might want to try it.
Brian

[email protected]
Sent via blueyonder.
Please address personal E-mail to:-
[email protected], putting 'Brian Gaff'
in the display name field.

Newsgroup monitored: alt.comp.blind-users

[Adriani90]

@michaelDCurran is theoretically possible even to track APIs such as Open GL or direct x? I understand that direct x might have performance issues, for texture mapping it seems it is used quite often though.