Dxvk: [D3D9] DotNet, D3D9 and multiple adapters

[redmcg]

This is related to the following issues:

• https://github.com/doitsujin/dxvk/issues/1408; and
• https://github.com/ValveSoftware/Proton/pull/3513

Looking through the now open sourced WPF code, I found why the rendering issues in the Elite Dangerous Launcher occur:
https://github.com/dotnet/wpf/blob/c4e74175da6108104491e1ae64982444d7c8713d/src/Microsoft.DotNet.Wpf/src/WpfGfx/core/common/display.cpp#L1629

Basically, if D3D9 returns more adapters than EnumDisplayDevices, DotNet assumes a "mode change" and attempts initialisation again.

I wrote a tool, named d3d9Test.exe, that can demonstrate the issue. The source is in a gist located here:
https://gist.github.com/redmcg/5b11d2d18ff29da5d9d4886b2e1699a1

In my set-up, I have a dedicated Nvidia GPU and an Intel iGPU. When I have the iGPU disabled, I get the following output from the d3d9Test (with DXVK):

D3D9 says you have 2 adapter(s):
 - \\.\DISPLAY1 (GeForce GTX 1050 Ti)
 - \\.\DISPLAY1 (Intel(R) UHD Graphics 630 (Coffeelake 3x8 GT2))

EnumDisplayDevices:
- 0: \\.\DISPLAY1 (OK)
- 1: \\.\DISPLAY2 (not attached to desktop)

EnumDisplayDevices says 1 device(s) are OK

DotNet assumes a mode change, which causes issues in rendering, when m_uD3DAdapterCount > m_rgpDisplaysCount (2 > 1)
See https://github.com/dotnet/wpf/blob/c4e74175da6108104491e1ae64982444d7c8713d/src/Microsoft.DotNet.Wpf/src/WpfGfx/core/common/display.cpp#L1629

And the Elite Dangerous launcher has a rendering issue. But if I set my display to extended, I get the following output:

D3D9 says you have 2 adapter(s):
 - \\.\DISPLAY1 (GeForce GTX 1050 Ti)
 - \\.\DISPLAY1 (Intel(R) UHD Graphics 630 (Coffeelake 3x8 GT2))

EnumDisplayDevices:
- 0: \\.\DISPLAY1 (OK)
- 1: \\.\DISPLAY2 (OK)

EnumDisplayDevices says 2 device(s) are OK

DotNet should be happy

And the launcher renders OK.

So I believe DXVK should return only the one adapter if EnumDisplayDevices is returning only one Display.

Here's the output from the same setup on Windows.

With iGPU disabled:

D3D9 says you have 1 adapter(s):
 - \\.\DISPLAY1 (NVIDIA GeForce GTX 1050 Ti)

EnumDisplayDevices:
- 0: \\.\DISPLAY1 (OK)
- 1: \\.\DISPLAY2 (not attached to desktop)
- 2: \\.\DISPLAY3 (not attached to desktop)
- 3: \\.\DISPLAY4 (not attached to desktop)
- 4: \\.\DISPLAY5 (not attached to desktop)
- 5: \\.\DISPLAY6 (not attached to desktop)
- 6: \\.\DISPLAY7 (not attached to desktop)

EnumDisplayDevices says 1 device(s) are OK

DotNet should be happy

With display extended:

D3D9 says you have 2 adapter(s):
 - \\.\DISPLAY1 (NVIDIA GeForce GTX 1050 Ti)
 - \\.\DISPLAY5 (Intel(R) UHD Graphics 630)

EnumDisplayDevices:
- 0: \\.\DISPLAY1 (OK)
- 1: \\.\DISPLAY2 (not attached to desktop)
- 2: \\.\DISPLAY3 (not attached to desktop)
- 3: \\.\DISPLAY4 (not attached to desktop)
- 4: \\.\DISPLAY5 (OK)
- 5: \\.\DISPLAY6 (not attached to desktop)
- 6: \\.\DISPLAY7 (not attached to desktop)

EnumDisplayDevices says 2 device(s) are OK

DotNet should be happy 

System information

• GPU: GeForce GTX 1050 Ti and Intel(R) UHD Graphics 630 (Coffeelake 3x8 GT2)
• Driver: Nvidia 440.59.0 and Mesa 19.3.4
• Wine version: 5.0 stable
• DXVK version: 1.5

Happy to provide additional logging if required.

[Joshua-Ashton]

Yes, native D3D9 enumerates by displays and it's singular main adapter.

Currently we just enumerate by adapters even if they do not have a respective display -- this is super useful on iGPU and multi GPU setups (especially in the former where display attachment is kinda questionable given the whole PRIME crap). But you are right in that this does not match native behaviour...

So to fix this we'd need to implement proper multi-monitor support (which is something we've been meaning to do for a while but I'm not entirely sure if the Wine infrastructure is there right now,) as well as matching the native behaviour for adapter enumeration.

Ideally I'd like to leave in the old behaviour as some option too given how convenient it is for me (ie. testing games and traces quickly on AMD and NV, where the latter has no display attached) and probably others.

[Joshua-Ashton]

I am curious, what is the behaviour of Elite: Dangerous' launcher when

https://github.com/dotnet/wpf/blob/c4e74175da6108104491e1ae64982444d7c8713d/src/Microsoft.DotNet.Wpf/src/WpfGfx/core/common/display.cpp#L70 is false, ie:

//          HKCU\Software\Microsoft\Avalon.Graphics\MultiAdapterSupport
//      or
//          HKLM\Software\Microsoft\Avalon.Graphics\MultiAdapterSupport

-> 0

[redmcg]

I've tested that with dotnet48 and that resolves the issue (but not with dotnet40, they must of introduced this afterwards).

I can also work around the problem using the DXVK_FILTER_DEVICE_NAME environment variable.

[Joshua-Ashton]

Shame, I thought setting might be more sane than potentially breaking every/some iGPU and multi GPU setup people are using right now. Blegh.

[redmcg]

It's a valid work-around. I've documented it here:
https://github.com/redmcg/wine/wiki#launcher-doesnt-show-the-play-button

dotnet40 is manually installed anyway, so it's just as easy to install dotnet48.

But you do need to set a value for every executable - so it'd be set on a per-game basis.

[Joshua-Ashton]

We'd also need some way of finding what GDI DISPLAYs a Vulkan adapter is directly attached to. This is gonna be pretty messy and hacky, and I was hoping it was not going to be necessary (cause it sucks from every single aspect) which is partially why I've avoided it so far :-/

[Joshua-Ashton]

What is the name of the wpf dll that calls that function btw? Ie. frog.dll

I have a terrible idea for a hack.

[redmcg]

Do you mean this one? wpfgfx_v0400.dll

I'm pretty sure that's what the code at https://github.com/dotnet/wpf/tree/c4e74175da6108104491e1ae64982444d7c8713d/src/Microsoft.DotNet.Wpf/src/WpfGfx is compiled in to.

[Joshua-Ashton]

Does this workaround the issue (not suggesting this as any form of real fix btw, more curiosity)

  UINT STDMETHODCALLTYPE D3D9InterfaceEx::GetAdapterCount() {
#ifdef _MSC_VER
    void* returnAddress = _ReturnAddress();
#else
    void* returnAddress = __builtin_return_address();
#endif

    MEMORY_BASIC_INFORMATION mbi;
    if (::VirtualQuery(returnAddress, &mbi, sizeof(mbi))) {
      HMODULE module = reinterpret_cast<HMODULE>(mbi.AllocationBase);

      WCHAR moduleName[MAX_PATH + 1] = {};
      ::GetModuleFileNameW(module, moduleName, MAX_PATH);

      if (wcsstr(moduleName, L"wpfgfx_") != nullptr)
        return 1;
    }

    return UINT(m_adapters.size());
  }

[redmcg]

Yep - that works too

Edit: FYI - I had to pass a UINT value to __builtin_return_address to get it to compile. i.e:
void* returnAddress = __builtin_return_address(0);

[Joshua-Ashton]

@redmcg Can you try with this branch?

https://github.com/doitsujin/dxvk/tree/d3d9-display-enum

My work there is not fully done, but it should be safe enough to test.

[redmcg]

Thanks @Joshua-Ashton, I tried the Elite Dangerous Launcher with both:

• the iGPU disabled; and
• with the display set to extended

and it rendered correctly on both.

However, I did notice a couple of things when running d3d9Test.exe that are worth pointing out.

When set to extended I get the following output:

D3D9 says you have 2 adapter(s):
 - \\.\DISPLAY2 (GeForce GTX 1050 Ti)
err:   D3D9Adapter::GetAdapterIdentifier: Failed to query display info
 - \\.\DISPLAY2 (GeForce GTX 1050 Ti)

EnumDisplayDevices:
- 0: \\.\DISPLAY1 (OK)
- 1: \\.\DISPLAY2 (OK)

EnumDisplayDevices says 2 device(s) are OK

DotNet should be happy

So, there's the error, and the same adapter is shown twice.

And with the iGPU disabled I got:

D3D9 says you have 1 adapter(s):
 - \\.\DISPLAY2 (GeForce GTX 1050 Ti)

EnumDisplayDevices:
- 0: \\.\DISPLAY1 (OK)
- 1: \\.\DISPLAY2 (not attached to desktop)

EnumDisplayDevices says 1 device(s) are OK

DotNet should be happy

So the adapter is listed as DISPLAY2 which is also listed as 'not attached to desktop'.

When I go Settings -> Devices -> Screen Display in Gnome, it shows the Nvidia as being DISPLAY2. So maybe Wine should be showing DISPLAY1 as unattached and DISPLAY2 as OK.

[Joshua-Ashton]

that doesn't make much sense to me given i only report displays that are OK...

[Joshua-Ashton]

hmmm

[Joshua-Ashton]

oh right
I made a typo

[Joshua-Ashton]

This should be fixed now

[redmcg]

Nailed it - thanks @Joshua-Ashton. All works perfectly now.

Output for extended display is now:

D3D9 says you have 2 adapter(s):
 - \\.\DISPLAY1 (GeForce GTX 1050 Ti)
 - \\.\DISPLAY2 (Intel(R) UHD Graphics 630 (CFL GT2))

EnumDisplayDevices:
- 0: \\.\DISPLAY1 (OK)
- 1: \\.\DISPLAY2 (OK)

EnumDisplayDevices says 2 device(s) are OK

DotNet should be happy

and when the iGPU is disabled:

D3D9 says you have 1 adapter(s):
 - \\.\DISPLAY1 (GeForce GTX 1050 Ti)

EnumDisplayDevices:
- 0: \\.\DISPLAY1 (OK)
- 1: \\.\DISPLAY2 (not attached to desktop)

EnumDisplayDevices says 1 device(s) are OK

DotNet should be happy

And of course the Elite Dangerous Launcher is rendering correctly.

Thanks again.

[Joshua-Ashton]

No worries! Have fun! :)