Dxvk: World of Warcraft rendering distortion when MSAA is enabled

[Kerrung]

Hi, @doitsujin !
One of the Mesa developers said that this is not a Mesa bug in his opinion.

All looks fine when no any AA enabled:

But World of Warcraft has a rendering distortion when MSAA(x2-x8) is enabled:

Software information

World of Warcraft, any graphics settings used, but MSAA is enabled.

System information

• OS: Arch Linux
• GPU: AMD Radeon HD7770
• Driver: AMDGPU Linux 4.17.2, today's freshest Mesa-git
• Wine version: today's freshest Wine-git + Staging-git
• DXVK version: 4f28d57

Apitrace file(s)

• Sadly, but I absolutely haven't any more idea how I can record a apitrace, because it return only a lot of error messages, but it unfortunately don't creates a trace file. I tried recording a apitrace with WineD3D and then even with DXVK, but the trace file was not created anyway. This works for other games, but unfortunately does not work with WoW. I also tried it on Windows, but no result too...

Log files

• wow-wine-dxvk-console-output.txt
• Wow-64_d3d11.log
• Wow-64_dxgi.log

Please, tell me if you need any more info or logs etc...

[doitsujin]

One of the Mesa developers said that this is not a Mesa bug in his opinion.

The DXVK bug which causes the Vulkan validation errors was introduced after you filed the Mesa bug
report and has been fixed in 0.60. This issue must be unrelated, but I can't help you without a trace file.

Please run the game with validation layers enabled and post the output.

[Kerrung]

Please run the game with validation layers enabled and post the output.

@doitsujin ok, but I will do it tomorrow or maybe the other day because now I have gone to bed.
thank you.

[Worfhard]

MSAA should render correctly if you disable "Multisample Alpha-Test" at least on a RX 480 mesa-git llvm-svn-7. Otherwise I get the same kind of distortion.

[Kerrung]

MSAA should render correctly if you disable "Multisample Alpha-Test" at least on a RX 480 mesa-git llvm-svn-7. Otherwise I get the same kind of distortion.

@Worfhard thank you, but this is not a bug fix, but is only a workaround. But my goal is to help developers really fix bugs.
regargs

[doitsujin]

@Worfhard do you happen to know what exactly that option does, or rather, how exactly it works?

[hakzsam]

@Worfhard if you have a RX480, can you record a trace of the problem using renderdoc please?

[Kerrung]

@hakzsam World of Warcraft has a free trial unlimited in time. So you can try to test this game yourself.
Free download here.

To reproduce this issue you need enable "Multi-sample Alpha-Test" simultaneously with MSAA.

[Kerrung]

@doitsujin

The DXVK bug which causes the Vulkan validation errors was introduced after you filed the Mesa bug
report and has been fixed in 0.60. This issue must be unrelated, but I can't help you without a trace file.
Please run the game with validation layers enabled and post the output.

Mesa (git) 99c6cae227
DXVK a196451
Wine-git 86864486be + Staging-git 729a9eb

This console output for some reason turned out to be very short, but I installed the vulkan-validation-layers Arch Linux package and ran the following command:

$ env VK_INSTANCE_LAYERS=VK_LAYER_LUNARG_standard_validation WINEDEBUG=-all WINEPREFIX=~/.wine wine64 start /unix '/home/user/.wine/drive_c/Program Files (x86)/World of Warcraft/Wow-64.exe'

WARNING: radv is not a conformant vulkan implementation, testing use only.
ATTENTION: default value of option mesa_glthread overridden by environment.
WARNING: radv is not a conformant vulkan implementation, testing use only.
WARNING: radv is not a conformant vulkan implementation, testing use only.
WARNING: radv is not a conformant vulkan implementation, testing use only.
WARNING: radv is not a conformant vulkan implementation, testing use only.
mesa: for the -simplifycfg-sink-common option: may only occur zero or one times!

It's short, but that's all there is. Tell me please if I did something wrong.
Regards.

[fuchsi]

Multi-sample Alpha-Test with any MSAA option enabled shows similar distortions.

GPU: Radeon R9 380
Kernel: 4.16.9-gentoo
Mesa Version: 18.1.4
DXVK Version: 0.62
Wine: wine-staging-3.12

Output with Validation Layers: wow.log

I think the important part is:

(null)(ERROR / SPEC): msgNum: 15 - Shader requires extension VK_KHR_shader_draw_parameters but is not enabled on the device
    Objects: 1
       [0] 0x0, type: 0, name: (null)
Validation(ERROR): msg_code: 15: Object: VK_NULL_HANDLE (Type = 0) | Shader requires extension VK_KHR_shader_draw_parameters but is not enabled on the device

I hope this helps.

[doitsujin]

I think the important part is:

That's a bug in the validation layers which has already been fixed in newer versions. Completely unrelated to the MSAA thing.

[Olie440]

It's a bit late but I've done a trace and turned on validation both with the option enabled and the option disabled. I've included all the generated files and the output from stdout. I ran the same scene for both (human new char scene) so the only difference should be the option being on and off.
wow-trace.zip

[K0bin]

That's not an apitrace.

[doitsujin]

@Olie440 the validation output is useful, there's a bug in the shader compiler. Please test if a53e05339174ae626d1207f636c69e548fcdd6ee fixes the issue.

[Olie440]

@doitsujin Thanks for getting back so fast. It took a while to setup MinGW64 but i've given https://github.com/doitsujin/dxvk/commit/a53e05339174ae626d1207f636c69e548fcdd6ee a test and its now throwing different validation errors. Here's the log:
wow-validation-a53e053.txt

[doitsujin]

But does it fix the rendering errors?

The new errors happen because the validation layers don't support transform feedback yet. Not a bug.

[Olie440]

Rendering errors are still present in that build.

[Olie440]

I tried to do a api trace using the x64 dll's but it caused the game to crash. It still output data though, i'm not sure how useful this will be but ill include it anyway: Wow-a53e053.trace.zip

[doitsujin]

Completely useless, sadly.

[Olie440]

Thought as much :/ is there anything else I could run to help debugging?

[Dehir]

Guess this bug has something to do (is the same bug) with the bug im facing with rx480 too. It relates to all effects that are present and gets worse in example raids/dungeons where is lots of areal spells/effects happening. Tryed to debug this out but havent been able to make proper debug setup to make anything out. This is more Vulkan problem atm than DXVK?

Game is still somewhat playable if you drop down all effects to low/minimum to reduce load.

[mati865]

WoW with everything maxed out doesn't have glitches on Nvidia so I'd blame Mesa.
Somebody could also try with AMDVLK and see if it gets better.

[Olie440]

I've found that if I turn off this option it works fine and can get 60 fps (with lower quality AA). I'm happy to try it tonight using AMDVLK see if this is still an issue.

[doitsujin]

@Olie440 any news on the AMDVLK front?

[Olie440]

@doitsujin Unfortunately not, I was unable to get it working on my setup.

[doitsujin]

I finally got battle.net and WoW to work and can reproduce the problem, I'll investigate this issue.

[doitsujin]

Should be fixed with the above commit.

[doitsujin]

Nevermind, that fix is wrong and doesn't fix the problem, it's just that WoW randomly lost my settings.

[Olie440]

Thank you for taking the time to looking into it :smiley:

[doitsujin]

Fixed in mesa by this commit.