Dxvk: Unigine Superposition launcher crash

[nairaner]

Software information

Unigine Superposition launcher crashes in kernel32

System information

• GPU: Nvidia GTX 750 Ti
• Driver: proprietary
• Wine version: wine-staging 2.21
• DXVK version: latest

Apitrace file(s)

https://mega.nz/#!5YI2VQAB!p0069ISZdqoeMSXdPX20fJXlQQM8-C36ZjWhtYl48To

Log files

• d3d11.log:
  launcher_d3d11.log

• dxgi.log: launcher_dxgi.log

• wine.log

[SveSop]

Possibly the "D3D11Texture2D" interface creation that is not implemented yet causing this?
Get the same as you, and doing the spirv-opt trick on the 2 dumped shaders does not help for this problem.

warn:  D3D11Texture2D::QueryInterface: Unknown interface query
warn:  D3D11Texture2D::QueryInterface: Unknown interface query
warn:  D3D11Texture2D::QueryInterface: Unknown interface query
debug: Compiling shader VS_cdfd077b92b7b670e3a0bbf1f30a5e771d521c63
debug: Compiling shader PS_9726ef301ae6484bf30d792aacfe3fd27fa4e83c
warn:  D3D11Texture2D::QueryInterface: Unknown interface query

Not sure tho... "warn:" should possibly not be gamebreaking? :)

[doitsujin]

Will have to take a look at which interfaces it actually needs, but this sounds like a lot of work. In the worst case it may even require something as stupid as DXGI<->GDI interop. It's likely that the crash is related to that though.

[SveSop]

There is currently a bug generally in wine with the SuperPosition gui. Its relatively "garbled", but if you know appx where the buttons are, it works.

However, if you set "d3d11.dll" to "disabled", it will switch to DX9 (i think), and work fine.. but the benchmark will ofc not work for "DirectX". This bug is with ALL versions of Wine, so not really related only to running with dxvk dll's.

So it might be more to it than fixxing D3D11Texture2D transformation perhaps?

Bugzilla: https://bugs.winehq.org/show_bug.cgi?id=43683

[oscarbg]

Hi guys,
I got it working more or less on Vega!
on native Win10 RX Vega..
the trick is not put d3d11.dll and dxgi.dll and launch the launcher..
of course it launches correctly and then selected custom medium quality windowed 1080p and before running putted back dxvk dlls on the folder (latest as of today)
see results:

superposition_d3d11.log
superposition_dxgi.log

seems logs complain a lot of:
D3D11Device::CreateGeometryShaderWithStreamOutput: Not implemented
CreateBuffer: D3D11_BIND_STREAM_OUTPUT not supported
D3D11DeviceContext::SOSetTargets: Not implemented
finally a warn about:
warn: DxbcCompiler: Unhandled opcode class: DxbcOpcode::BfRev

adding launcher logs anyway:
launcher_dxgi.log
launcher_d3d11.log
which also complain about D3D11Device::CreateBuffer: D3D11_BIND_STREAM_OUTPUT not supported

[pingubot]

thx for the tip, after optimizing the shaders it also fully works on nvidia. As far as i can tell, it renders completely correct (plain wine is not able to do that). I am again really impressed whats already possible at this stage of the project.

[ZeroFault]

@doitsujin

I'm getting these validation errors from spirv-val on these spir-v files in the Superposition benchmark.

PS_ae8e6b1471b25a5db1b500dee170dc3a122bcd6c.spv
error: 514: Illegal number of components (5) for TypeVector
PS_b894ff8ee03e2fe3bf870fe31502f9b7d1b50aeb.spv
error: 376: Illegal number of components (5) for TypeVector
PS_d7347a5c4bb3b87ffab4c4eef3e49364215baf07.spv
error: 530: Illegal number of components (5) for TypeVector
PS_f3075e40e8dee2daa325b9c63685f1215fc5da6d.spv
error: 514: Illegal number of components (5) for TypeVector

invalid-spirv.zip

[doitsujin]

@ZeroFault Are you on Nvidia hardware?

Thing with Nvidia is that their driver is broken and requires packing the depth value into the component vector for depth-compare operations, which seems to cause trouble here since cube map array coordinates are already four-dimensional. I don't know if it is possible to fix this issue without breaking Nvidia support entirely.

You can try commenting out the following line in dxbc_options.txt:

      this->packDrefValueIntoCoordinates = true;

[SveSop]

@ZeroFault I found that adding --skip-validation helped for those TypeVector errors.
spirv-opt --inline-entry-points-exhaustive --skip-validation --eliminate-dead-code-aggressive

@doitsujin I mentioned washed out colors on the other thread..

[SveSop]

The attached 3 shaders dumped from Superposition fails validation with spirv-val.
To be able to optimize those to work with nVidia, they need --skip-validation

Since im quite the n00b here, is it the driver that dump those .spv files, or is it dxvk?

broken.tar.gz

[ZeroFault]

@doitsujin
Yes I am using Nvidia. Unfortunately, disabling the packDrefValueIntoCoordinates causes issues with rendering. I wish Nvidia would stop having so many issues with its driver.

@SveSop DXVK is dumping the .spv files

[pingubot]

@doitsujin : have you opened bug tickets at nvidia already for all the issues which you already found ?

[oscarbg]

HI guys,
some updates( all on Win 10 native):
1) implementing bfrev fixed graphics issues on AMD Vega now renders correctly at medium and high settings..
2) tested also on Nvidia fixing the shaders (spir-opt legalize-hlsl) without the bfrev commit and also exact graphics issues were present but after that patch also renders perfectly on Nvidia..

so basically two issues to say:
1) settings shaders at maximum quailty (maybe called ultra) is only path that renders with graphical issues.. maybe that's the only path making use of geometry shaders and hitting this issues seriously:
D3D11Device::CreateGeometryShaderWithStreamOutput: Not implemented
CreateBuffer: D3D11_BIND_STREAM_OUTPUT not supported
D3D11DeviceContext::SOSetTargets: Not implemented

2) with a bat file I legalize shaders with something like:
for /f %%f in ('dir /b *.spv') do spirv-opt %%f --legalize-hlsl -o %%f.leg
@ZeroFault @SveSop @doitsujin
I'm using the spirv-opt executable included in the new Vulkan SDK 1.0.68 for Windows:
and as I say I don't see any errors like:
error: 514: Illegal number of components (5) for TypeVector
and also don't need to use --skip-validation argument in spirv-opt as I say it runs correctly with:
spirv-opt %%f --legalize-hlsl -o %%f.leg

may be spirv-tools may diverged a bit between the included in SDK and in github master..
I get in SDK:
spirv-opt --version
SPIRV-Tools v2018.1-dev v2018.0-6-g9e19fc0

[oscarbg]

@pingubot @doitsujin don't worry, I, as crazy as I am :-), bombarded all NV Vulkan dev techs I know pointing to this issue:
https://twitter.com/oscarbg81/status/964301753184333824
but if it's not enough let's ping also to his github names:
@pdaniell-nv @pixeljetstream what do you say about:

Thing with Nvidia is that their driver is broken and requires packing the depth value into the component vector for depth-compare operations, which seems to cause trouble here since cube map array coordinates are already four-dimensional. I don't know if it is possible to fix this issue without breaking Nvidia support entirely.

also that some shaders that are consumed correctly by AMD closed driver or even RADV fail on Nvidia without doing some:
spirv-opt --inline-entry-points-exhaustive --skip-validation --eliminate-dead-code-aggressive
on this shaders..
even with that or a full "spirv-opt --legalize-hlsl" pass some shaders still present issues..

[pdaniell-nv]

What specific SPIR-V OpImage function is having the issue? Eyeballing our shader compiler it appears dref is picked up as a separate operand, rather than the 4th component of the coord, from functions like OpImageDrefGather. What driver version are you using?

[pdaniell-nv]

Could someone share a crashing SPIR-V shader with me. It wasn’t immediately obvious to me how to get the project going under windows so just a problematic shader is probably the easiest thing to debug at this point. Thanks.

[oscarbg]

@pdaniell-nv IMHO the fastest way to test is this:

a)test on Windows to avoid setting up Wine-staging or from https://github.com/roderickc/wine-vulkan
b) if you don't want to compile by yourself download precompiled binaries from: https://haagch.frickel.club/files/dxvk/
for example from latest commit download both d3d11.dll and dxgi.dll from:
https://haagch.frickel.club/files/dxvk/r608.8ebffc1/64/bin/
or
https://haagch.frickel.club/files/dxvk/r608.8ebffc1/32/bin/
as needed by the app..

if you don't trust this guy, you can also build DXVK from Windows (basically to obtain the 2 DLLs this project is about (d3d11.dll and dxgi.dll)
for 64 bit DLLs:
I download https://nuwen.net/mingw.html launch his shell with compilers in path and then
clone this project and modify build-win64.txt:
comment "exe_wrapper = 'wine':"

exe_wrapper = 'wine'

and remove "/usr/bin/x86_64-w64-mingw32-" from all this lines:
c = '/usr/bin/x86_64-w64-mingw32-gcc'
cpp = '/usr/bin/x86_64-w64-mingw32-g++'
ar = '/usr/bin/x86_64-w64-mingw32-ar'
strip = '/usr/bin/x86_64-w64-mingw32-strip'
so basically leaving
c = 'gcc' ,etc..
then following his instructions you should obtain the 2 DLLs ..

c) put the dlls in the executable to test..
altough I pointed to this issue because of developer talking about Nvidia problems I would suggest for a simple failing program test ComputeMark2 program..
you have to trust the file I uploaded because this program has dissappeared from the web altough it was popular by review sites for new GPUs in the early D3D11 days (2010s):
https://github.com/doitsujin/dxvk/files/1670646/ComputeMark2.zip

it's a 32bit app so 32bit DLLs..

it a good test because has very few shaders (less than 10) mostly compute shaders that all fail to compile under
Windows 10 17083+Nvidia GTX 970 +390.65 drivers..
start a cmd.exe:
set DXVK_SHADER_DUMP_PATH=c:\dxvk\shaders
set DXVK_SHADER_READ_PATH=
set DXVK_HUD=1 (optional but to verify the D3D11 it's using Vulkan)
ComputeMark2.exe

that should get spv files and dxbc files c:\dxvk\shaders
using spirv-opt.exe from Vulkan SDK 1.0.68 I run on all shaders:
(with a script)
spirv-opt --legalize-hlsl file.spv -o file.spv.legal and copy spv.legal to .spv

my bat is

for /f %%f in ('dir /b *.spv') do copy /y %%f %%f.ori >nul
for /f %%f in ('dir /b *.spv') do spirv-opt %%f --legalize-hlsl -o %%f.leg
for /f %%f in ('dir /b *.spv') do copy /y %%f.leg %%f >nul

and then run the program but with these legalized shaders:
set DXVK_SHADER_DUMP_PATH=
set DXVK_SHADER_READ_PATH=c:\dxvk\shaders
ComputeMark2

you should continue seeing a windowed black screen and logs that it generates show failing creating shaders o compute pipelines don't remember..

I run spirv-opt --legalize-hlsl because that is shown to fix issues on NV GPUs on Unigine Heaven and Valley benchmarks (i.e. this SPIR-V programs weren't compiling successfully but with this pass they make NV SPIR-V compiler happy)..
in practice means going from the app running but seeing a black screen to rendering perfectly fine..

as said ComputeMark2 for example runs fine on same system with DXVK on AMD Vega GPU
even without spirv-opt --legalize-hlsl step..

if you wait a day or so I will upload the ComputeMark2 spv shaders with and without HLSL legalization so you can check what's going wrong..
altough I recommend checking by yourself for example that ComputeMark2 tests run and visualize correctly..

hope ComputeMark2 fixing triggers fixing most or all other remaining NV issues modulo bugs in DXVK project of course..

[GabrielMajeri]

@pdaniell-nv You can also get some DXVK binaries from OpenSUSE's OBS.

This archive contains all the vertex and fragment shaders for Unigine Valley (there's a lot of them, I'm not sure which one is problematic). The .spv files are SPIR-V files, the .dxbc files are the Direct3D11 shader files. The game launches but shows a black screen, and DXVK says it failed to create the graphics pipelines.

shaders.zip

[oscarbg]

@GabrielMajeri @pdaniell-nv
this is good to test but not perfect because as has said before with
spirv-opt --legalize-hlsl
this "NV problematic" spirv shaders get fixed..
instead Computemark2 shaders are less than 10 and continue being problematic on Nvidia driver even after "legalized" with spirv-opt --legalize-hlsl...
so I believe NV must see why hlsl-legalized spv shaders don't compile fine and then attack the spv shaders that can be fixed by spirv-opt --legalize-hlsl..
of course just my IMHO..

[Guy1524]

@pdaniell-nv
Here is one example of a shader which requires spirv-opt on NVIDIA from Unigine Valley:

W/O spirv-opt (crashing): https://paste.ubuntu.com/p/SqyDxQbKwn/

W/ spirv-opt (not crashing): https://paste.ubuntu.com/p/xDgZXbhfWc/

[oscarbg]

@pdaniell-nv @doitsujin @SveSop @pingubot @Guy1524 @GabrielMajeri @ZeroFault

just found that AMDVLK latest 2.0.15 open source driver on Vega build runs Superposition (previously ones didn't launch correctly) and what a surprise(!) also need "hlsl-legalization" (spirv-opt --legalize-hlsl) on some shaders to render correctly.. so very similar to Nvidia driver and this just proves that we just were lucky with AMD RADV and closed AMDGPU PRO driver in that accepted more relaxed SPIR-V shaders..
still Nvidia has more problems than even AMDVLK seems..

[pdaniell-nv]

In the https://paste.ubuntu.com/p/SqyDxQbKwn/ shader, it's causing a crash because there is "OpTypeImage %float 3D 1 0 0 1 Unknown" in the shader that we don't handle. There is no crash in https://paste.ubuntu.com/p/xDgZXbhfWc/ because the dead-code elimination has removed it since it's not referenced anywhere. We have no support for depth-sampling a 3D image so never expected to see a 3D image created with depth set. Any ideas why this OpTypeImage is being emitted?

[ZeroFault]

Hmm, looks like a bug in the DXBC to SPIRV compiler to me.

[pchome]

https://github.com/doitsujin/dxvk/search?utf8=%E2%9C%93&q=defImageType&type=

[oscarbg]

one more post here today..
I tested forcing disabling depth bounds test and 2 other caps requeriments for DXVK working on Intel HD530 with Anvil Mesa driver and tested lots of games..
good news is compatibility is very high.. lots work OK..
even Superposition and doesn't need hlsl legalization.. just as good as RADV..
proof:

[doitsujin]

@pdaniell-nv DXVK generated those depth types for all sampled image types. As far as I'm aware, it's not illegal to do that in SPIR-V, although certainly unnecessary if the type is not going to be used anyway. In any case, thanks for the information - I restricted that to 2D and Cube textures now though, so this particular problem should be fixed.

There are however more issues. I don't have an Nvidia card myself to test at the moment, but an example of a compute shader that apparently crashes the driver is https://pastebin.com/6cmCE49u this one. Inlining the cs_main function seems to do the trick in this case.

@oscarbg good job. Disabling depth bounds test on Intel is something I had in mind as a potential workaround, but the other two (image store/load without format) are a bit more problematic.

[nairaner]

@pdaniell-nv I already reported this example in Nvidia customer care, but all you need to do is start start.bat file in debugger

https://mega.nz/#!ZJI31JwS!k2grgsQwxmvojVvhqhSnHYIQkv30Yr5FHRqfrbs6LbI

1. There are 3 different shaders in bin\example\shader_opt:
2. default produced by dxvk (crash): CS_3e57aa98bdc58df0415b95c62877ef9e630eff9f.spv_unopt
3. optimized using SPIRV Tools (no crash): CS_3e57aa98bdc58df0415b95c62877ef9e630eff9f.spv_opt
4. modified by removing inline (no crash): CS_3e57aa98bdc58df0415b95c62877ef9e630eff9f.spv_mod

Disassemblies:

• default : unopt_dis
• modified: mod_dis

1. Replace wanted shader with CS_3e57aa98bdc58df0415b95c62877ef9e630eff9f.spv

2. Start program using bin\example\start.bat

[pingubot]

@doitsujin : just tested heaven and superposition with the latest dxvk build and it still crashes here on nvidia.. Should we move the discussion her to thread #64 ?

Edit: trying to get heaven to run again, i noticed that i needed much less rounds of optimizing the shaders to get it running, so your code change indeed fixed some of the crashing issues it seems.

Edit 2: Same applies for valley. I needed to optimze the shaders only one time, the rest works fine.
In the past it crashed a dozen times+ a dozen round optimizing was necessary. So it looks like only some minor issue may be left now.

[SveSop]

@oscarbg

with a bat file I legalize shaders with something like:
for /f %%f in ('dir /b *.spv') do spirv-opt %%f --legalize-hlsl -o %%f.leg
@ZeroFault @SveSop @doitsujin
I'm using the spirv-opt executable included in the new Vulkan SDK 1.0.68 for Windows:
and as I say I don't see any errors like:
error: 514: Illegal number of components (5) for TypeVector
and also don't need to use --skip-validation argument in spirv-opt as I say it runs correctly with:
spirv-opt %%f --legalize-hlsl -o %%f.leg

If you look at spirv-opt --help, you can see that --legalize-hlsl does the following:

               Runs a series of optimizations that attempts to take SPIR-V
               generated by and HLSL front-end and generate legal Vulkan SPIR-V.
               The optimizations are:

        inline-entry-points-exhaustive
        eliminate-dead-functions
        private-to-local
        scalar-replacement
        eliminate-local-single-block
        eliminate-local-single-store
        eliminate-local-multi-store
        simplify-instructions
        eliminate-dead-branches
        eliminate-dead-inserts
        eliminate-dead-code-aggressive

               Note this does not guarantee legal code. This option implies
               --skip-validation.

Note the latest about --skip-validation

What i and others did was try to pinpoint which of THOSE was necessary to run superposition, and it seems as 3 shaders needs: --inline-entry-points-exhaustive --skip-validation --eliminate-dead-code-aggressive
Running eg. Unigine Valley/Heaven, the --skip-validation is not necessary.

Recap: Running --legalize-hlsl does ALL of the above optimizations, including --skip-validation, therefor you will not see that particular error mentioned earlier :)

[pdaniell-nv]

@doitsujin Regarding https://pastebin.com/6cmCE49u it appears we have a couple of issues with this. Our inliner didn't handle this well, which is why when you inlined it yourself it worked better. I'm not sure yet why our inliner had a problem, but I'll report it to our compiler team to investigate.

The other issue we have with that shader is we don't support OpImageQuerySizeLod or OpImageQueryLevels on non-sampled images. If these ops were used on the %16 image "OpTypeImage %float 2D 1 0 0 1 Unknown" they would have worked okay. Core Vulkan/SPIR-V doesn't really support levels with storage images; there is no way to access texels from Lod>0 through OpImageRead/OpImageWrite. Because of this there is probably limited utility in using OpImageQuerySizeLod or OpImageQueryLevels on non-sampled images.

In that shader, you can get it to work by making the following modification:

         %31 = OpLoad %16 %u0  # changed from %17 to %16
         %33 = OpImageQuerySizeLod %v2uint %31 %30
         %34 = OpLoad %16 %u0  # changed from %17 to %16
         %35 = OpImageQueryLevels %uint %34

That appears to fix the inlining crash too, so they're probably related.

I may investigate a SPIR-V spec fix for this. I think in unextended SPIR-V 1.2 that OpImageQuerySizeLod and OpImageQueryLevels should only be defined to work with sampled images. It looks like the SPV_AMD_shader_image_load_store_lod extension adds Lod support to storage images.

[pdaniell-nv]

@nairaner This looks like another case of OpImageQueryLevels being used with a storage image rather than a sample image. If possible, try to use OpImageQuerySizeLod and OpImageQueryLevels only with sampled images.

Actually, the CS_3e57aa98bdc58df0415b95c62877ef9e630eff9f.spv shader looks similar to https://pastebin.com/6cmCE49u.

[SveSop]

@pdaniell-nv I did some testing with Unigine Heaven, and found the two shaders i need to get things running.
Take a look at : nVidia issues #64

[pchome]

Here is wine log for default Medium preset but Windowed on GeForce GTX 750 Ti 387.34.
Looks normal as I see this benchmark for the first time. First scene starting with too dark environment but later all looks good. Min:11-15/Avg:18/Max:23-25 fps
dxvk-superposition-wine.log

It's custom linux build w/ dll-redirect support and "auto legalizer", so I can run lancher with d3d11.dll and set redirects to d3d11_vk.dll only for superposition.exe. Then it runs.
Otherwise launcher starts as black screen with no controls visible. Trick with replacing original dlls with dxvk "on the fly" works too.

Edit: opengl mode: dxvk-superposition-wine-opengl.log

[doitsujin]

@pdaniell-nv that makes sense, again thanks for the info. That looks like a problem in my implementation of the resinfo instruction (and possibly others) when used on image UAVs, which are mapped to non-sampled images.

[pingubot]

@doitsujin : thanks for also fixing that issue. Heaven still crashes here with the current dxvk build, so there is still minimum one shader issue left which @pdaniell-nv hopefully should be able to clarify with the 2 shaders which @SveSop provided. Those are the only 2 right now for Heaven on Nvidia Linux which still cause crashes.

[SveSop]

@pingubot Could you test if Superposition starts and runs until scene 9 with the 3 shaders attached in "working.tar.gz"?
The broken ones i attach in "broken.tar.gz". I ran: spirv-opt --inline-entry-points-exhaustive --skip-validation --eliminate-dead-code-aggressive to get them working.

Just be aware that there is 1 or maybe more shaders that does not work tho, cos it will crash at "scene 9" when running benchmark or cinematic mode.
broken.tar.gz
working.tar.gz

[SveSop]

@pingubot Found the 4th one that will make Superposition run without crashing. Same deal "working.tar.gz" and "broken.tar.gz" for comparison. Superposition Preset: 1080p Medium

@pdaniell-nv This is with nVidia Linux driver 390.25 tho. And if you look at my earlier post with the screenshot Screenshot Its almost as its a thin layer of dust on everything. (Perhaps not that bad, since it looks like a lab from the 60'ies heh)
broken.tar.gz
working.tar.gz

[SveSop]

Have been so caught up with finding crashing shaders, that i havent really payed much attention to the logs.. But now that those 4 shaders get things running somewhat "ok", i saw the logs having quite a few:
err: D3D11Device::CreateGeometryShaderWithStreamOutput: Not implemented
Guess something that will be implemented at some point?

err:   D3D11Device::CreateBuffer: D3D11_BIND_STREAM_OUTPUT not supported
err:   D3D11DeviceContext::SOSetTargets: Not implemented

More of the same i guess.
Attaching logs for giggles and laughs :)
superposition_d3d11.log
superposition_dxgi.log

[oscarbg]

@doitsujin sorry for lack of clarity for Intel HD graphics working I disabled depth bounds which upped support to feature level 10.1 but for 11.0 only needed disabling shaderStorageImageReadWithoutFormat as shaderStorageImageWriteWithoutFormat is already supported (see: https://vulkan.gpuinfo.org/displayreport.php?id=2600#features) the other one I have to disable is variableMultisampleRate (as you can see from linked report is missing on Intel Mesa aka Anvil driver)..
anyway I still have working antialiasing on some apps like on Witness on iGPU so what you are using variableMultisampleRate for?
good news is that variableMultisampleRate is supported on Windows Intel HD graphics closed source Vulkan driver (see: https://vulkan.gpuinfo.org/displayreport.php?id=2529#features) so should have to disable only depth bounds and shaderStorageImageReadWithoutFormat there.. it also shows that Mesa devs can implement variableMultisampleRate support on Intel HD graphics.. perhaps do a request on mesa-dev mailing list? I mean is not like depth bounds test support that seems is not supported in HW by current Intel HD graphics..
but somewhat sad is I tested also Intel HD graphics on Windows closed source Vulkan driver disabling this extensions but I'm getting a hang somewhere inside their driver igvk64.dll I think.. so have to debug more..
will report success and some patches if found the root of the cause..

@SveSop thanks for the detailed explanation of why I was not getting validation errors when running full --legalize-hlsl command good to know it disables validation..

[oscarbg]

@pdaniell-nv @doitsujin
I think I'm on to something interesting with DXVK+NV using ComputeMark2..
Please test these repro:
ComputeMark2repro.zip
TLDR: executing ComputeMark MandelS test DXVK generates valid SPIR-V files and no errors are returned by Nvidia Vulkan driver regarding creation of compute or graphic pipelines, yet we see blank screen.. on AMD GPU all work nice..

it's ComputeMark2 with DXVK precompiled and included spir-v files of all benchmarks..
I'm using the latest dxvk including all latest 2 NV fixes:
[dxbc] Use correct image query instructions for UAVs
[dxvk] Only emit depth image types for 2D and Cube images
Please run either ComputeMark.exe or runanddump.bat file that will also redump files..

it's very easy from Windows10 just run ComputeMark.exe then from "Burn test demo" select the individual test you want and select Go button: MandelS and Fluid3D are of interest.. a window will be created for running the test but this window will remain black during all the test..
at least for me on Nvidia GTX 970+390.65 driver..

the thing is all spir-v files produced by DXVK are now valid according to spirv-val (i.e. doesn't require running spirv-opt)..
you can check all SPIR-V files on spirvshadersnv folder and from that folder a valshaders.bat will run spir-val for each spir-v file without any warnings/or errors..

even one test (MandelS) doesn't report any errors creating pipelines (see ComputeMark_d3d11mandels.log captured with "DXVK_LOG_LEVEL=debug").. still the window obtained is black..

the other interesting test is Julia3D this produces 2 DxvkComputePipeline errors
"err: DxvkComputePipeline: Failed to compile pipeline"
altough as said SPIR-V files are now valid according to spir-val..
screen renders black..

of course all this run and render OK on AMD Vulkan driver on Windows..

as said the more interesting test is MandelS as it doesn't even create any error creating graphics or compute pipelines..

[pdaniell-nv]

Thanks @oscarbg. I ran the new ComputeMark2repro.zip. The shaders that have OpBitFieldInsert found a bug in our compiler in that we didn't handle the "Offset" and "Count" operands being anything other than signed int. This causes a crash in the compiler. I fixed it locally to handle both signed and unsigned and I was able to get all ComputeMark.exe tests to compile without fault in our compiler.

Unfortunately, all tests produce a black window. I didn't get a chance to investigate why this is. My guess is maybe there is a mismatch between the shader and the pipeline layout such that some resources aren't being picked up.

Have you tried running this with Vulkan validation enabled? Perhaps tweak DXVK to emit integers for the "Offset" and "Count" operands of OpBitFieldInsert and OpBitField{S,U}Extract and see what the Vulkan validation says.

In the meantime we'll work on getting the compiler fixes for the driver released.

[pdaniell-nv]

FYI, I force enabled validation layers locally and there are a few errors being output. One of them which may give a clue is this one:

>
DS(ERROR / SPEC): object: 0x452 type: 23 msgNum: 364905100 - vkUpdateDescriptorSets() failed write update validation for Descriptor Set 0x452 with error: Write update to descriptor in set 1106 binding #3 failed with error message: Attempted write update to image descriptor failed due to: ImageView (77) with usage mask 0x23 being used for a descriptor update of type VK_DESCRIPTOR_TYPE_STORAGE_IMAGE does not have VK_IMAGE_USAGE_STORAGE_BIT set.. The spec valid usage text states 'If descriptorType is VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER, VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE, VK_DESCRIPTOR_TYPE_STORAGE_IMAGE, or VK_DESCRIPTOR_TYPE_INPUT_ATTACHMENT, the imageView and imageLayout members of each element of pImageInfo must be a valid VkImageView and VkImageLayout, respectively' (https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html#VUID-VkWriteDescriptorSet-descriptorType-00326)
>

[doitsujin]

Edit: The validation errors actually seem to be caused by an application bug, it's trying to use a depth-stencil image with a UAV but never sets the appropriate D3D11 bind flags. Might call for an application-specific workaround.

[oscarbg]

@pdaniell-nv thanks for giving an inspection to the repro and glad to help NV fix some bugs.. to be clear I'm not a dev of DXVK only pointed you to issues using it on NV GPUs vs on AMD..
@doitsujin sad it's an app bug.. thanks also for giving some of your time to investigate the cause.. seeing your findings and if not much work would be glad if you implement that workaround.. primary reason is maybe is might not be the unique D3D11 app doing it badly but working OK on D3D11 drivers..
also strangely enough this app works correctly on Wine D3D11->OGL wrapper on AMD and NV GPUs if I recall correctly so seems in some way OpenGL is more forgiving or Mesa devs also ignore/set some flags independent of what the app says..

[doitsujin]

@oscarbg OpenGL simply does not have explicit usage flags, so you can't get them wrong in the first place.

[pingubot]

@pdaniell-nv : Many thanks. Can you also take a look at the shaders which @SveSop provided which crash the driver in unigine heaven and superposition ? They both work with the amd drivers.

[jozefkucia]

Edit: The validation errors actually seem to be caused by an application bug, it's trying to use a depth-stencil image with a UAV but never sets the appropriate D3D11 bind flags. Might call for an application-specific workaround.

I think it's a DXVK bug. CreateUnorderedAccessView() should fail if D3D11_BIND_UNORDERED_ACCESS is not set. There is probably a test for this behavior in Wine conformance tests.

[pdaniell-nv]

@pingubot Are you referring to https://github.com/doitsujin/dxvk/issues/70#issuecomment-366338851?

[pingubot]

@pdaniell-nv , yes, 70 and 74+75

[doitsujin]

I figured out what was causing the ComputeMark issue, the application wants to use the swap chain image with an unordered access view, which was not handled properly. It's the first time I see an application do that.

Fixed in 8813ff979a49d2ae0ae3e6d1a88ad0ac12d48993.

[oscarbg]

@doitsujin aargh.. first thanks to all @jozefkucia @pdaniell-nv and for your effort.. I feel somewhat guilty now.. I remember reading that many many years ago by the guy who implemented that.. really I vaguely remembered that this app "rendered right from the RWTexture it use to simulate" so I already have in my mind some though that ComputeMark was a "special" D3D11 to test so I insisted you to fix support it the same way a year ago already insisted Wine D3D11 devs to make it work..
I have been spending some time to refind it.. and so here, just for completeness, let's point to original individual demos (with source code!) done by Voxilla very shortly (within 1 month) D3D11 drivers started appearing for AMD 58xx series.. (circa end 2009) (maybe some of you were kids still by then :-)) :
users.skynet.be/fquake/
here you can find also some demo not included in ComputeMark:
3D waves simulator (DX11) tough I expect it to work..
and now I see under "DX11 DirectCompute Mandelbrot and Julia viewer" on this site,
a DX11 feature level DX10 version:
http://users.skynet.be/fquake/MandelDX11_10.zip
and note this:

DX10 support
In order to support DX10 GPUs the DX11 feature level DX10 version should be used.
It will try to make use of compute shader 4 instead of compute shader 5.
This requires an additional pass with a pixel shader to copy compute shader output to the screen.

this demo should work now on Intel HD 530 Mesa driver now that you disabled depth bounds requeriment so gonna give it a try..

[oscarbg]

@doitsujin been playing with these demos and found that is has built a new Mandel version with AVX512 demo (http://users.skynet.be/fquake/Mandel_AVX512.zip)..
seems no source code shared but what it's interesting on this demo it uses HLSL doubles..
maybe one of few codes/demos using it..
using your latest DXVK says with a popup window: GPU doesn't support doubles (FP64)
as expected, as DXVK doesn't support that optional D3D11 cap..
as said not really needed, only poiting for if you someday feel really bored..
see doubles in use:

void CSMandelJulia_scalarDouble1( uint3 Gid : SV_GroupID, uint3 DTid : SV_DispatchThreadID, uint3 GTid : SV_GroupThreadID, uint GI : SV_GroupIndex )

{
int counter = max_iterations & (~(UNROLL-1));

double u,v;
u = (double)(a0 + daDTid.x);
v = (double)(b0 + dbDTid.y);

double a,b;
a = (julia) ? (double) ja0 : u;
b = (julia) ? (double) jb0 : v;

bool inside;
double ur, vr, t;

`

[oscarbg]

Two comments from previous posts:
*Please don't waste time with http://users.skynet.be/fquake/MandelDX11_10.zip I wasted my time already it doesn't even work natively on Windows..
*Note also Wine D3D11 also doesn't support doubles from my testing.. @jozefkucia, right?

[jozefkucia]

*Note also Wine D3D11 also doesn't support doubles from my testing.. @jozefkucia, right?

Yes, wined3d doesn't support doubles yet.

[SveSop]

One funny side-effect of https://github.com/doitsujin/dxvk/commit/4c693fc2624a8d37288815125aba19fa0a3aef0b is that Superposition GUI almost works as it should.. It's still no better than wine-devel, as the text in the menus will be gone once you click on them, but vs. the black box before spirv-tools auto-optimize its a improvement :)

Now if only @pdaniell-nv would come up with an idea why #90 happens to me and another user with GTX 970, but not for one with GTX 750, that would be great :)

[pdaniell-nv]

Nothing springs to mind. Do you see any validation errors?

There are a few physical device feature and property differences between GTX 750 and GTX 970. There is this new LunarG layer that fakes the physical device properties (https://www.lunarg.com/new-vulkan-dev-sim-layer/). It might be worth simulating the GTX 750 on your GTX 970 and see if it produces the same result. If so, then I guess DXVK is reacting to some physical device property.

If you want me to look more into it, would you mind making it easy to reproduce on Windows. Perhaps with some instructions or a .zip package with some binaries. Thanks.

Or if there are specific shaders you suspect I would be happy to look at this if you attach them here.

[SveSop]

@pdaniell-nv Since you hopefully have kept up with the dxvk<->spirv problems when it comes with different validations for nVidia, things work "out-of-the-box" for AMD, but nVidia needs different spirv-opt (like --legalize-hlsl) to not crash. After doing this, there is no validation errors on the shaders, and driver does not crash... but for me atleast produces this weird coloring.

I will look into this simulation thing, and see if i can "fake" a 750.

I have not tested this in "real" windows, but what i have tested is Unigine Superposition https://benchmark.unigine.com/superposition?lang=en (free version), both wine-staging-2.21 and wine-vulkan-3.2 with DXVK. Tested 387.34 and 390.25 driver in Linux. The driver revisions is different in Windows as you are quite aware of, but what 390.25 corresponds to in windows i dont know.
In theory if it is the driver and the same bug with dxvk, it should happen in windows to.
Binary builds of DXVK: https://haagch.frickel.club/files/dxvk/

There SHOULD not be a difference if i use the Ubuntu binary .deb driver vs. downloading the .run file from nVidia.com? I mean, the "driver" part is closed source binary blob and not really what is being compiled afaik?

[SveSop]

@pdaniell-nv
Did some fiddling with the simulation layer, and saw that both the "DXVK HUD" showed changes, and VulkanCapsViewer showed changes when i used hardware .json files. I downloaded a few for testing from https://vulkan.gpuinfo.org/

Tried with various 750Ti driver versions and Linux/windows++ and tested with a few different from 970. Problem did not change, although i had a couple of instances with random weird artifacts. I also tested with Intel/Amd, but not suprisingly that ended up in a crash.

It does seem to do with surface lightning of some sort.. Might be hard to see just from the couple of screenshots i have put up, but when (in Superposition) i move to dark spots with low light, the surfaces look mostly normal... its when the light kinda shines "too bright" on the surface that it looks that bad. I guess its the same in Unigine Valley, as some darker scenes does not look that bad, but when the well-lit hillsides show up, its just "too bright".

[SveSop]

@pdaniell-nv Also take a look at my screenshots https://github.com/doitsujin/dxvk/issues/90#issuecomment-368323734

And my comment above in the same thread does not show this bug in "The Talos Principle" either with wine vulkan, or wine dxvk. Two different graphics engines tho.

[pingubot]

@pdaniell-nv : As you have asked for shader exampled which you can check, i already mentioned the shaders which @SveSop found for Unigine Heaven, Unigine Valley and Unigine Superposition, which still crash the Nvidia linux driver 390.25. To make it much easier for you, here is a summary of all the links and informations spread in different threads. Really looking forward to your findings:

Unigine Heaven:

The bad shaders which crash the driver:
bad shader heaven
The corresponding fixed ones after using: spirv-opt --inline-entry-points-exhaustive
fixed shader heaven

Unigine Valley:

The bad shaders which crash the driver:
bad shader valley

The corresponding fixed ones after using: spirv-opt --inline-entry-points-exhaustive
fixed shader valley

Unigine Superposition:

The bad shaders which crash the driver:
bad shader superposition

The corresponding fixed ones after using: spirv-opt --inline-entry-points-exhaustive --skip-validation --eliminate-dead-code-aggressive
fixed shader superposition

If you need any other information, please inform us. It really would be fantastic to be able to use dxvk on nvidia without needing to use the spir optimizer by either nvidia fixing possible bugs in the driver or dxvk fixinf possible bugs in the code.

Many thanks !
Christian

[pdaniell-nv]

@pingubot Thanks for breaking it down for me, it really helps.

For the Heaven shaders and the Valley shaders they hit a bug in our compiler in that we don't handle OpBitField{S,U}Extract correctly when the "Offset" and "Count" parameters are something other than signed integer. If possible, modify the generator to use int parameters here. We have implemented a compiler fix for this already and it should be released in an upcoming driver.

I didn't finish looking at all the superposition shaders. For the first one PS_1c31935ff36f56ad8a195a03b959cf58e4ee0bec the problem is that OpTypeImage is specifying a non-float depth (or shadow) image. This isn't something we support. It's not clear to me if this is our bug or not. I think it is, I guess nobody has tried creating an integer depth image before now. You should be able to work-around it by using a float type when doing depth images, if possible. In the meantime I'll investigate a fix for this.

For the PS_8c0e1a4293e345daf650578b35851da3fa78cd86 shader I'm seeing an OpTypeVector with a component count of 5. This should be 4 or less.

Shaders PS_55dfe220ecfc27eb1c452ecffbff3397e52ec94a and PS_c38d348a092ddb481d51a49f03c0bc0c8096bc85 have the same problem as PS_1c31935ff36f56ad8a195a03b959cf58e4ee0bec in that they're using a non-float depth image.

[doitsujin]

@pdaniell-nv

I don't think declaring Integer depth images makes any sense, so I guess I'll change the compiler to only emit depth image declarations when they are actually needed - which is a bit of a hassle, but should prevent bugs like this one in the future. For now I've filtered out the integer depth image declarations.

As for the signed integer issue, are you aware of any other instructions which make this assumption? I changed the compiler to use signed integers for those instructions (as well as OpBitFieldInsert).

I also disabled the workaround that packs the reference value from depth-compare operations into the coordinate vector, which was why the five-dimensional vectors were generated. Hopefully this workaround is actually not needed (anymore) for those instructions.

In any case, thanks again for your input, this has been really helpful so far..

[ZeroFault]

@pdaniell-nv

Thank you for help on these issues. We really appreciate it.
Your OpBitField suggestion helped compile those shaders from Heaven's benchmark successfully.

There is another issue in Heaven's benchmark that we need help with. These shaders are crashing in the Nvidia driver. This happened after we removed the packing Dref value into the coordinate vectors in the lastest commit of DXVK. These shaders work fine on AMD.

dref_failingshaders.zip

[SveSop]

@doitsujin Somewhat minor issue with that commit
https://github.com/doitsujin/dxvk/commit/8bfd12067aaaaab34a1cd9036be1e66b99f12434#commitcomment-27846864

[doitsujin]

@SveSop I'm fully aware that the commit breaks pretty much everything on Nvidia. But the old workaround was fundamentally broken as it caused the compiler to generate invalid SPIR-V in a lot of cases.

What I need is clarification as to why depth-compare operations don't work properly on their driver and what I can to to work around it.

[pingubot]

@pdaniell-nv , it would be fantastic if you could clarify @doitsujin's question to make dxvk usable on nvidia again..

[SveSop]

@doitsujin Since you were fully aware, a comment about "Will break nVidia rendering" on the commit text would be nice :)

[pchome]

@doitsujin

I'm fully aware that the commit breaks pretty much everything on Nvidia.

Not sure about this one but recent commits fixes invisible character's and NPC's heads in The Witcher 3. Heads a bit not synchronized with body while they (characters) moving (_that looks like in scary movie when face changed reveals villain's real nature_) but visible.

[pdaniell-nv]

@ZeroFault Thanks for attaching the broken shaders. I investigated the issue and it appears to have hit another bug in our compiler. I'm confirming with our compiler guys to be sure. In this case it appears we're not handling OpImageSampleDrefExplicitLod correctly. Our bug here is probably what drove the work-around DXVK implemented for NVIDIA that packs the depth parameter with the coordinate. FWIW, you shouldn't need this work-around for cube, cube-array and 2d-array depth images. I think those are probably the cases that generate the OpTypeVector(5), so you can fix that too.

A slightly simpler work-around is to simply use a +1 larger Coordinate vector for the dref Ops. The last component is actually ignored and the proper Dref is the one it actually uses. The larger Coord vector is needed because internally we move the Dref into that last slot and our bug was that we neglected to increase the size of the vector to take the Dref. So with your work-around of putting the Dref into the Coord effectively made it +1 bigger and worked-around the bug.

I'll see what our compiler folks say and try to get the fix in the driver asap.

[pdaniell-nv]

Thinking about this more, it seems crazy we have this bug because surely Vulkan CTS would have caught it, because it definitely tests all the Dref image ops.

Turns out a bug in glslang is masking it, and most Vulkan CTS tests are written in GLSL.

With GLSL you have to pack the Dref into the cord. The prototype is "textureLod(sampler2DShadow, vec3, float)". The Dref is in the last slot of the vec3 coordinate. When glslang generates the SPIR-V what it should do is shrink the coord down to a vec2 since OpImageSampleDrefExplicitLod with a 2D depth image only needs a vec2, but it doesn't, it leaves it as vec3. This masked our bug.

[doitsujin]

I don't really think it's a bug in glslang, it's just being lazy and exploiting the SPIR-V spec:

Coordinate must be a scalar or vector of floating-point type. It contains (u[, v] . . . [,
array layer]) as needed by the definition of Sampled Image. It may be a vector larger
than needed, but all unused components will appear after all used components.

Anyway, I'll implement a new workaround.

[doitsujin]

0e9b7d7ccdcc6ca457576f44c2e2201c59e4a968 adds a new workaround for the dref issue.

[pchome]

@pdaniell-nv while you watching
https://forums.geforce.com/default/topic/1014038/nvidia-driver-bug-with-glsl-mat3-in-vulkan-/

Met this while googling some time ago. Maybe it's not related or not a bug.

[oscarbg]

Hi all,
@doitsujin
fixed Intel HD Windows driver support.. the issue is about calling vkCreateInstance on his driver ends up calling your dxgi CreateDXGIFactory or CreateDXGIFactory1 method which also ends up calling vkCreateInstance
I created a very lame patch to solve the issue.. (please see last post on issue https://github.com/doitsujin/dxvk/issues/62 with the diff I apply)
would be nice if a somewhat "more correct" patch gets merged eventually..
of course disable shaderStorageImageReadWithoutFormat and with that
just completed testing DXVK in the latest Vulkan Desktop driver remaining: the Intel HD Windows Vulkan driver..
results seems pretty good:

[doitsujin]

Closing because the original issue has been resolved.