Dxvk: Nvidia driver crashes in different applications/games

[pingubot]

Hi,

as the nvidia drivers seem to have issues with dxvk atm, @doitsujin requested to create a meta bug to sum up the titles where the driver crashes (you can see that in the backtrace). Please add the necessary information which is requested for the usual bug reports also.

I will start with the following:

System information

• GPU: GTX 970
• Driver: Nvidia 390.12
• Wine version: Wine-Vulkan 3.1
• DXVK version: Snapshot from 04.02.2018

Unigine Heaven

Crashes after loading

Heaven_dxgi.log
Heaven_d3d11.log
backtrace-heaven-20180204.txt

Ungine Valley

Crashes after loading

Valley_dxgi.log
Valley_d3d11.log
backtrace-20180204.txt

Cu,
Christian

[doitsujin]

All driver crashes I've seen so far happen in the driver's shader compiler and are caused by "bad" (but valid) SPIR-V shaders. Starting with commit ad6c45d6b10fd80535f5e7d6d8474eecfbbf1926, DXVK logs the names of the shaders being compiled if DXVK_LOG_LEVEL=debug is set, which may at least help isolate these issues.

The names that appear in the log file are the same as the file names when setting DXVK_SHADER_DUMP_PATH=/foo/bar

@pingubot please update your log files accordingly.

[ZeroFault]

Here's mine on my GTX 1080.

Logs are from native Windows 10.

Heaven_d3d11.log
Heaven_dxgi.log
vulkanlog.txt

vulkan log including performance output.
vulkanlog.txt

backtrace from Winestaging-2.21
heaven_backtrace.log

Edit: This is with tessellation turned off in Heaven benchmark
Edit Edit: Added a more detailed vulkan log with performance suggestions

[pingubot]

Hi All,

here is mine, system like in the original report:

Heaven:

Heaven_dxgi.log
Heaven_d3d11.log

Valley:

Valley_dxgi.log
Valley_d3d11.log

All settings turned to minimum, no tesselation active.

Many thanks !

[pingubot]

@ZeroFault : How did you create the vulkanlog btw ?

[ZeroFault]

with this
vk_layer_settings.txt

Stick that file in the same directory as the application you are running and when you set the DXVK_DEBUG_LAYERS=1 variable it should output it.

I wasn't able to get it work on Wine-staging though. I had to do it on native Windows.

More information about the file here

[pingubot]

@ZeroFault , many thanks, will try with wine vulkan later on

[ZeroFault]

@doitsujin

I was finally able to get everything rendering correctly by using the SPIR-V optimizer in SPIRV-TOOLS on the dumped shaders in Heaven's benchmark. I used the --legalize-hlsl option on the optimizer and everything started working.

The only thing not working correctly yet other than tessellation is the MSAA, but it seems like a simple fix by enabling the sampleRateShading on the device according the vulkan debug output.

Heaven_d3d11.log
Heaven_dxgi.log
vulkanlog.txt

[doitsujin]

Thanks for testing that, that gives me something to work with.

Enabling the sampleRateShading feature almost certainly won't fix MSAA though, it'll only silence the validation errors.

[pingubot]

@doitsujin : Doing the mentioned shaded optimizer trick which @ZeroFault mentioned, Valley runs fine here now .
@ZeroFault : FYI, with wine-vulkan the vulkanlog.txt is created with your provided vk_layer_settings file .

[SveSop]

I cant for the life of me getting this to work at all tho, so it must be something i have done wrong.

I did however find out that the 32-bit dll's seemed to either have to be in BOTH c:\windows\system32 AND c:\windows\syswow64 folder when you have a 64-bit prefix. Are you guys running 32-bit prefixes?

I just get an error that it cant create window + the dxgi.log say ExtensionSet::addInstanceExtensions: Failed to query instance extensions

I use wine-vulkan and can run the "cube.exe" from the VulkanSDK tho so it seems as my .json file is set up correct under the wine prefix.
Valley_dxgi.log
playonlinux.log

[pingubot]

@SveSop : Have you tried running the cube.exe in the Bin32 folder also ? Looks like you are missing 32bit vulkan support..32bit dlls go to syswow, 64bit ones to system32 in a 64bit wine prefix.

[SveSop]

@pingubot Yeah, had a minor brainfart there with thinking syswow64 = 64, and system32 = 32... Thanks!

And yeah, you are right.. The cube_32 does have the same error. Hmm.. I have libvulkan1:amd64 and libvulkan1:i386 ... needs investigating 👍

[SveSop]

Oki.. What i have tried:

1. Created a new winevulkan32.json file, pointing to /syswow64/winevulkan.dll
2. Added the "Drivers" registry key to HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Khronos\Vulkan\Drivers, and pointed to the winevulkan32.json

This seem to get loaded, as when i start vulkaninfo32, i get an error saying:
"Failed to open dynamic library "c:\windows\syswow64\winevulkan.dll" with error 1114"

1. Created a new nvidia_icd32.json pointing to /usr/lib32/nvidia-390/libGLX_nvidia.so.390.25

Same problem.

The nvidia_icd.json that works when running 64-bit points to /usr/lib/nvidia-390/libGLX_nvidia.so.390.25
From what i understand the doc's, the .json files in the "icd.d" folder is scanned for "usable" libraries, and if i only have my nvidia_icd32.json file there, and run vulkaninfo in Linux, i get "Wrong ELF class: ELFCLASS32".. If i add the json pointing to the 64 bit library, it works. Conclusion (in my little brain), would be: If a 64 bit app needs a library, it uses the 64 bit one, and if a 32 bit app needs one, it would use the 32 bit one.

Running the d3d11-triangle.exe file from my 64-bit compiled /bin folder, produced the attached logs (although seemingly not producing anything but a quick window that instantly closes)
d3d11-triangle.EXE_d3d11.log
d3d11-triangle.EXE_dxgi.log

[pingubot]

@SveSop : I think it would make sense to end the discussion here cause your issue is not dxvk related, you have an issue with wine-vulkan,

I had exactly the same problem, showing the same error message when i tried wine-vulkan first time. I missed to compile in vulkan support in the 32bit wine version. I am pretty sure that you also have a 32bit wine version without vulkan support compiled in.

It also doesn't make sense to try dxvk without being able to run cube.exe and the 32-bit cube.exe with wine vulkan. If both work, you can go ahead and try dxvk.

Here is the log to the wine-vulkan issue which i opened, the wine-vulkan dev was really helpful to get my issue sorted out:

https://github.com/roderickc/wine-vulkan/issues/7

If that doesn't help, you could try to open an issue at wine-vulkan. But i am nearly 100% sure that the issue you are facing is build related.

[SveSop]

@pingubot Right.. Got it nailed down to my graphics-drivers ppa not being able to install the vulkan1 library, cos they have not updated it to the same version as my amd64 version.. so i had to download and install the .deb's manually to get both amd64 and i386 versions... and wham, i can run cube and cube32 :)

Attaching the logs from heaven (used the vk_layer_settings.txt that was posted above, but have not done any spir-v-tools things)
Heaven_d3d11.log
Heaven_dxgi.log
vulkanlog.txt

[SveSop]

Well.. since I (and others with nVidia cards it seems) are not so handy with the SpirV-tools package.. is there any, possibly simpler dx11 things we can test to see if things work?

Or, for that matter.. a wee bit more detailed howto on how to "fix" the shaders so that Heaven/valley run for me :)

By "more detailed" i mean a step-by-step instruction other than "by using the SPIR-V optimizer in SPIRV-TOOLS on the dumped shaders in Heaven's benchmark. I used the --legalize-hlsl option on the optimizer and everything started working."

I downloaded the "SPIRV-Tools-master-linux-RelWithDebInfo.zip" file, and run ./bin/spirv-opt --legalize-hlsl shader_d3d11.cache -o shader.cache
(the shader_d3d11.cache file i find under the /users/myuser/Heaven folder)
Error is: error: input:0:0:0: Invalid SPIR-V magic number '31306373'.

Clearly i am not very versed in the "just use the SPIR-V-Tool" thing :)

[SveSop]

Right
DXVK_SHADER_DUMP_PATH=directory
DXVK_SHADER_READ_PATH=directory
Dumped loads of .spv and .dxbc shaders in my folder. Made a subfolder under this one, and ran:
for i in *.spv; do ../bin/spirv-opt --legalize-hlsl $i -o fixed/$i; done

That generated some new .spv files in my "fixed" folder, and i pointed the "DXVK_SHADER_READ_PATH=" to the "fixed" folder... Am i on the right track?

[ZeroFault]

@SveSop Yes, that's how I did it. It looks like there's a fork that does this at runtime that you could try also. You will need to set DXVK_SHADER_DUMP_PATH and DXVK_SHADER_READ_PATH on that fork.

[SveSop]

@doitsujin Since it seems as the problem can be solved by using the spirv-opt --legalize-hlsl hack, is there something you need to weed out what the problem is?

A dump of the shaders pre-"legalize"? I did not seem to pinpoint the exact shader that crashed from the logs, as "fixing" the last one mentioned in the log did not seem to put the process any further. Perhaps it crashes on the "next-in-line" and thus not being logged due to the crash?

Ideas are welcome :)

[doitsujin]

Perhaps it crashes on the "next-in-line" and thus not being logged due to the crash?

Shaders are logged before they are sent to the driver, but you need to fix both the vertex and the fragment shader for a given pipeline.

is there something you need to weed out what the problem is?

I guess one thing would be to manually test which of the optimization passes that --legalize-hlsl applies are actually required.

A dump of the shaders pre-"legalize"?

Not sure if that will actually help, but it certainly won't hurt either. If the cause of the issue is anything like the one I already fixed, it might actually help.

[ZeroFault]

@doitsujin
I narrowed down the specific optimization of spir-v optimizer to the --eliminate-dead-code-aggressive option. Though, honestly, I'm not sure if that will work for every game since I only tested a couple of games.

[pingubot]

@ZeroFault : nice ! @doitsujin : Would it be possible to disassemble and compare a broken and a fixed shader if you have one where it crashes ?

[doitsujin]

@ZeroFault Thanks. Just to make sure I understand correctly, applying that one option is enough to satisfy the Nvidia driver?

[ZeroFault]

Yes

[volca02]

https://github.com/KhronosGroup/SPIRV-Tools/blob/master/source/opt/aggressive_dead_code_elim_pass.cpp

This is the source code of the elimination pass for that option.

[SveSop]

@volca02 @ZeroFault @doitsujin
--eliminate-dead-code-aggressive was not enough for me with 390.25 (GTX 970), as i had to add --inline-entry-points-exhaustive aswell. Not sure if that is purely driver related, or what it is... or for that matter dependant on the game/test you run.

I tested with BOTH optimizations, and Unigine Heaven + Unigine Valley worked. I tested this by starting with all the optimizations the --legalize-hlsl option does, and ended up (after god knows how many runs) with those two.

Each time i was able to run without crash, i deleted all my shaders and started fresh with 1 optimization less. Without those two, i am not able to run Heaven/Valley.

I would think possibly https://github.com/KhronosGroup/SPIRV-Tools/blob/master/source/opt/inline_exhaustive_pass.h could be worth looking at too.

Not sure what happens tho, cos dxvk seems to generate usable shaders until one random shader crashes everything. If i could pinpoint the few shader that crashes, i have a theory that everything would work if those would be "optimized".

[pchome]

I have played a bit with this shader dump code and ended up with linking SPIRV-Tools-opt library.
https://github.com/KhronosGroup/SPIRV-Tools/blob/dd8400e15077e640043b096d7634976d72fe2f8a/source/opt/optimizer.cpp#L70

Something like dump/read always enabled and new shaders stored to read directory after optimization.

#include "spirv-tools/optimizer.hpp"

spvtools::Optimizer spirv_opt(SPV_ENV_VULKAN_1_0);

spirv_opt.RegisterLegalizationPasses();

std::vector<uint32_t> spirv_fixed_data;
if (!spirv_opt.Run(spirv_dump_data.data(), spirv_dump_data.size(), &spirv_fixed_data)) {
  Logger::debug(str::format("Shader NOT optimized ", m_name));
}
...

or

#include "spirv-tools/libspirv.hpp"

spvtools::SpirvTools spirv_core(SPV_ENV_VULKAN_1_0);

if (!spirv_core.Validate(spirv_dump_data.data(), spirv_dump_data.size())) {
  Logger::debug(str::format("Shader NOT valid ", m_name));
}
...

But have some problems with data exchange.

[ZeroFault]

@SveSop
You're right, I was testing on native Windows 10 with a GTX 1080 ( driver 390.77) and didn't account for differences in driver versions. Maybe the driver fixes will get to the Linux driver soon.

I tested again with the Linux Nvidia driver (387.34) with the options --inline-entry-points-exhaustive --eliminate-dead-code-aggressive and it worked with no crashes.

[doitsujin]

@pchome how did you build that library? I haven't succeeded in building spirv-tools on mingw.

[pingubot]

not sure what it means, but trying to fix some of the unigine superpos shaders leads to:
superposfixedshaders/$name ; done
error: 376: Illegal number of components (5) for TypeVector
error: 475: Illegal number of components (5) for TypeVector
error: 475: Illegal number of components (5) for TypeVector

[pingubot]

@SveSop , @ZeroFault : For me it is not enough to use the 2 mentioned options to get unigine suprposition 2 run. Can one of you confirm that , see #70

[pchome]

how did you build that library?

I'm using Linux native build #60 and prebuilt spirv-tools libs from https://github.com/KhronosGroup/SPIRV-Tools/releases. But I'm sure I can also build spirv-tools by myself on linux.

[ZeroFault]

@pingubot
When you run --legalize-hlsl option it bypasses the spirv-validator and it runs the optimizer anyway. With just the other optimizer arguments, if it fails the validator the optimizer won't run on it. So that could be the reason why it fails to work for you on just the inline-entry-points-exhaustive --eliminate-dead-code-aggressive options

[SveSop]

@pingubot I have not tried the "move d3d11.dll" trick to get superposition to run, so i cant tell if those two is enough for superposition.

Will get around to testing it later tonight tho, so i can give feedback when i have tried :)

[ZeroFault]

@doitsujin

I was able to get Spirv-tools to build on mingw.

Make sure you have the required dependencies in the external directory of the spirv-tool project.

For 32 bit

First, make a build directory.

Then run inside the build directory,

CC="/usr/bin/i686-w64-mingw32-gcc" CXX="/usr/bin/i686-w64-mingw32-g++" cmake -G Ninja -DCMAKE_SYSTEM_NAME="Windows" -DCMAKE_FIND_ROOT_PATH=/usr/i686-w64-mingw32/ -DSPIRV_BUILD_COMPRESSION=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=install -DSPIRV_SKIP_TESTS=ON -DRE2_BUILD_TESTING=OFF <your/spirv-tools/directory/goes/here>

then run,

ninja install

For 64 bit

Make another directory build64.

Then run inside the build directory,

CC="/usr/bin/x86_64-w64-mingw32-gcc" CXX="/usr/bin/x86_64-w64-mingw32-g++" cmake -G Ninja -DCMAKE_SYSTEM_NAME="Windows" -DCMAKE_FIND_ROOT_PATH=/usr/x86_64-w64-mingw32/ -DSPIRV_BUILD_COMPRESSION=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=install -DSPIRV_SKIP_TESTS=ON -DRE2_BUILD_TESTING=OFF <your/spirv-tools/directory/goes/here>

ninja install

[SveSop]

@pingubot Doing the "rename d3d11.dll while starting the launcher" trick, those two optimizations --inline-entry-points-exhaustive --eliminate-dead-code-aggressive was enough to get it running.

I got three error messages from spirv-opt, but it did load.

error: 376: Illegal number of components (5) for TypeVector
error: 475: Illegal number of components (5) for TypeVector
error: 475: Illegal number of components (5) for TypeVector

I dont think getting the launcher window to load is a dxvk thing only, as that is bugged to pieces in both wine-staging and wine-release/wine-vulkan++

@doitsujin I did notice tho, that the colors seemed really "washed-out"(almost gray). I have also noticed a bit of "bland" colors when running Unigine Valley/Heaven too. Not sure if that is from dxvk or if that is nVidia Vulkan driver?

[pingubot]

@SveSop : have you actually tried running the benchmark ? Cause for me it also loads fine but break somewhere in the middle..

[SveSop]

@pingubot You are right. I cycled through a couple of scenes and was too quick to say it was working.

--legalize-hlsl does not output those 3 "TypeVector" errors, and go past scene 9 where the crash occurs. So i guess more than those 2 options are needed. Sorry... Back to testing :)

EDIT: Try adding --skip-validation

[volca02]

I managed to create a quick and dirty way of including spirv-tools as a subproject (via wrap). It uses a shell script to bridge between meson and cmake, and is probably not production ready, but could save someone a few minutes/hours of work to make it work. The build.sh can be used separately to build the spirv-tools as indicated by @ZeroFault

https://github.com/volca02/spirv-tools-meson

[pingubot]

@ZeroFault : how do you figure out the bad shaders which cause the issues ? Are those the last which get compiled or the last which are mentioned below the last log entry of compiling graphics pipeline ?

[SveSop]

@pingubot I would like to learn this too.
It is not the last line in the log-file tho, cos i tested that, and no dice.

I am not sure the shaders gets compiled one by one in order either.. perhaps depending on some multithread stuff? The shader compilation atleast? I also tested to delete all shaders mentioned in the logfile incase it somehow crashed before logging, but that ended up being all the .spv files..

So.. atleast with my extremely limited knowledge, the crashing shader is being logged just as the rest, but causes a crash that is not logged as an error :)

[SveSop]

@doitsujin Yay! :)
Todays commit : Commit definately changed something.
I was able to narrow it down to 2 shaders running Unigine Heaven.

I have attached the two shaders in "fixed.tar.gz", and the same ones BEFORE running : spirv-opt --inline-entry-points-exhaustive as "broken.tar.gz".

If someone else with nVidia + Unigine Heaven can put only those 2 shaders from "fixed.tar.gz" in their DXVK_SHADER_READ_PATH= folder and check?

Did some testing on Superposition, but it seems it breaks on more than just 2 shaders.. most likely due to a more advanced graphics engine perhaps? I did however was able to get it to run with a lot less spirv-opt runs, so afaik that commit contributed to something :)
broken.tar.gz
fixed.tar.gz

[ZeroFault]

@SveSop It's not working on that. What graphic settings are you running for Heaven? Different graphic settings generate different shaders.

[ZeroFault]

@pingubot It should be the last compiling graphics pipeline in the log to find the crashing shaders, if you are crashing.

[SveSop]

@ZeroFault 1080p DX11, Ultra, Tess=Disabled, Antialiasing=Disabled.
nVidia driver 390.25 - GTX 970

Heaven runs for me with ONLY those two shaders "pre-loaded" or what you would call the DXVK_SHADER_READ_PATH thing.
If i just run Heaven till it crashes with no fixing or anything, the last line in Heaven_d3d11.log reads:
debug: Compiling shader PS_baf4b1562256cda8afef46ceeb41d42ec0d0216f
It is not that shader that causes the crash. "Fixing" that shader does NOT work.

The two shaders is named:
PS_262a5081c076d190d2ae0de21f2cc8a38a01f134.spv
PS_2dbb78a2cc3f5b6a3c423b5b068de5f47ccc5217.spv
Are listed in the log-file, but is not listed or near the end of the log.. or has any errors on it.

EDIT: Oki, i guess the filename is generated out of some random hash compiled against whatever you have on your computer, and thus the NAME is not the same on every computer perhaps? So placing those two files for anyone but me won't work, even tho it is the correct shaderfile?

[pingubot]

@SveSop : great finding !yes, i can confirm using only these 2 fixed shaders, it works. Can you link that info to the other thread #70 , where @pdaniell-nv is active so that he also can compare those 2 working shaders with the non working ones.

[ZeroFault]

@SveSop Sorry, those 2 shader files did work for the linux driver. I also tested in Windows (Nvidia driver 390.77) and it does not require the fixed shader files for it to work. So they may have fixed it in the driver for Windows.

[SveSop]

@ZeroFault Are you saying that DXVK does not require any spirv-opt fixing in Windows running 390.77? Or just that there are OTHER shaders that needs fixing? :)

[ZeroFault]

@SveSop Not for Heaven's benchmark at least. I didn't have to use the spirv-opt --inline-entry-points-exhaustive at all.

[ZeroFault]

However, there are still failing shaders remaining, but they are not causing a crash on both Windows or Linux. --legalize-hlsl doesn't work for them.

debug: Compiling graphics pipeline...
debug:   vs  : VS_b759414d4739d2532c5f94cdd008c1233f0fd9be
debug:   fs  : PS_2dbb78a2cc3f5b6a3c423b5b068de5f47ccc5217
err:   DxvkGraphicsPipeline: Failed to compile pipeline

debug: Compiling graphics pipeline...
debug:   vs  : VS_b759414d4739d2532c5f94cdd008c1233f0fd9be
debug:   fs  : PS_262a5081c076d190d2ae0de21f2cc8a38a01f134
err:   DxvkGraphicsPipeline: Failed to compile pipeline

[SveSop]

Unigine Valley - Ultra no AA. Had to do spirv-opt --inline-entry-points-exhaustive on 1 shader to get it running. (Linux 390.25 driver)
Attaching working.zip and broken.zip.
broken.zip
working.zip

[oscarbg]

@ZeroFault
ok you say "--legalize-hlsl doesn't work for them" but are they trully valid?
I mean have you run spirv-val on this shaders to see if they have errors as --legalize-hlsl disables validation

--legalize-hlsl
Runs a series of optimizations that attempts to take SPIR-V..
Note this does not guarantee legal code. This option implies
--skip-validation.

[ZeroFault]

@oscarbg
Yes I ran the shaders through spirv-val, spirv-val didn't return any errors from them.

[oscarbg]

Ok..
then similar to a repro of ComputeMark2 I just posted on other thread
https://github.com/doitsujin/dxvk/issues/70
because that's what the NV engineer is following..

[doitsujin]

As a workaround, I added support for the SPIR-V optimizer to master, you can enable it with DXVK_SHADER_OPTIMIZE=1. It is not enabled by default because it may take well over 30 minutes to compile certain shaders with the optimizer enabled.

Please let me know if it works and if there's anything that works with Vesim's fork but not the master branch.

[SveSop]

@doitsujin
With : meson --cross-file build-win32.txt build.w32

~/WineHQ/dxvk/build.w32$ ninja
[5/124] Generating 'src/dxvk/dxvk@sta/hud_text_frag.h'.
../src/dxvk/hud/shaders/hud_text_frag.frag
[6/124] Generating 'src/dxvk/dxvk@sta/hud_text_vert.h'.
../src/dxvk/hud/shaders/hud_text_vert.vert
[76/124] Linking target src/dxgi/dxgi.dll.
FAILED: src/dxgi/dxgi.dll 
/usr/bin/i686-w64-mingw32-g++  -o src/dxgi/dxgi.dll 'src/dxgi/dxgi@sha/dxgi_adapter.cpp.obj' 'src/dxgi/dxgi@sha/dxgi_device.cpp.obj' 'src/dxgi/dxgi@sha/dxgi_enums.cpp.obj' 'src/dxgi/dxgi@sha/dxgi_factory.cpp.obj' 'src/dxgi/dxgi@sha/dxgi_main.cpp.obj' 'src/dxgi/dxgi@sha/dxgi_output.cpp.obj' 'src/dxgi/dxgi@sha/dxgi_presenter.cpp.obj' 'src/dxgi/dxgi@sha/dxgi_swapchain.cpp.obj' -Wl,--no-undefined -Wl,--as-needed -shared -Wl,-soname,dxgi.dll -Wl,--start-group -Wl,--out-implib=src/dxgi/libdxgi.dll.a src/util/libutil.a src/dxvk/libdxvk.a src/spirv/libspirv.a /home/cybermax/WineHQ/dxvk/lib32/vulkan-1.lib /home/cybermax/WineHQ/dxvk/lib32/libSPIRV-Tools.a /home/cybermax/WineHQ/dxvk/lib32/libSPIRV-Tools-opt.a -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group -static -static-libgcc -static-libstdc++ -Wl,--add-stdcall-alias 
WineHQ/dxvk/lib32/libSPIRV-Tools-opt.a(loop_descriptor.cpp.obj):loop_descriptor.cpp:(.text+0xe5b): undefined reference to `__divmoddi4'
WineHQ/dxvk/lib32/libSPIRV-Tools-opt.a(loop_descriptor.cpp.obj):loop_descriptor.cpp:(.text+0xeb8): undefined reference to `__divmoddi4'
WineHQ/dxvk/lib32/libSPIRV-Tools-opt.a(loop_descriptor.cpp.obj):loop_descriptor.cpp:(.text+0x50f5): undefined reference to `__divmoddi4'
collect2: error: ld returned 1 exit status
[81/124] Compiling C++ object 'src/dxbc/dxbc@sta/dxbc_module.cpp.obj'.
ninja: build stopped: subcommand failed.

build-win64.txt worked tho...

EDIT: Good news tho, as Superposition is a 64-bit app, it loaded without DXVK_SHADER_READ_PATH= when i used DXVK_SHADER_OPTIMIZE=1

[SveSop]

Unable to run with DXVK_DEBUG_LAYERS=1 and "vk_layer_settings.txt"

lunarg_core_validation.report_flags=error,warn,perf
lunarg_core_validation.log_filename=./vulkanlog.txt
lunarg_core_validation.debug_action=VK_DBG_LAYER_ACTION_LOG_MSG

lunarg_parameter_validation.report_flags=error,warn,perf
lunarg_parameter_validation.log_filename=./vulkanlog.txt
lunarg_parameter_validation.debug_action=VK_DBG_LAYER_ACTION_LOG_MSG

lunarg_object_tracker.report_flags=error,warn,perf
lunarg_object_tracker.log_filename=./vulkanlog.txt
lunarg_object_tracker.debug_action=VK_DBG_LAYER_ACTION_LOG_MSG

google_threading.report_flags=error,warn,perf
google_threading.log_filename=./vulkanlog.txt
google_threading.debug_action=VK_DBG_LAYER_ACTION_LOG_MSG

google_unique_objects.report_flags=error,warn,perf
google_unique_objects.log_filename=./vulkanlog.txt
google_unique_objects.debug_action=VK_DBG_LAYER_ACTION_LOG_MSG

Crash if i do that.. for some reason.

Anyway.. attaching d3d11.log and dxgi.log for ref.
superposition_d3d11.log
superposition_dxgi.log

[doitsujin]

Can anyone reproduce the 32-bit build issues that @SveSop has? It works for me.

[volca02]

I can compile both 32bit and 64bit versions without issues here.

i686-w64-mingw32-g++ (GCC) 7.3.0

x86_64-w64-mingw32-g++ (GCC) 7.3.0

[SveSop]

@volca02 Hmm.. Is "Ubuntu distro" generally crap when it comes to doing any compiling?
At work atm, but 17.10 does NOT come with mingw-w64 7.3 (nor gcc 7.3). Not sure there is any ppa providing mingw-w64 7.3 either.... Think it is 6.3 or something, but have to check when i get home.

gcc-mingw-w64 (6.3.0-14ubuntu3+19.3)
gcc-mingw-w64-i686 (6.3.0-14ubuntu3+19.3)

Perhaps

[volca02]

@SveSop You could try rebuilding the spirv-tools libs for yourself and replace them in dxvk/lib. The compile options to do so are present in this very issue thread.

[SveSop]

@volca02 So, is the spirv-tools lib i need to compile in different from that found in the VulkanSDK folder in 1.0.68.0/x86_64/lib/glslang/ after compiling that?

Cos putting those in ended with a segfault when compiling....

Oh, yeah, i see that its a special meson-spirv-tools thing, that is so self-explanatory to make that only fucking retards cant do that... LIKE ME :(

[SveSop]

@volca02
Yeah, i need to be spoon-fed those simple things, so i ended up making a patch to your script. It will tell you to edit the build.sh if you have not put in your spirv-tools source folder, and ask if you want to build 64 bit and 32 bit.

Use it if you like :)

@doitsujin Yes, compiling @volca02 source and replacing the files in dxvk made it compiled fine, and works without any DXVK_SHADER_READ_PATH on Unigine Heaven/Valley/Superposition. Takes a wee bit longer to start, but that would be expected.

script.zip

[pingubot]

@doitsujin i want to give you a short update about the current status after your recent fixes and workarounds for nvidia ( [dxbc] Implemented new workaround for depth-compare ops on Nvidia ):

Unigine Heaven works fine !
Unigine Valley works fine !
Unigine Supoerposition works, but shows the washed out colours.
Crysis2 works fine, will update my bug report about it

Thank you very much for all your hard work getting dxvk also running on nvidia ! Also many thanks to @pdaniell-nv for helping out finding issues in the shaders and drivers.

[pdaniell-nv]

Awesome! Superposition might be washed out because it uses 14 constant buffers, but we only support 12. I'm not sure there is an easy work-around. We're investigating increasing our limit, but if we do it probably won't be in a driver until next month at the earliest.

[SveSop]

Great job @doitsujin ! Valley and Heaven now runs without the need for DXVK_SHADER_OPTIMIZE=1.
No errors from shader compiles in debug log. Attaching both for reference :)

Heaven_d3d11.log
Valley_d3d11.log

[ZeroFault]

@pdaniell-nv
Thanks a lot for the help!

I may have found more broken shaders in BioShock Infinite. As of the latest DXVK commit. These shaders are crashing the Nvidia driver. I ran these drivers through spirv-validator and nothing came back wrong. Also, no vulkan debug layer errors.

Tested on:
OS: Windows 10
Driver: 390.77
Graphics Card: GTX 1080
I organized the shaders in folders according to the pipeline they were bound to at the time as I wasn't sure which specific shader was crashing it in the pipeline.

bioshock_failing_shaders.zip

[doitsujin]

@ZeroFault Bioshock Infinite has other issues at the moment (it does not work on RADV either), might not necessarily be the shaders. Might be worth taking a look regardless.

[oscarbg]

@pdaniell-nv IMHO makes sense for Nvidia to increase the Vulkan limits which are below D3D1x standards.. as a reason this "Graphics API translator" project should be enough.. if not let me say: perhaps @doitsujin can make some changes but then all others projects will need similar "massages" to work on Nvidia..
I'm referring to the new VKD3D project: a D3D12->Vulkan wrapper by Codeweavers/Wine (https://source.winehq.org/git/vkd3d.git/)
this project doesn't generate (yet) independent drop-in D3D12 library with exact entrypoints to Windows one to test D3D12 games like DXVK.. but as soon as integrated in Wine and is mature enough to start running games will probably hit these same limitations as some game uses >12 cbuffers.. (perhaps @jozefkucia can share thoughts here about this issue)
IMHO of course, and no pressure to NV to implement fast.. take your time and do it well can be a month, two or whatever, after all we have been with this limitation for two years and no one has complained..
and we also want to see Vulkan evolve! and as said in my initial tweet hope to see new Vulkan features at GDC and traditional day 1 NV driver..
as a final note: we may have been lucky with Wine D3D11->OpenGL translator in Superposition (assuming it translates these constant buffers to OGL UBOs) because I'm just seeing on your lastest GPUs and driver NV OpenGL limits and I see GL_MAX_*_UNIFORM_BLOCKS=14 (http://opengl.gpuinfo.org/gl_generatereport.php?reportID=2460) so it has all UBOs needed to run..
and thanks for you support..

[doitsujin]

@oscarbg

we may have been lucky with Wine D3D11->OpenGL translator in Superposition (assuming it translates these constant buffers to OGL UBOs) because I'm just seeing on your lastest GPUs and driver NV OpenGL limits and I see GL_MAX_*_UNIFORM_BLOCKS=14

I think that's the minimum number of UBOs that a GL implementation is required to support, whereas for Vulkan it really is only 12. D3D11 on the other hand requires 14, just like OpenGL, and reserves one extra slot for the driver/hardware, but DXVK doesn't make use of that.

[oscarbg]

@doitsujin thanks for clarification.. I wasn't aware of minimum OpenGL limits, but clearly NV is exposing the minimum as other OpenGL drivers seem to expose more, similar as in Vulkan case..
anyway as said other projects like VKD3D will probably hit similar issues in the near future on Nvidia GPUs if they not take these limits into account very carefully.. IMHO so seems better if Nvidia uppers limits as needed a little (to D3D minimums).. can take his time of course..

[pdaniell-nv]

@doitsujin How many descriptor sets bindings does DXVK require? I assume since it's translating from DX which has no such concept it only needs one? Specifically I'm asking about the maxBoundDescriptorSets limit DXVK requires. Thanks.

[doitsujin]

@pdaniell-nv yes, it's only one.

[pchome]

@pdaniell-nv
With recent Nvidia-related workarounds in DXVK the only one "bad" shader left for The Witcher 3:
PS_213023725a464e8cb1639aa136090fc327d4faa7.dxbc.gz
PS_213023725a464e8cb1639aa136090fc327d4faa7.spv.gz
PS_213023725a464e8cb1639aa136090fc327d4faa7_fixed.spv.gz

I hope there is nothing new there. Also it's small so attaching dxbc-disasm.exe (from DXVK tests) output, in case it can be useful and will save your time:

//
// Generated by Microsoft (R) D3D Shader Disassembler
//
//
// Input signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// SV_Position              0   xyzw        0      POS   float   xy  
// TEXCOORD                 0   xy          1     NONE   float       
//
//
// Output signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// SV_Target                0   x           0   TARGET   float   x   
// SV_Target                1   x           1   TARGET   float   x   
// SV_Target                2   x           2   TARGET   float   x   
// SV_Target                3   x           3   TARGET   float   x   
// SV_Target                4   x           4   TARGET   float   x   
// SV_Target                5   x           5   TARGET   float   x   
// SV_Target                6   x           6   TARGET   float   x   
// SV_Target                7   x           7   TARGET   float   x   
//
      ps_5_0
      dcl_globalFlags refactoringAllowed
      dcl_constantbuffer cb0[1], immediateIndexed
      dcl_constantbuffer cb2[2], immediateIndexed
      dcl_sampler s0, mode_default
      dcl_resource_texture2d (float,float,float,float) t0
      dcl_input_ps_siv linear noperspective v0.xy, position
      dcl_output o0.x
      dcl_output o1.x
      dcl_output o2.x
      dcl_output o3.x
      dcl_output o4.x
      dcl_output o5.x
      dcl_output o6.x
      dcl_output o7.x
      dcl_temps 2
   0: round_ni r0.xy, v0.xyxx
   1: mad r0.xy, r0.xyxx, l(4.000000, 4.000000, 0.000000, 0.000000), cb2[1].xyxx
   2: add r0.xy, r0.xyxx, l(0.500000, 0.500000, 0.000000, 0.000000)
   3: mul r0.xy, r0.xyxx, cb0[0].zwzz
   4: gather4_indexable(texture2d)(float,float,float,float) r1.xyzw, r0.xyxx, t0.xyzw, s0.x
   5: gather4_aoffimmi_indexable(2,0,0)(texture2d)(float,float,float,float) r0.xyzw, r0.xyxx, t0.xyzw, s0.x
   6: mov o0.x, r1.w
   7: mov o1.x, r1.z
   8: mov o2.x, r0.w
   9: mov o3.x, r0.z
  10: mov o4.x, r1.x
  11: mov o5.x, r1.y
  12: mov o6.x, r0.x
  13: mov o7.x, r0.y
  14: ret 
// Approximately 0 instruction slots used

[pdaniell-nv]

@pchome That Witcher3 shader has found another bug in our compiler. This time we're not handling the 'Component' operand of OpImageGather correctly. We crash if a type other than signed integer is used. To work around the issue, make sure the 'Component' operand is a signed integer. So instead of:

   %116 = OpImageGather %v4float %114 %110 %uint_0

subtitute that with:
%116 = OpImageGather %v4float %114 %110 %int_0
from the shader above. Same with the other OpImageGather operation used in that shader.

Our compiler will be fixed shortly. Thanks for the information.

[ZeroFault]

@pdaniell-nv

The OpImageGather bug seems to be what is causing the Bioshock Infinite shaders I attached a few days ago to crash as well. Thank you for the information!

[doitsujin]

Workaround for that particular issue implemented in c3cf65c0150d03e53d8af3fbe41ce8bc96f911fd.

[ZeroFault]

@pdaniell-nv
Actually, there is only 1 shader failing in the driver now in Bioshock Infinite. It was affected by the OpImageGather bug as well, but it seems like that wasn't the only issue. Could you please help to figure this one out?

bioshock_shaderfailing2.zip

[pdaniell-nv]

It's not immediately obvious to me what's wrong with this shader. Does it crash the compiler or just not behave correctly?

[ZeroFault]

@pdaniell-nv It's crashing in the driver when vkCreateComputePipeline is called with that shader bound. There are no vulkan debug layer errors. There are warnings, but they don't look associated with this problem. Would a vktrace help you here?

[pdaniell-nv]

@ZeroFault What driver version are you seeing this with?

[ZeroFault]

@pdaniell-nv

Windows 10
Driver 391.01
GTX 1080

[pdaniell-nv]

Thanks for the extra info and the quick response. It appears this shader is using compute shared memory beyond the limit we advertise in maxComputeSharedMemorySize, which is 48kB. What's strange is that this shader appears to use more than 32kB of group shared memory, which I thought was the limit of DX11.

The relevant parts of the shader are:

%uint = OpTypeInt 32 0
%uint_25088 = OpConstant %uint 25088
%_arr_uint_uint_25088 = OpTypeArray %uint %uint_25088
%_ptr_Workgroup__arr_uint_uint_25088 = OpTypePointer Workgroup %_arr_uint_uint_25088
%g0 = OpVariable %_ptr_Workgroup__arr_uint_uint_25088 Workgroup

Perhaps that size was generated incorrectly and it should have been 6272? As it stands it needs 100,352 bytes of memory. We may be able to increase our limit for some GPUs, but for GTX 1080 I don't think we can do 100,352 bytes. The max is probably closer to 96kB.

[ZeroFault]

@pdaniell-nv
Thanks for the info! We'll take a look in to that later.

Also, I have a found another shader in BioShock Infinite that is definitely triggering a segfault in the driver.
bioshock_failingshaders3.zip

[pdaniell-nv]

Hmm that one is working fine for me.

[ZeroFault]

@pdaniell-nv

Hmm, I turned off vulkan debug layers (Vulkan SDK 1.0.68) and it doesn't crash anymore on that shader. It definitely says it crashed inside the driver with it on though. Weird.

[doitsujin]

@ZeroFault @pdaniell-nv there was indeed a bug with how that size was calculated, since the limit in D3D11 is 32kB. Fixed in 3efd437310d560d46b05179e8eb3ad7229c5adae.

Background: The MSDN doc for dcl_tgsm_raw says that the size is given in bytes, whereas the DXBC format documentation says that it's given in dwords. Turns out MSDN is correct.

[nairaner]

@pdaniell-nv
I found 2 new shaders that cause the crash when trying out Dauntless

dauntless_crash.zip

[thegamingrestoration]

Software information

ECHO
http://store.steampowered.com/app/551770/ECHO/
Unreal Engine 4 game

System information

• GPU: Nvidia Geforce GT 1030
• Driver: Nvidia proprietary driver 390.42
• Wine version: wine-staging 3.4
• DXVK version: 2346a2a

Apitrace file(s)

• Echo-Win64-Shipping.trace
  https://www.dropbox.com/s/3a5egkllgwueagn/Echo-Win64-Shipping.trace.tar.gz?dl=0

Log files

• d3d11.log: Echo-Win64-Shipping_d3d11.log
• dxgi.log: Echo-Win64-Shipping_dxgi.log
• FirstRun.log: FirstRun.log

TWO BROKEN SHADERS: ECHO_CS.tar.gz

The game crashes after the first loading screen unless you optimize these two shaders with the SPIRV-Tools.

[pdaniell-nv]

@nairaner for the Dauntless shaders our compiler appears to be chocking on the OpAtomicIAdd ops used with Buffer images. I'm not sure why yet, the SPIR-V looks valid. I'm asking internally for help.

@60fpshacks the Echo failures appears to be the same.

I'll report back when I know more.

[ZeroFault]

@doitsujin @pdaniell-nv It says this in the vulkan spec.
The Image Format of an OpTypeImage declaration must not be Unknown, for variables which are used for OpAtomic* operations.
Could this be related to the issue?

[ZeroFault]

Yeah, It looks like it. A shader with that issue compiled fine when I manually applied a format to it.

[doitsujin]

Hm, obviously a bug on my end then. Why does this even pass validation?

This is going to be nasty to fix because there is no way to deduce the format from the UAV declaration (which is also why shaderStorageImageReadWithoutFormat is still required).

Guess I'll have to alter the image type declaration dynamically based on whether it is accessed through atomic operations (in which case only 32-bit integer formats are allowed), and on GPUs which do not support shaderStorageImageReadWithoutFormat, when a typed load is used (which is possible if we remove typed read support for anything but 32-bit scalar formats on those GPUs). Although I really don't like the latter because it may make incorrect assumptions about the format in case a game uses typed loads on "unsupported" formats.

[doitsujin]

@ZeroFault @60fpshacks @pdaniell-nv
Commit fcff10aae7b93c7e93c2f7e5cfbeb25bb4f3ae14 should fix this issue for unsigned image formats. I still have to make sure that the atomic operations themselves use the correct integer type though, they are currently hardcoded to use unsigned integers. There might still be issues.

[thegamingrestoration]

Ok, commit fcff10a fixes Echo, now it boots and goes ingame without issues.

[doitsujin]

FYI, fcff10aae7b93c7e93c2f7e5cfbeb25bb4f3ae14 introduces a regression in that most shaders define the same image type multiple times. In practice, this seems to work on all drivers, but it is technically invalid SPIR-V. Sadly, fixing this issue will require major refactoring.

[ZeroFault]

@pdaniell-nv

I believe I found another translated shader, from Star Citizen, that is causing a crash in the Nvidia driver. I am not receiving any Vulkan debug layer errors and no SPIR-V validator errors.

GPU: GTX 1080
OS: Windows 10
Driver: 391.35

PS_2bde6343759f1da74a3db07ca69473295c20888d.txt

I believe that the line %456 = OpImageTexelPointer %_ptr_Image_uint %u7 %453 %uint_0_12 is the cause since if I remove it, the shader doesn't crash the driver anymore.

[nairaner]

I got a little bit different crash when playing Path of Exile. It probably is bug in dxvk, but I don't have time today and will try to get VkShaderModuleCreateInfo later
PoENvidiaCrash.log
PathOfExile_x64_d3d11.log
PathOfExile_x64_dxgi.log

Edit: I shouldn't use DXVK_SHADER_READ_PATH

[oscarbg]

@ZeroFault seeing past Nvidia uint/int bugs perhaps you can check in the meanwhile if changing uint to int compiles OK? I.e:

456 = OpImageTexelPointer %_ptr_Image_int %u7 %453 %int_0_12
Of course changing accordingly definitions of u7 and int_0_12 definitions to be signed integer

[doitsujin]

@ZeroFault please post the full DXBC shader, it looks like there are some unsupported instructions which don't get compiled. Otherwise there shouldn't be this stray OpImageTexelPointer instruction that is never even used.

@oscarbg using signed integers in this case would be incorrect.

[ZeroFault]

@doitsujin

Here is the dxbc shader.
starcitizen_badshader.zip

You're right that it it doesn't support the DXBC ImmAtomicUmin instruction. However, it should still be legal to declare that pointer anyway right?

[doitsujin]

@ZeroFault please test with d2c6ec5c41b39516c1b91ae9d9f9c66142beea88.

The code should be valid, the atomic operation is performed on a 2D array image and the location parameter is a three-component signed integer vector. More or less identical to what glslang generates for this GLSL shader:

#version 450

layout(set = 0, binding = 0, r32ui) uniform uimage2DArray img;

void main() {
        imageAtomicMax(img, ivec3(gl_GlobalInvocationID), 1u);
}

[ZeroFault]

@doitsujin
It compiles fine now with d2c6ec5.

Does the shader actually need to use pointer to be valid or something? So is this a bug with Nvidia again?

[nairaner]

This time i found a little bit different crash, because it resets the driver. I combined logs in case dxvk keeps working, but it doesn't seem to be a case.

I believe it can be reliably reproduced by entering Vault map in Path of Exile, so i should be able to get some more details if needed
nvidia-bug-report.log.gz
last_shaders.zip
combined.log

[AccountOneOff]

Heroes of the Storm hangs in any online game mode.

• GPU: GTX 960
• Driver: Nvidia 390.48
• Wine version: Staging 3.5
• DXVK version: acaf6c9

HeroesOfTheStorm_x64_d3d11.log
HeroesOfTheStorm_x64_dxgi.log

[ZeroFault]

@pdaniell-nv

I believe we found an issue with Vulkan buffer view creation with a Uniform Texel Buffer. When I create a buffer view with a range exceeding the maxTexelBufferElements limit for my GPU, it fails. I believe the range member passed into vkcreatebufferview is supposed to be in _bytes_ and not an element count as it seemingly looks like it is being interpreted as.

OS: Windows 10
GPU: GTX 1080
Driver: 391.35

I attached a test application application demonstrating this.

texelbuffertest.zip

[doitsujin]

274 seems to be related to this issue as well. To clarify, what happens there is that the game tries to create a buffer view where the number of bytes exceeds maxTexelBufferElements, but the number of elements does not, and the creation of the buffer view fails as a consequence.

[pdaniell-nv]

Thanks for the heads up. We'll fix this in our next driver.

[pdaniell-nv]

This should be fixed with 396.18.05, which can be found here: https://developer.nvidia.com/vulkan-driver.

[doitsujin]

Good to hear.

I'm going to close this old meta bug now because there don't seem to be any general Nvidia-specific issues left, if something new comes up, please upen a new issue.

[Kerrung]

@pdaniell-nv typo
https://developer.nvidia.com/vulkan-driver
369 instead 396

[pdaniell-nv]

Doh, thanks. It's now fixed.