Dxvk: Pipeline cache

[ryao]

@doitsujin I decided to make another issue to move the pipeline discussion out of #646.

It occurs to me that the pipelines produced by Vulkan cannot be cached because they are driver specific (and therefore would need to be invalidated by driver updates, provided that they are possible to cache in the first place).

However, the shaders that games pass to DXVK that are processed to create the pipelines can be cached. If DXVK were to dump those into a SQLite database indexed by checksum (or perhaps binary name), it could look up all shaders passed to it in past sessions. If it indexed by binary name, it would be able to begin precompiling pipelines as soon as it is initialized.

We probably would want some sort of decay to delete stale data that is no longer relevant due to game updates and maybe even a second level of decay where we keep the checksum around in case it is just infrequently used (such that we could apply the ARC algorithm or something similar).

This would decouple shader cache invalidation from driver version such that driver updates would not slow things down. If the DXBC is cached, then updates and improvements to DXVK could reuse the cache.

This is just a rough idea, but I wanted to put it out there for consideration. What do you think?

[doitsujin]

This idea is not new. I've tried to implement this several times (except the SQLite bit because you're making it far too complicated), but failed to get it to work.

[ryao]

@doitsujin In what way did it fail to work? Would you push your last attempt into a branch for others to see if they can get it working?

[doitsujin]

No, since the code it was based on has been removed. It sort of worked, but using a pipeline compiled from the cache would result in a driver crash for some reason.

[pchome]

@ryao

89 ?

[SpookySkeletons]

I've been looking for a good by-hand method of exporting DXBC to the driver in the right way so I can just package script something to regen my driver cache. Native support would be neat but is there currently any method to do this by hand?

[ryao]

@SpookySkeletons DXBC is not sent to the driver. It is compiled into SPIR-V, which is sent to the driver via Vulkan API calls. You would need to capture a trace of the API calls and buffers and then replay those in a separate binary to achieve what you want. That requires doing programming.

[doitsujin]

There's a working implementation in the dxvk-shader-cache-v2 branch. Enable it by setting dxvk.useStateCache = True in the configuration file. Right now, the cache file location is hardcoded to be dxvk.scache next to the game's executable.

Like the original implementation, it caches only the pipeline state vectors, which has the following implications:

• The cache files are relatively small (the highest I've seen so far is 12 MB for the Unity Blacksmith demo with well over 6000 pipelines)
• They will survive driver updates and can even be shared with other users if necessary (might become useful in the future, thinking of games like Overwatch here)
• They will also survive most DXVK updates, unless incompatible changes to the pipeline state have to be made

One of the problems that I've run into with past attempts is that compiling the exact same pipeline multiple times at the same time apparently crashes drivers. The new code works around this issue, and now it appears to be stable.

[jp7677]

@doitsujin Cool work!
Do you know if the lock to prevent concurrent compilation of the exact same pipeline affects performance visibly? Is this only an issue on RADV or also on NVidia?

[doitsujin]

Not sure if this particular issue affects Nvidia, but past attempts to implement this crashed randomly on both drivers. Need some time to investigate.

[jp7677]

Impressive, with this branch I get the following results on my GTX 1080 with the Arkham Knight built-in Benchmark:

DXVK cache disabled:
Min 17 Max 77 Avg 48
With pre-filled DXVK cache (DXVK: Read 6198 valid state cache entries):
Min 33 Max 90 Avg 54

I invalidated the NVidia cache between runs (using__GL_SHADER_DISK_CACHE_PATH env variable).

Once the Nvidia cache is filled I get more or less identical results when using or not using the DXVK cache (Min 32 Max 91 Avg 54). Very cool that driver updates will no longer have such a huge effect.

[jp7677]

PS: I guess it would be nice if you could keep the filename similar to the pattern of the log files, something like processname_dxvk.scache, thus for BatmanAK.exe:

BatmanAK_d3d11.log
BatmanAK_dxgi.log
BatmanAK_dxvk.scache

[doitsujin]

These details are subject to change anyway, including the way it is currently enabled.

[jp7677]

PPS: Something totally unrelated. I'd stumbled a little bit in the beginning with the configuration file because I used a lower case "true". Clearly my fault, you stated it correctly in you message and the wiki says the same. That said being case sensitive for boolean value feels a little bit strict. Would you prefer to keep it like this or would you accept a PR that at least also allows lowercase "true"/"false" and uppercase "TRUE"/FALSE" as well?

[jp7677]

These details are subject to change anyway, including the way it is currently enabled.

Ah, alright. Sorry, forgot that this is still work in progress ;)

[ryao]

@doitsujin I am pleasantly surprised. I had been concerned that I would not be helping when making suggestions unless I did some profiling and triage work myself. I have gradually made progress in my spare time toward those, but I am not in a position to do them yet. I am both new to this and have multiple obligations elsewhere, so it will be a while before I can do those things. However, I will look at your changes and try them out when I get a chance. Thanks for your amazing response to my feedback. :)

[Thaodan]

Not sure if its to early but could you use XDG_CACHE_HOME or something similar on linux? Writing to $PWD seems kinda dirty when it comes to cached/tmp files.

[ryao]

DXVK is actually a Windows DLL, so it would need to use the Windows equivalent. It is not aware that it is running on Linux. Anyway, as @doitsujin said, the details are subject to change. It is not even in master yet.

[daenney]

XDG_CACHE_HOME would probably map to LOCALAPPDATA on Windows. Typically you'd get $XDG_CACHE_HOME/<appname> on anything following the XDG spec whereas on Windows %LOCALAPPDATA%\<appname>\cache would be fairly common.

[Thaodan]

Looks like that
%LOCALAPPDATA%

[shmerl]

DXVK is actually a Windows DLL, so it would need to use the Windows equivalent.

What stops dxvk from detecting an OS (or may be the fact that it's running in Wine)? It can use locations conditionally (proper one on Linux and on Windows accordingly). A lot of applications are doing that.

For instance, dxvk can attempt to check if Linux syscall is available, and use that as an indicator of Linux usage.

Related.

[ryao]

@shmerl It is not necessary. Using a Windows specific path also would mean that each wine prefix’s shader cache is independent, which would allow for the distribution of game-specific shader caches. I guess it would make sense to support an environment variable to override the default location, but that is @doitsujin’s call.

[shmerl]

Using a Windows specific path also would mean that each wine prefix’s shader cache is independent, which would allow for the distribution of game-specific shader caches.

You can use subdirectories in $XDG_CACHE_HOME too, based on prefix and binary name. I.e. for example: $HOME/.cache/dxvk/prefix_abc/game_xyz.exe/.

[Thaodan]

Using a Windows specific path also would mean that each wine prefix’s
shader cache is independent, which would allow for the distribution of game-
specific >shader caches.

You can use subdirectories in $XDG_CACHE_HOME too, based on prefix and
binary name.
Thats what I have thought of.
Clearing the cache should be transparent mapping cache to that dir makes that
easier.

[ryao]

@shmerl The wine developers would likely advise against this sort of thing:

https://wiki.winehq.org/FAQ#I.27m_writing_a_Windows_app._How_can_it_detect_if_it.27s_running_under_Wine.3F

I agree with them.

[Thaodan]

There's no need to "detect" if running in wine if dxvk is build as winelib.

[ryao]

@Thaodan It is distributed as a native DLL that you can use on Windows. It is built as a winelib for development purposes. Those builds are not distributed.

If using $XDG_CACHE_DIR is important, then Steam could set the environment variable to point to $XDG_CACHE_DIR. I don’t think that DXVK is the right place to put that. The decision is up to @doitsujin. He is both the one doing the implementation work and the project lead. My suggestion is to use the correct Windows location by default and then allow an override through an environment variable.

[Thaodan]

So thats not a feature in development?

Anyway testing if an env var is set not os
specific.
Another way would be to load a global conf
where something like dxvk.cache can be set.

[shmerl]

I'd say it's moot. Using standard cache location for dxvk cache isn't very different from using it for driver cache. The fact that it's Wine doesn't change much.

[ryao]

If you can set an environment variable to activate a behavior that switches to the $XDG_CACHE_DIR directory, then you can set an environment variable to the correct path.

How the location of the cache is determined a minor detail compared to reviewing the code implementing the cache itself. The location could easily be changed in a follow up patch if @doitsujin wants to do that.

I think anything that makes it detect its existence inside of Wine is a mistake and that it should be handled externally via the Windows environment variables being passed to it. A simple environment variable passed by steam would do what you want without putting detection code into DXVK.

[ryao]

At the moment, you need to pass an environment variable to turn it on. I imagine that it could be changed to be passed a filesystem path for the cache to turn it on. Then there would be no default location.

[shmerl]

Environment variables sounds like a good solution. But you also can check if for example XDG_CACHE_DIR or HOME are set. They shouldn't be set on Windows, so it's not very different.

[ryao]

It seems that I misremembered what I read @doitsujin write. This is not currently controlled via an environment variable. It would be nice to have that feature.

[ryao]

@shmerl I should have read the code before posting that. There is already an environment variable called DXVK_STATE_CACHE_PATH. Set that and you can put the cache where you want. See DxvkStateCache::getCacheFileName():

https://github.com/doitsujin/dxvk/blob/dxvk-shader-cache-v2/src/dxvk/dxvk_state_cache.cpp#L376

Edit: And I should have read the documentation. There is also a dxvk.conf file where this can be turned on/off:

https://github.com/doitsujin/dxvk/wiki/Configuration

[ryao]

Also, it seems like the default location under SteamPlay is inside the game cache. For example, on Rise of Nations, it goes into ./.local/share/Steam/SteamApps/common/Rise of Nations/patriots.dxvk-cache (my steam install is in .local because it is old).

[ryao]

@doitsujin From my initial read of this, it just stores entries in an append only file. That could work for an initial implementation, although this could be problematic if there is a game under heavy development and the cache isn't invalidated after game updates. Perhaps that could be handled externally.

Also, this is nitpicky, but the code as written won't try to create directories if the path given to it is invalid. It would be a nice touch, but it is something that could be left to someone else to implement. Not having it autmatically make directories for a first stab is alright.

I still need to read it more in depth. I only did a very cursory read and some basic tests. I don't actually have many Direct3D 11 games to test. At the moment, my Steam library only has 1 such game as far as I am aware.

[ryao]

One other thing. For those that have difficulty appreciating the joys of cross-compiling software, here is a 32-bit build:

wget https://dev.gentoo.org/~ryao/dist/dxvk-win32-v0.72-39-g20c89c3.txz{,.sig}
gpg --verify dxvk-win32-v0.72-39-g20c89c3.txz.sig dxvk-win32-v0.72-39-g20c89c3.txz

The standard warnings about not downloading and executing binaries from strangers on the internet apply. The Gentoo users here ought to consider my key trustworthy, but I would expect them to build their own binaries. Someone on reddit goofed when trying to build this, so I decided to share my own build. The binaries were stripped post build to save space on the web server.

[aufkrawall]

Got already 18MB of disk space for cached shaders in that crazy HotS game. :grin: Well, this feature could be a game changer.

[doitsujin]

This is now implemented in master and enabled by default.

[ryao]

@doitsujin Awesome.

I have what I hope will be a quick question (or two). Is the DXBC distributed with the games or do the games include HLSL and run it through the HLSL compiler before passing it to DXVK? If it is the latter, would implementing a HLSL front end to DXVK’s compiler and replacing Microsoft’s HLSL compiler with it help? I understand what a large task writing a compiler front end is, so I don’t seriously expect you to do it if the answer is yes, but I am curious.

[shmerl]

There should be no point for games to ship HLSL, when they can use DXBC. That's a major feature that OpenGL was lacking and Vulkan introduced (shader bytecode in the form of SPIR-V).

[ryao]

@shmerl I wasn't sure if they had shipped the HLSL or not. Thanks for clearing that up.