Dxvk: Using compression for pipeline cache?

[shmerl]

I run a test and compressed my current Shadow Warrior 2 pipeline cache (using pixz). It got reduced from 6,022,860 bytes to 140,336 bytes. That's almost 97.67% space saving. pixz is a rather strong compression though (lzma2), but may be using something more light on the CPU will still reduce storage space (better I/O time and etc.).

Not really critical feature, but with a lot of games, this starts accumulating.

[doitsujin]

Seriously, one ~10MB cache file per game is an issue when the average game needs 10-60GB of space?

I'm aware that they compress very well because a lot of the data consists of zeroes, but IMHO this pointless. DXVK not only has to be able to read the cache file fast (which should be a non-issue even when compressed), but it has to be able to append new entries to a file without rewriting the entire file every single time, and most compression algorithms just don't allow that. The only somewhat viable option would be a simple RLE or something like that, obviously with reduced savings.

[popsUlfr]

Might consider giving BTRFS a shot as filesystem, I'm getting some nice compression ratios for instance for Ethan Carter Redux :

sudo compsize EthanCarter-Win64-Shipping.dxvk-cache
Type       Perc     Disk Usage   Uncompressed Referenced  
TOTAL       13%      944K         7.0M         7.0M       
lzo         13%      944K         7.0M         7.0M

The shader caches don't seem to get very big though.

[shmerl]

I'm using BTRFS for actual games installation, but using XFS for root partition (including home), since it's generally faster.

it has to be able to append new entries to a file without rewriting the entire file every single time

Does this happen on the fly, or at some periods? If periodically, rewriting the file might be less critical, but yeah, appending is a lot less intense than rewriting the whole file. However given that compressed file is a lot smaller too, writing I/O won't be such an issue as well.

[lieff]

Also note that good compression takes lot of time (much more than decompression), so re-writing all file is not a good idea. Some specifics algs can be used to append compressed data, but note that changing this deprecates all already created caches.

[K0bin]

The files are usually around 10mb, 30mb at worst, why are you even discussing how to compress them?

[mleise]

Just to add a different angle: Compression would speed up the initial load of the game from a mechanical hard drive. That doesn't make it a high priority though.

[Leopard1907]

First world problems...

[camdenorrb]

Yeah, speeding up the reading would be nice <3

[Cloudef]

Using fuse for transparent compression of a directory (mount) is also option. (DXVK doesn't have to support this)

[doitsujin]

With the very limited number of games that would actually benefit from this feature, I don't think spending too much time on it is worth it, especially since it would either break existing state caches, or force me to maintain a second code path.