Dxvk: Out of Address Space Megaissue

[Joshua-Ashton]

I'm making this issue to consolidate a bunch of issues on the D9VK tracker which are all essentially the same thing, running out of address space (there's like 4 separate reports of the same thing in Borderlands 2 and Rocket League).

The main issue is that these games are running out of address space and crashing with failed to map memory.

One thing I did notice, was that I could reproduce none of these issues (ie. Borderlands 2) on Windows with DXVK, but I could very easily when using Proton + DXVK.

One thing I did also try was disabling pulseaudio entirely which helped for a while but it still inevitably crashed (that thing was eating up way too much!)

Another thing I also tried was with WineSteam (wine-tkg) where I was not able to reproduce the issue with DXVK master, compared to Proton with DXVK master, but the address space (VMSize) was still scarily high! I'm not sure what I was looking at on Windows, but even the regular Memory in Task Manager + whatever Visual Studio's Debugger is telling me was much lower on Windows at ~2.3GB.

I disabled the game overlay (using the Steam Properties tab for the game) and also tried disabling Fossilize (bShaderEnableFossilizeLayer 0 in the console.) for Proton which didn't help.

Supercedes https://github.com/Joshua-Ashton/d9vk/issues/383 https://github.com/Joshua-Ashton/d9vk/issues/221 https://github.com/Joshua-Ashton/d9vk/issues/170 and https://github.com/Joshua-Ashton/d9vk/issues/93

[ericwomer]

Does this issues fall into this category? https://github.com/Joshua-Ashton/d9vk/issues/165

[Joshua-Ashton]

No idea because you deleted your gists.

Also the game uses both DXGI and D3D9 which may be an issue with nvapi hack thats default

[ericwomer]

Fixed.

[Joshua-Ashton]

I did some testing to compare address space usage between Windows and Wine/Proton and when we get into game. Wine/Proton is throwing away 1.3GB more address space than is used on Windows. No idea what/where though.

[ericwomer]

Could be a driver/library issue, as with UE4 Editor on Linux, running with Vulkan, opening and closing a bunch of windows relating to a skeletal mesh and animations, I don't know exactly where, will cause a crash relating to running out of memory.

[ericwomer]

[Here[(https://gist.github.com/ericwomer/f29e22d55467538a3adbddc837eb6fc1) is a log output for UE4 that I believe is related to the issue I was talking about.

[Joshua-Ashton]

That's unrelated: UE4 is 64-bit, we're talking about address space not you just running out of physical memory here.

[ericwomer]

It ran fine under OpenGL4, but with Vulkan it 'ran out of memory'. But I do have a 2 gb card which isn't much.

[jrugia]

I've had this issue with quite a few games even before D*VK and the first thing I do is patch the game to enable Large Address Aware. I've been using this tool for patching: https://github.com/randomstuff/pe-set-laa

I've had a quite a few successes with this method: Legend of Grimrock 2, Risen 1 + 2, Titan Quest, Race Driver Grid, Tomb Raider Anniversary/Legend/Underworld and Far Cry 2...just to name a few.

[VortexAcherontic]

@Joshua-Ashton and @jrugia

Hm I tried setting PROTON_FORCE_LARGE_ADDRESS_AWARE=1 %command% as an additional launch option in Proton to achieve the same as your linked git repo but Borderlands2 still crashes with the out of memory message with Textures set to High =/

How ever this dose not happen if I run BL2 with WineD3D instead.

Any further suggestions?

[kelvinhbo]

Not sure if this is related but, turning Ambient Occlusion on takes a massive performance hit on D9VK titles, this setting seems to be broken, because on Windows it doesn't have the same effect, games I've tested are Borderlands 2 and Rocket League on NVIDIA cards.

[pchome]

@Joshua-Ashton

No idea what/where though.

Try this on running Proton game (replace winecfg.exe w/ game name):

mem usage summary

cat /proc/$(pidof winecfg.exe)/smaps_rollup | less

mem usage per lib/exe/etc.

cat /proc/$(pidof winecfg.exe)/smaps \
| grep -A1 '/' | tr '\n' ' '| sed 's/ -- /\n/g' | sed -E 's/ +/ /g' \
| sort -k8,8gr -k6,6 | less

_Edit: swap -k8,8gr -k6,6 for sorting by name._

proc details

Maybe this will help to find suspect for detail analysis, IDK.

[wgpierce]

I did some more analysis on the interesting repro scenarios. I summarized information about where all the VA space goes with this script and imported it into libreoffice and did a pivot table on the two columns to sum up the total maps for each distinct map type and did a sort to get the top consumers:

pmap -q $(pgrep Launcher.e) | sed -e 's/\[ anon \]/[anon]/' | sed -e 's/\[ shmid=0x5800e \]/[shmid]/' | sed -e 's/\[ stack \]/[stack]/' | awk '{print $2"," $4}' | sed 's/K//' > map_sizes.csv

Here are the top consumers for some different scenarios in Borderlands 2 (a D3D9 game). Units are KB. These tests are on Nvidia 440 with the BL2 Ultra HD Texture Pack enabled running under Proton. Wine is at least 4.11 and DXVK is at least 1.5. Values didn't really change with newer or older wine/DXVK versions.

After starting up with DXVK enabled:

Total Result                  | 2949448
-
[anon]                        | 2104764
nvidiactl                     | 419636
memfd:pulseaudio              | 262144
libnvidia-glcore.so.440.44    | 28660
steamclient.so                | 26156
u1000-Shm_2019bf0e            | 16388
libnvidia-glvkspirv.so.440.44 | 11288

When the game crashes gives the "Ran out of video memory! Exiting..." error. Do note that the reference to video memory is a red herring -- it runs out of address space:

Total Result                  | 3904024
-
[anon]                        | 2206960
nvidiactl                     | 1263412
memfd:pulseaudio              | 262144
libnvidia-glcore.so.440.44    | 28660
steamclient.so                | 26156
u1000-Shm_2019bf0e            | 16388
libnvidia-glvkspirv.so.440.44 | 11288

After starting up with DXVK disabled, using WineD3D instead (and basically the steady state since it does not crash):

Total Result               | 2553480
-
[anon]                     | 1935996
memfd:pulseaudio           | 262144
#169462                    | 146484
nvidiactl                  | 53128
libnvidia-glcore.so.440.44 | 28660
steamclient.so             | 26156
u1000-Shm_eb9ca525         | 16388

A few interesting points:

• Using WineD3D, VA space usage usage is not significantly higher than Windows. In Windows, it typically sits at 2.3GB VA space usage.
• [anon] mappings total ~200MB more in the DXVK case, which is more but not as significant as nvidiactl
• nvidiactl is significantly higher using DXVK in grows enormously in the out of memory case. mmaping against /dev/nvidiactl corresponds to mapping system memory as you can see in the Nvidia open source code at /nvidia/nv-mmap.c, but I'm not 100% this is what is causing that to appear.

[pchome]

So, if the issue is not reproducible on Windows + DXVK, then it's likely Wine doing something unwanted. Either

• doing double job to allocate memory,
• UE3 limits are broken in Wine+DXVK (MemoryBudgets section, etc.),
• UE3 engine receives available memory from ddraw, d3d8, nvapi, etc. (which is the same function in Wine, AFAIK), which is different to the value reported by DXVK. (wined3d_device_get_available_texture_mem ?)

Maybe WINEDEBUG=+d3d tracing will show something.

[DaRkL3AD3R]

I don't know if this is the same issue, but I use DXVK on Windows and somewhere in the last 7 days something broke it for every game I test. They all crash on initialization with many of them referencing out of video memory or no hardware accelerator detected. Rolling back to the artifact for depth bias fix corrects the issue.

I haven't bisected builds yet to see which one caused this issue but I think it's related to this topic? I'll update if I find the exact commit that broke things for me.

It was this commit https://github.com/doitsujin/dxvk/commit/7c53a997effc64adbe52a8f67ee6bb6b5f2fe580

[dlshinobi]

I don't know if this is the same issue, but I use DXVK on Windows and somewhere in the last 7 days something broke it for every game I test. They all crash on initialization with many of them referencing out of video memory or no hardware accelerator detected. Rolling back to the artifact for depth bias fix corrects the issue.

I haven't bisected builds yet to see which one caused this issue but I think it's related to this topic? I'll update if I find the exact commit that broke things for me.

It was this commit 7c53a99

Yes, this is happening for me too. But I'm on Linux.

[Oliver-1xok]

Rocket League sometimes freezes in the lobby. The music continues until the song is finished. I get the following error message:

As soon as I click on okay, the game ends. Could it have something to do with this?

I'm not getting a dump and I don't see anything unusual. The game don't seems to use up all the video memory when I look at the memory usage with the "nvidia-smi" command. Everything looks normal.

I'll try to get more information. Does Proton write log files somewhere?

Apart from that, Rocket League runs perfectly with D9VK.

[wgpierce]

@Oliver-1xok yes that freeze is this issue. It is not a video memory issue as referenced earlier in this issue. We already have the logs we need.

@DaRkL3AD3R @dlshinobi I hope you filed another issue if this is still affecting you? Seems like a different problem from this issue.

[Joshua-Ashton]

Those issues by that commit should already be fixed in master now @wgpierce

[Animajosser]

I was told that my question was related to this. Walking Dead Season 1 crashes at launch when starting with vanilla wine and dxvk. I don't know enough about dxvk and wine to say anything about it, but installing a wine build with support for forcing LARGE_ADDRESS_AWARE (I used the 32 bit proton build here https://github.com/Kron4ek/Wine-Builds) and setting PROTON_FORCE_LARGE_ADDRESS_AWARE=1 or WINE_LARGE_ADDRESS_AWARE=1 will fix it. The LARGE_ADDRESS_AWARE patch is not implemented upstream yet as far as I can see, but it seems like that would be a fix for walking dead season 1 + dxvk compatibility with wine vanilla. If the other problems are also caused by this, it could be a fix for all of them.

By the way: I copy pasted part of this as an edit into my own closed issue (If someone else has the same problem they can find it in 2 places this way).

[wgpierce]

@Animajosser yes, LARGE_ADDRESS_AWARE is already being used. That can help 32-bit programs that use less than 4GB of address space with DXVK. This issue is all about programs that try to use >4GB of address space with DXVK because 32-bit programs cannot address >4GB of address space. When the program does try to map >4GB of memory, it fails. You can see this failure in this comment above

[rtentser]

I have a similar issue with Disco Elysium. The game doesn't crash, but after starting i'm getting no graphics except character's model and some objects. No background, just a blackness. And i'm getting some "out of memory" errors for my GT 710 with 1GB VRAM in terminal. I can reproduce it in 1.5.3.

But the game works fine with 1.4.6, so it's a regression.

[doitsujin]

Disco Elysium is a 64-bit game and therefore cannot run into this issue. The problem you're running into is #1100.

[wgpierce]

As was suggested in the first comment in this issue, PulseAudio can be disabled entirely to save the 200MB seen in my comment above.

I found a bit less severe mitigation that doesn't require disabling sound outright or doing any in-depth tweaking in kakra's old proton readme note on 32 bit programs (edited to fix the pulse path):

Pulseaudio users can somewhat improve the situation by limiting the SHM size of pulseaudio to 1 MB. Simply add shm-size-bytes=1048576 in /etc/pulse/daemon.conf and restart.

In my testing for Borderlands 2, this saved ~166MB of address space at launch when you reach the title screen. For Borderlands: The Pre-Sequel, this saves ~350MB of address space at launch.

Games afflicted by this issue will likely eventually still run out of address space, but this mitigation can help delay that. This mitigation is not a solution.

[mirh]

In my testing for Borderlands 2, this saved ~166MB of address space at launch when you reach the title screen.

So.. That would put memory usage on par with Windows, correct?

Did you also try to experiment with the shm-size/hugepages tips?
Otherwise the only other thing I could think is tinkering with chunks size and whatnot.

[wgpierce]

So.. That would put memory usage on par with Windows, correct?

No. It decreases the starting address space usage, but games are still afflicted by the issue at hand and will likely eventually still run out of address space.

For hugepages, that only looked to be a problem if your kernel is configured with CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y, but I think most distros only ship with CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y -- so only really a problem if you're toying around with hugepages, which most users aren't. So that didn't seem relevant, and I didn't try anything with it.

[mirh]

I meant, in the wined3d case.
If so, then, it really is just something about dxvk (or vulkan more generally?) and linux.
And perhaps it would be better to involve some driver developer.

[ComradeClaritux]

Just wanted to mention that anyone experiencing this issue with Rocket League can bypass it completely with the new experimental DX11 client (-dx11 at launch).
https://www.reddit.com/r/RocketLeague/comments/f2v9yb/directx_11_experimental_client/

[Joshua-Ashton]

The game is x64 now anyway so it isn't an issue on the dx9 client either.

[ComradeClaritux]

The dx9 client is still 32-bit, as it used to be, but the new client is x64

[Oliver-1xok]

The dx9 client is still 32-bit, as it used to be, but the new client is x64

I don't know if that has anything to do with this. But the error probably also occurred under Windows:
https://steamcommunity.com/app/252950/discussions/0/3441214221470525684/

I assume that the 32-bit client will be history in the near future. In March the Linux and MacOS client will be discontinued, then they will most likely switch to the 64-bit client and DX11.

[wgpierce]

Thanks mirh for pointing me to some of the other bugs. This kind of issue has also been seen in

Gallium9:
https://github.com/iXit/wine-nine-standalone/issues/60
https://github.com/iXit/wine-nine-standalone/issues/24

WINE (likely using WINED3D):
https://bugs.winehq.org/show_bug.cgi?id=44375
https://bugs.winehq.org/show_bug.cgi?id=45375#c10 (and below comments)

and in particular, see these interesting comments:
https://bugs.winehq.org/show_bug.cgi?id=44375#c8
https://bugs.winehq.org/show_bug.cgi?id=44375#c12

[wgpierce]

I've been investigating this for a while and have found a few more interesting points.
To ensure an apples-to-apples comparison, here is the VA space usage on different scenarios under these constraints:
Borderlands 2 with UHD texture pack, 1920x1080 resolution, and all settings maxed;
Nvidia HW and drivers on Linux with Wine and Windows using D3D9 and DXVK after loading into a level for the first time:

     | Windows               | Linux
-----|-----------------------|------------------
D3D  | 2,000 MB (native D3D) | 2,437 MB (WineD3D)
DXVK | 2,570 MB              | 3,694 MB

(Note that my previous comparison was at the menu screen.)

From this, we can see a few more things:

• WineD3D actually uses ~450MB more VA space than Windows D3D (the first comment in the issue was using and AMD (I was using Nvidia in my comparison) and DXVK, not native D3D).
• On Windows, DXVK uses ~600MB more than not using it (and using native D3D).
• On Linux, DXVK uses ~1,100MB more than not using it (and using WineD3D) (!).

I collected some more information in the exact condition described above on Linux and Wine with DXVK. In one run, I profiled the VA space usage at that time and matched it up with the DXVK vkAllocateMemory allocations which were then mapped with vkMapMemory.

These are the top users of VA space on the specific run (in KB) (same methodology as before):

Total Result                  | 3692320
--
[anon]                        | 2126728
nvidiactl                     | 1127932
memfd:pulseaudio              | 262144
libnvidia-glcore.so.440.59    | 28660
steamclient.so                | 26512
u1000-Shm_d4b1d418            | 16388
libnvidia-glvkspirv.so.440.59 | 11288

And using the attached patch, the allocations I see from DXVK in that run are in DXVK_memory_allocations.log. With some parsing and removing never-mapped allocations and freed allocations by memory handle, I isolated the unfreed allocations at that time in DXVK_unfreed_allocations.log. At that time, there were 33 32MB mapped allocations which had not been freed. 33*32MB = 1,107MB, nearly exactly what the VA space which was mmap'd against nvidiactl reports.

Attached files for clarity:
https://github.com/doitsujin/dxvk/files/4271496/0001-dxvk-memory-logging.txt
https://github.com/doitsujin/dxvk/files/4271497/DXVK_memory_allocations.log
https://github.com/doitsujin/dxvk/files/4271498/DXVK_unfreed_allocations.log

[pchome]

Possible sources of this allocations (just in case):

D3D9CommonTexture::CreateBufferSubresource

https://github.com/doitsujin/dxvk/blob/1e426cdb085f5f077753d80d13d8a97c9fbe3894/src/d3d9/d3d9_common_texture.cpp#L142-L170

D3D9CommonBuffer::CreateStagingBuffer

https://github.com/doitsujin/dxvk/blob/1e426cdb085f5f077753d80d13d8a97c9fbe3894/src/d3d9/d3d9_common_buffer.cpp#L102-L122

DxvkMemoryAllocator::pickChunkSize

https://github.com/doitsujin/dxvk/blob/1e426cdb085f5f077753d80d13d8a97c9fbe3894/src/dxvk/dxvk_memory.cpp#L391-L411

[wgpierce]

Turning on the DXVK config option d3d9.evictManagedOnUnlock = True solves the issue in BL2 at least with no side effects. Total VA space usage in that game is capped to ~3GB, even at 1440p.
This option was fixed with commit d1dc217d, which made it into release 1.5.5.
Beware that using this config option may break games which depend on being able to write to memory after its unlocked.

[CSahajdacny]

I am happy that, at least, somebody is working for to fix this issue, instead of just blaming the game.
I want to try this with Borderlands the pre-sequel with UHD pack installed.

What I have to do for to enable d3d9.evictManagedOnUnlock = True on that game?

[wgpierce]

What I have to do for to enable d3d9.evictManagedOnUnlock = True on that game?

Use dxvk.conf, read the README.

[CSahajdacny]

What I have to do for to enable d3d9.evictManagedOnUnlock = True on that game?

Use dxvk.conf, read the README.

Thanks!
I report that I tested this d3d9.evictManagedOnUnlock = True dxvk.conf method with Borderlands: The Pre-Sequel with the UHD pack installled and now it doesn't crash anymore, but the framerate is not solid as it used to be on my RTX 2060 SUPER & i7 4790K
Here is the log:
steam-261640.log

[walmartshopper]

Using d3d9.evictManagedOnUnlock = True with Rocket League causes missing text which makes it unplayable.

[philipl]

Using d3d9.evictManagedOnUnlock = True with Rocket League causes missing text which makes it unplayable.

Same for borderlands 2. It fixes the crash but the text is almost all missing.

[K0bin]

Is that still the case with 1.5.5?

[philipl]

Is that still the case with 1.5.5?

It is not. I thought I was running with 1.5.5 because I copied 1.5.5 into my prefix. But it turns out that steam actually replaces the dlls from the original distribution files every time you run the program. So I replaced the distribution files and then it worked without missing text (which I guess is what the fix in 1.5.5 does).

So now it looks correct and doesn't crash. Nice.

[walmartshopper]

Can confirm. Updated to proton 5.0-4 which includes 1.5.5 and the rocket league DX9 client works with d3d9.evictManagedOnUnlock = True. However, with the rocket league 1.74 update and proton 5.0-4, the 64-bit DX11 client is now working, so I'm going to switch to that unless I start having problems.

[js6i]

I was looking into TOXIKK and it's suffering from the same issue too (https://github.com/ValveSoftware/Proton/issues/2011). What I noticed is that each time it called DxvkContext::clearCompressedColorImage it would allocate 32 megs of new device memory for the staging buffer and never free it later. Is this expected behavior?
I "fixed" this problem by applying the following at the end of this function:

diff --git a/src/dxvk/dxvk_context.cpp b/src/dxvk/dxvk_context.cpp
index 86829a4a..802b55e5 100644
--- a/src/dxvk/dxvk_context.cpp
+++ b/src/dxvk/dxvk_context.cpp
@@ -562,7 +562,7 @@ namespace dxvk {
       image->info().access);

     m_cmd->trackResource<DxvkAccess::Write>(image);
-    m_cmd->trackResource<DxvkAccess::Read>(stagingSlice.buffer());
+    m_cmd->freeBufferSlice(stagingSlice.buffer(), stagingSlice.getSliceHandle());
   }

It seems to me like one should not need more staging buffers but I'm new to this so it might just be a bug waiting to happen. I was able to run through the tutorial like that though and noticed no issues and had only 1 staging buffer allocated.
I also tried in on a completely different title (Deep Rock Galactic) and it went pretty badly - surely that's not the way as such, but maybe not a dead end? (to be precise - it rendered OK with just this change, but went south when I changed other functions that allocate staging buffers in the same way)
So the general idea would be to reuse staging buffers more if possible - I'm not sure if execution order lets you to reuse all of them, but the read-only ones, especially if just filled with zeros, should be fine to reuse, right?

[iWeaker4you]

I don't think everything is related to DXVK, for example Final Fantasy X / X-2, especially X-2 is constantly crashing with Wine version 5.0 / Proton5.0+, it is unplayable, also detect that PROTON_FORCE_LARGE_ADDRESS_AWARE has problems and is not functional in this title, so much more crashed when activated, i try version 4.11.13 and use this only PROTON_USE_D9VK = 1 PROTON_USE_WINED3D = 1% command% and it works great, it has graphic problems in the save menus but nothing serious, I could get to the ship and save the game, something that I can't do in Proton 5.0, because it crashes as soon as the first fight starts, even if I set those parameters.

[adamdmoss]

Want to draw attention to https://github.com/doitsujin/dxvk/issues/1318#issuecomment-593503624 which is getting lost in the thread.

Turning on the DXVK config option d3d9.evictManagedOnUnlock = True solves the issue in BL2

(subsequent comment confirms that it also fixes BLTPS+HD and keeps virtual address space usage much lower.)

[betam4x]

The situation has gotten a lot better (nearly flawless) with Final Fantasy X, however, it still locks up at certain points. I can provide a save file if needed.

EDIT: The first battle you are able to use a summon is where it freezes...a couple hours in. I can also provide a game license (to the developers ;) ) if needed.

[mozo78]

-------- Оригинално писмо --------

От: "Adam D. Moss" [email protected]

Относно: Re: [doitsujin/dxvk] Out of Address Space Megaissue (#1318)

До: doitsujin/dxvk

Изпратено на: 15.05.2020 01:52

Want to draw attention to #1318 (comment) which is getting lost in the thread.

Turning on the DXVK config option d3d9.evictManagedOnUnlock = True solves the issue in BL2

(subsequent comment confirms that it also fixes BLTPS+HD and keeps virtual address space usage much lower .)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub , or unsubscribe .

[WinterSnowfall]

Someone might have asked this before, but is DXVK doing any memory defragmentation at the moment?

Perhaps something like a d3d9.lowPriorityDefragmentManagedOnUnlock logic might help more games without incurring a serious performance hit, which tends to happen sometimes when d3d9.evictManagedOnUnlock = True is used.

I've only read a bit about how Vulkan memory allocation works, so I could be horribly wrong.

[doitsujin]

The idea isn't really new, but implementing something like that is far from trivial and no one has tried yet.

Ideally that would be something we could enable by default, in order to improve the situation for more games, but as of right now we don't have an awful lot of data to work with. What are known games that work only with evictManagedOnUnlock?

[WinterSnowfall]

The idea isn't really new, but implementing something like that is far from trivial and no one has tried yet.

Can this provide any inspiration perhaps? They advertise having a "Defragmentation system".

What are known games that work only with evictManagedOnUnlock?

I'm hoping others can answer this. Personally, if the LAA approach doesn't solve it, I just give up :). I don't own exceptionally stellar hardware, so evictManagedOnUnlock is something I avoid.

[doitsujin]

Can this provide any inspiration perhaps? They advertise having a "Defragmentation system".

You can't just plug it in and get it for free, we actually have to recreate all associated images, buffers and views while dealing with potential thread synchronization issues and making sure that the GPU isn't accessing any of the objects that we are deleting. That's the hard part.

It also doesn't really solve any of our actual problems; fragmentation isn't bad in practice, we just have lots and lots of persistently mapped memory in D3D9, which is hard to get rid of due to the way D3D9 works.

[betam4x]

-------- Оригинално писмо -------- От: "Adam D. Moss" [email protected] Относно: Re: [doitsujin/dxvk] Out of Address Space Megaissue (#1318) До: doitsujin/dxvk Изпратено на: 15.05.2020 01:52 Want to draw attention to #1318 (comment) which is getting lost in the thread. Turning on the DXVK config option d3d9.evictManagedOnUnlock = True solves the issue in BL2 (subsequent comment confirms that it also fixes BLTPS+HD and keeps virtual address space usage much lower .) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub , or unsubscribe .

This has no effect for Final Fantasy X.

[aufkrawall]

I'd say that with Proton 5.0-9, latest DXVK master and Nvidia 450.51, the situation in Borderlands 2 has gotten much better than it used to be. I installed HD textures, set visual fidelity to maximum 1440p, fast travelled to several locations in a row and started some fights without crashing. In the past it used to sometimes crash even after the first fast travel. That was with RADV though. Anybody might want to give it a try with recent Proton / DXVK?

[jl452]

"tree of savior, itos version, steam" 32 bit + Proton-5.9-GE-2-MF (prerelease? download 7 day before) + dxvk1.7
crush 100% with 1-3 teleport in any town with out of memory

PROTON_FORCE_LARGE_ADDRESS_AWARE=1 not help
WINE_LARGE_ADDRESS_AWARE=1 + PROTON_FORCE_LARGE_ADDRESS_AWARE=1 not help (small test)
WINE_LARGE_ADDRESS_AWARE=1 + PROTON_FORCE_LARGE_ADDRESS_AWARE=1+ d3d9.evictManagedOnUnlock = True + shm-size-bytes=1048576 in /etc/pulse/daemon.conf (but i need restart after change pulse config? i not do restart yet) == NO CRUSH

DXVK_HUD=memory show
before:
Vidmem heap 0: 300-1000 MB (15-xx%) (not remember)
Sysmem heap 1: 85-100 MB (2%) (not remember)
Vidmem heap 2: 290-310 MB (90-110%) (remember!)
after:
Vidmem heap 0: 704 MB (39%)
Sysmem heap 1: 84 MB (2%)
Vidmem heap 2: 144 MB (56%)

not see any different in graphics

thanks!

[jl452]

not sure, maybe "Vidmem heap 2" with 3d9.evictManagedOnUnlock = True dependent on fullscreen (~300 MB and crush) / virtual wine desktop (144 MB)

Proton-5.11-GE-1-MF

no, its randomly (50%/50%) use 300 MB or only 150 MB in same config

[kenan1099]

Does this issue affect dx11 titles as well?
I tried dark souls 3 in lutris (wine steam) and I'm getting the same error in the logs.
Note: I am using an intel hd 4600, which doesn't have complete vulkan support, however im only getting a memory issue.

[K0bin]

It does affect D3D11 titles but Dark Souls 3 is a 64 bit binary.

[kenan1099]

It does affect D3D11 titles but Dark Souls 3 is a 64 bit binary.

Oh I see, tho weirdly when I run out of VRAM (stolen mem), Darksouls just crashes. Heck, if I play at higher resolutions, the game crashes instantly without even loading the map.

[kenan1099]

Ok I tried skyrim special edition with d3d9.evictManagedOnUnlock = True and WINE_LARGE_ADDRESS_AWARE=1 but memory usage climbs to 1376 MB (89%) and either stays on the loading screen while spamming me with out of memory messages or crashes right after the "you're finally awake" scene.
I used dxvk 1.7 and dxvk master c4f37d39 from self compilation or pipeline, all had the same error.
NOTE: on intel hd graphics the maximum allowed stolen mem is just 1.5 GB.
Oh and just in case it was vulkan, I tried dxvk 1.5.1 (since vulkan 1.0 was fully supported) and I got the same exact error.

[ovvldc]

Ok I tried skyrim special edition

Original Skyrim is 32bit but Special Edition is 64bit.

[ishitatsuyuki]

Just FYI, I was debugging D9VK for a game running on Windows (not Wine) using RenderDoc, and RenderDoc crashed the game every time I tried to capture due to the driver returning VK_ERROR_DEVICE_OUT_OF_MEMORY. I patched the game executable afterwards to make it LAA and now capturing works without issues.

EDIT: apparently it still crashes if you do too much things in RenderDoc, but patching with LAA indeed works as described above.

[K0bin]

Proton on Linux automatically treats 32bit games as if the LAA flag was set. Unfortunately plenty of games still run out of the 3GB of address space that gives you.

[alasky17]

Ideally that would be something we could enable by default, in order to improve the situation for more games, but as of right now we don't have an awful lot of data to work with. What are known games that work only with evictManagedOnUnlock?

TOXIKK (324810) is one - it has random crashes during gameplay that are resolved by adding this to dxvk.conf. I'm not sure if it is still the plan to enable by default at some point? It would be nice to at least set it by default for games known to need it.

[aufkrawall]

evictManagedOnUnlock makes Borderlands 2 very stuttery while moving.

[ivyl]

ROBOTICS;NOTES DaSH (Steam ID: 1111390) is a 32bit DirectX 11 game that also runs out of address space with DXVK. It crashes for me most of the time just after all the splash screens as the menu is about to be displayed. Sometimes it makes it to the menu though, but crashes soon after starting a new game.

With wined3d (requires staging patches for deferred context support) the game sits just below 3GB and plays fine.

I've tried disabling shm for PulseAudio - game launches with about 3.8GB of the address space used but still crashes a few minutes in the game during the scene featuring ~three characters.

[Rodancoci]

I encountered this exact issue trying to play Mass Effect 1 (a 32-bit game) with high-res texture mods (ALOT & MEUITM).

I had almost given up playing it on Linux but evictManagedOnUnlock fixed the crash I experienced on the intro and so far the game has been very stable (knock on wood) with no noticeable performance hit.

[rsw0x]

With regards to pulseaudio's memory usage, it does seem to be an issue with 32bit games on wine in general: https://www.winehq.org/pipermail/wine-devel/2018-November/134897.html

Also,

shm-size-bytes=1048576 in /etc/pulse/daemon.conf

You need to put it in your client.conf too. By default, it will reserve a very large portion of the address space -- 64MiB for server + 64MiB for client on my machine. Seems extreme for 32bit applications.

[rakantoh]

evictManagedOnUnlock makes Borderlands 2 very stuttery while moving.

Yeah, and i think that at this moment it no have fix. Hopefully it fix soon.

[K0bin]

The fix would be to do what DXVK does with that option disabled.

[rakantoh]

The fix would be to do what DXVK does with that option disabled.

but with the option disabled the game crashes ramdomly

[rakantoh]

the dxvk problem that i reported apparently is only in borderlands 2, in the pre sequel dxvk works normally.

[K0bin]

It's a known issue and can't be fixed. Nothing else to say about it.

[rakantoh]

I write this comment for two reasons.

1. Apparently the problem in borderlands 2 is only in the remaster version (handsome collection), i played borderlands 2 (normal version) with latest dxvk (1.7.3) for a entire week, and i played more than 3 hours daily without close unexpectly, only a two or three times.
2. Unfortunately i report another game with the "out of memory" problem with dxvk: F.E.A.R. 1.

[chumik6000]

Well the main problem is when ReduceSystemMemoryUsage set to true, my skyrim have terrible stutters. With your patch it has no stutter, but game run out of memory. Also i found that if i delete my textures folder, i can load my save (with perple/missing textures), so as i understand dxvk patch have problem with vram memory allocation. But if i use enb memory patch game use memory good, but game have unplayeble stutters. Is there is a way to use enb mem patch and avoid dxvk stutters? TY!

[Squall-Leonhart]

Proton on Linux automatically treats 32bit games as if the LAA flag was set. Unfortunately plenty of games still run out of the 3GB of address space that gives you.

LAA is only 3GB's on a 32bit operating system where the kernel maintains the last GB for itself, if proton is only affording 3GB to processes, it is doing it wrong. LAA patched applications can use 4GB's on 64bit operating systems and so proton should provide the same.

It's a known issue and can't be fixed. Nothing else to say about it.

There is no such thing as can't be fixed, there is just idea's that haven't been had yet.

Skyrim classic resolved the issue via ENB pipelining into a 64bit helper process

If such a method doesn't exist yet, be a pioneer and make it yourself, there are no can'ts, only won'ts.

[SveSop]

Proton on Linux automatically treats 32bit games as if the LAA flag was set. Unfortunately plenty of games still run out of the 3GB of address space that gives you.

LAA is only 3GB's on a 32bit operating system where the kernel maintains the last GB for itself, if proton is only affording 3GB to processes, it is doing it wrong. LAA patched applications can use 4GB's on 64bit operating systems and so proton should provide the same.

It's a known issue and can't be fixed. Nothing else to say about it.

There is no such thing as can't be fixed, there is just idea's that haven't been had yet.

Skyrim classic resolved the issue via ENB pipelining into a 64bit helper process

If such a method doesn't exist yet, be a pioneer and make it yourself, there are no can'ts, only won'ts.

Well, I am not sure it is that simple in reality.
I tested a little appy i found that fills memory. https://download.cnet.com/Memory-Filler/3000-2094_4-75744702.html

There are probably other examples out there, but i just tested this. On my Win10 box (64-bit w/8GB ram), i managed to fill something like 3408MB before the program complained about it could not fill more. I ran the same app in wine (compiled with LAA patch), and i was able to fill 3080MB.
So, why would i not be able to fill 4096MB? (Atleast on my real win10 box).
I was easily able to start 2 instances of the app both filling 3408MB of ram (although it seemed as some swapping started to happen on the last leg there).

Surely it WOULD help with 400'ish more MB for wine and this problem, so maybe it can be fixed a bit? Is there a "hard limit" of 3GB (3072MB) in play for wine, that possibly could be upped a bit?

However, the difference is for wine that it actually is running a "32-bit os", and is probably not as true sysWOW64 as Win10 would be. (wine vs wine64 when running).

[Squall-Leonhart]

Well, I am not sure it is that simple in reality.

LAA specifically refers to Addressing, you will never see the Physical working set or private set reach this high as these are reported minus fragmentations affect on the address space, in the context of 3 vs 4GB, 3.5-3.6 Working Set are achievable and i've seen upwards of 3.8 physical in a particular emulation case loading a large texture pack. (and this is with windows usual bottom up allocation scheme, top down is more efficient/controllable)

Under windows (and particuarly process explorer) this is demonstrated in the Virtual Size field.

However, the difference is for wine that it actually is running a "32-bit os", and is probably not as true sysWOW64 as Win10 would be. (wine vs wine64 when running).

Wine is not an OS, its a Windows on Linux system implementing the software apis and runtime and doesn't have to have any of the limitations that the 32bit native windows operating system suffers from having to run the driver stack and kernel. These are arbitrary restrictions that can be removed. You are not running a kernel driver stack in that unavailable 1GB so it can be made directly and entirely available to the application in the same way WoW64 does.

[SveSop]

...Hence my usage of " around the OS part :)

I know its not a emulator or a virtual os, but the way wine is atm, you have wine (the 32-bit "launcher" for 32-bit apps), and wine64 (64-bit "launcher").
wine: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked
wine64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked

So, when the 32-bit version of "wine" loads all its system libraries, and various other stuff to launch that 32-bit program... how much is left? Is that the possible 3.5-3.6GB you mean? Cos the way wine is atm, i think you are somewhat limited as the 32-bit part of wine is both built and runs as a 32-bit "loader" that loads whatever systemlib's needed within that 4GB limit.

I think this is somewhat out of the scope of DXVK tho :)

[jl452]

I tested a little appy i found that fills memory. https://download.cnet.com/Memory-Filler/3000-2094_4-75744702.html
There are probably other examples out there, but i just tested this. On my Win10 box (64-bit w/8GB ram), i managed to fill something like 3408MB before the program complained about it could not fill more. I ran the same app in wine (compiled with LAA patch), and i was able to fill 3080MB.
So, why would i not be able to fill 4096MB? (Atleast on my real win10 box).

it's real interesting
on wine-6.2 (Staging) i cant run "fill" - it say "Invalid floating point operation"
but it work on Proton-6.1-GE-2
1) 3015 without WINE_LARGE_ADDRESS_AWARE=1 PROTON_FORCE_LARGE_ADDRESS_AWARE=1
2) 3071 with WINE_LARGE_ADDRESS_AWARE=1 PROTON_FORCE_LARGE_ADDRESS_AWARE=1
i go from 4096 to 2048 by /2 range
3) but when i restart memfill and try 3075 - it work, i try 3250 - it failed, i try 3075 second - it failed

wineserver: using server-side synchronization.
wine: RLIMIT_NICE is <= 20, unable to use setpriority safely
wine: Unable to use RTKit to control niceness, rtkit-daemon not found or MinNiceLevel >= 0
002c:fixme:winediag:LdrInitializeThunk wine-staging 6.1 is a testing version containing experimental patches.
002c:fixme:winediag:LdrInitializeThunk Please mention your exact version when filing bug reports on winehq.org.
WARNING: radv is not a conformant vulkan implementation, testing use only.
WARNING: radv is not a conformant vulkan implementation, testing use only.
0024:fixme:ntdll:NtQuerySystemInformation info_class SYSTEM_PERFORMANCE_INFORMATION

here "try 3075 second"

0024:err:virtual:try_map_free_area mmap() error Cannot allocate memory, range 0xea030000-0x2040000, unix_prot 0x3.
0024:err:virtual:try_map_free_area mmap() error Cannot allocate memory, range 0xea030000-0x2040000, unix_prot 0x3.
0024:err:virtual:try_map_free_area mmap() error Cannot allocate memory, range 0xea040000-0x2041000, unix_prot 0x3.
0024:err:virtual:try_map_free_area mmap() error Cannot allocate memory, range 0xea030000-0x2031000, unix_prot 0x3.

WINEPREFIX=/.../wine_prefix/memfill.test.proton/ WINE_LARGE_ADDRESS_AWARE=1 PROTON_FORCE_LARGE_ADDRESS_AWARE=1 /.../Proton-6.1-GE-2/dist/bin/wine /.../memfill.exe

free -m
              total        used        free      shared  buff/cache   available
Mem:          15194        4236        3388         465        7569       10172
Swap:             0           0           0

[SveSop]

Untick the "MovNTQ" box, and it writes on regular wine aswell. (Have no clue why it would work with Proton and not wine tho).

It is kinda weird, cos once you get the message "Can not fill so much memory", and you go back to a previous working amount, it sometimes fails until you go even lower, and then you sometimes can go up again :) I think i was up to 3088 a bit there.

Since the app is able to fill more than 2GB it is LAA aware afaik, so switches should really not matter.
I think the WINE_LARGE_ADDRESS_AWARE=1 is only meant to "force non-LAA apps to be LAA". The other way (if using windows), is to patch the binary (there are proggys for that), or recompile with LAA option enabled... and since this particular app is able to fill > 2GB, it is compiled as LAA. (Should be no difference in result when using the wine flag).

[qarmin]

I'm not sure if this is related and if can help, but Mesa already got method to fix(some) issues with out of memory crashes in Gallium Nine

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9377
https://www.phoronix.com/scan.php?page=news_item&px=Mesa-21.1-Better-D3D9-32-bit

[SveSop]

Hmm.. That memfd thing sounded interesting.

[WinterSnowfall]

Pulseaudio users can somewhat improve the situation by limiting the SHM size of pulseaudio to 1 MB. Simply add shm-size-bytes=1048576 in /etc/pulse/daemon.conf and restart.

In my testing for Borderlands 2, this saved ~166MB of address space at launch when you reach the title screen. For Borderlands: The Pre-Sequel, this saves ~350MB of address space at launch.

Games afflicted by this issue will likely eventually still run out of address space, but this mitigation can help delay that. This mitigation is not a solution.

I know it ultimately does not solve the problem, and that Joshua also mentioned it up top, but I feel this should be "stickied" or documented as a known factor that may alleviate it in various degrees. I don't think a lot of people (especially not those reporting bugs linked to this megaissue) are aware how much PulseAudio can hog out of the available address space without the above tinkering (or getting rid of it completely).

I was frequently getting crashes after 1-2 hours of playing X3:Albion Prelude, whereas after limiting the shm size of the PA server & clients, as per above, I've had 6+ hour sessions without issues.

For some games it just might be enough to free up the space they'd typically expect to have available in 32bit windows. It should definitely be something people are made aware of and should try out if they run into these sort of things outside of games which are known to eventually crash anyway.

[philipl]

The memfd based paging technique from Gallium Nine would solve this problem, because it's fundamentally the same problem - it allows unlocked allocations to be removed from the address space without completely evicting them. How readily it could be brought over is a separate question. The internal implementations are completely different.

[Joshua-Ashton]

The equivalent we would have would be just calling vkUnmapMemory.

[NerosTie]

I experience random crashes with Final Fantasy X-2 (only X-2). The only thing I can provide is my log:

steam-359870.tar.gz

It looks like it's an issue with DXVK because 2 people say it doesn't crash with WineD3D (I'll try later, but it will take time to be sure):

https://github.com/ValveSoftware/Proton/issues/555#issuecomment-803299030
https://www.protondb.com/app/359870#fDRoVlZVkw

(not sure if PROTON_FORCE_LARGE_ADDRESS_AWARE=0 is relevant)

I would like to create a new issue and provide a trace, but the crashes are so random I can't reproduce it in a short time (=the trace will be very heavy). But I suspect it could be related to this megaissue, can you tell with only the log file?

Tested with:

DXVK 1.7.3
Proton experimental-5.13-20210311
GPU: AMD RX 580 (Mesa 21.0.1)

EDIT: ok, I had a crash with wined3d too in the same situation I had with DXVK (when a combat starts), so now I'm not sure at all...