Rpcs3: Out of video memory when switching between tabs

[Psycho-A]

When switching to the browser and other tabs in the Windows Explorer on running RPCS3, after third or fourth switching to emulator tab, video memory abruptly overflows to the end, and emulator crashes. This happens on any game.

OS Win 7 x64, NVidia GTX 760 Ti 2 GB, last emulator versions, Vulkan renderer.
Didn't notice this on older revisions.

[AniLeo]

Needinfo, missing GPU

[Psycho-A]

NVidia GTX 760 Ti.

[kd-11]

Is the game window minimized when this happens? It would be awful if the driver refuses to allocate swapchains just because another window is infront of it so I think it has to have minimized somehow, probably coming from fullscreen mode using alt-tab.

[Psycho-A]

@kd-11, it happens even when we're not switching from fullscreen to window or reverse, and even when fullscreen never been. Just switching to another window using explorer's taskbar with next calling rpcs3 window from here too, may provoke this issue. I guess it minimized when we're whitch this off by selecting another program's window.

[kd-11]

ok, cool. Check if its fixed here https://ci.appveyor.com/project/kd-11/rpcs3/build/0.0.3-1129. If it causes vulkan to freeze permanently let me know. Disables rendering if swapchain creation is denied (assuming its caused by minimizing) and restores rendering if it detects were coming from a minimized state. Of course if the trigger is not because of minimizing it wont work properly, but that can be worked around

[annadess]

True, I can confirm that with this fix minimizing won't cause the emulator to crash now. And only a single "Swapchain creation failed because dimensions cannot fit." message is fired.

Hardware details again if needed:
Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz
nVidia GTX 1060 3GB
Windows10
Renderer: Vulkan

Screenshot showing the last picture Vulkan rendered before the emulator was minimized:

Log:
RPCS3.zip

[Psycho-A]

@kd-11, this build seems to work well for me, although I'm getting this error in log:

E {rsx::thread} RSX: Swapchain: Swapchain creation failed because dimensions cannot fit. Max = 0, 0, Requested = 1280, 720 x8

[annadess]

@Psycho-A that was there before the fix too though.

[Psycho-A]

Error repeated again, on v0.0.4-6075. First minimizing GS window in Vulkan leads this:

E {rsx::thread} RSX: Swapchain: Swapchain creation failed because dimensions cannot fit. Max = 0, 0, Requested = 1280, 720 x21
F {rsx::thread} class std::runtime_error thrown: Assertion Failed! Vulkan API call failed with unrecoverable error: Out of video memory (VRAM) (VK_ERROR_OUT_OF_DEVICE_MEMORY)
(in file c:\projects\rpcs3\rpcs3\emu\rsx\vk\VKHelpers.h:286)

[kd-11]

The patch has not yet been merged into master

[kd-11]

This has been fixed in master.