Wgpu: Potential memory leak

[YuuriMomo]

Description
After moving to wgpu, few people testing the branch started to complain about the game getting slower and slower with the time. We started to dig in into the problem and we found out that

_ZN15veloren_voxygen6render8renderer8Renderer23create_consts_ui_locals17h3c8b311cd8cd64b0E which goes to _ZN4wgpu6Device17create_bind_group17hd8df2faa8758b09eE
looked like it was leaking something internally in wgpu when using vulkan backend.

AFAIK it doesn't happen on master which still uses old backend.

Repro steps
https://gitlab.com/veloren/veloren/-/tree/capucho/wgpu

Extra materials

Platform
Information about your OS, version of wgpu, your tech stack, etc.
Windows 10
WGPU 0.5.4
vulkan backend

[kvark]

It looks like the descriptors aren't getting freed properly. Thank you for filing! I'll get this fixed ASAP.

[kvark]

Unable to reproduce on Metal. I modified the cube example to create a bind group each frame. No memory growth observed.

Edit: actually, how much do you see it leaking? I'm seeing the growth of about 10mb/minute.

[YuuriMomo]

for me it was about 1mb/s in veloren main menu.

[kvark]

@YuuriMomo I modified the cube example in https://github.com/kvark/wgpu-rs/tree/cube-mt-bind-group to recreate the bind group each frame, to simulate extreme conditions of what you might be doing in Veloren. Running it on Vulkan through Vangrind doesn't reveal anything lost on a per-frame basis, other than these warnings:

kvark@ant /hub/3d/wgpu-rs $ valgrind --leak-check=full target/debug/examples/cube
==49791== Memcheck, a memory error detector
==49791== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==49791== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==49791== Command: target/debug/examples/cube
==49791== 
==49791== Thread 2:
==49791== Conditional jump or move depends on uninitialised value(s)
==49791==    at 0x1399484A: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x139EB7F2: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x139F2044: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x139F2137: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x1395022C: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x8631F2: ash::vk::DeviceFnV1_0::cmd_begin_render_pass (vk.rs:6181)
==49791==    by 0x5D212D: ash::device::DeviceV1_0::cmd_begin_render_pass (device.rs:1254)
==49791==    by 0x504D6D: <gfx_backend_vulkan::command::CommandBuffer as gfx_hal::command::CommandBuffer<gfx_backend_vulkan::Backend>>::begin_render_pass (command.rs:279)
==49791==    by 0x68F412: wgpu_core::command::render::<impl wgpu_core::hub::Global<G>>::command_encoder_run_render_pass (render.rs:795)
==49791==    by 0x711D2F: wgpu::backend::direct::<impl wgpu::Context for wgpu_core::hub::Global<wgpu_core::hub::IdentityManagerFactory>>::encoder_end_render_pass (direct.rs:871)
==49791==    by 0x6B3227: <wgpu::RenderPass as core::ops::drop::Drop>::drop (lib.rs:1499)
==49791==    by 0x22DE4E: core::ptr::drop_in_place (mod.rs:174)
==49791== 
==49791== Conditional jump or move depends on uninitialised value(s)
==49791==    at 0x1399484A: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x139EB7F2: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x139F2044: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x139F221E: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x1394FD81: ??? (in /usr/lib64/libnvidia-glcore.so.440.82)
==49791==    by 0x863230: ash::vk::DeviceFnV1_0::cmd_end_render_pass (vk.rs:6193)
==49791==    by 0x826B0B: ash::device::DeviceV1_0::cmd_end_render_pass (device.rs:1319)
==49791==    by 0x7FE737: <gfx_backend_vulkan::command::CommandBuffer as gfx_hal::command::CommandBuffer<gfx_backend_vulkan::Backend>>::end_render_pass (command.rs:290)
==49791==    by 0x69650A: wgpu_core::command::render::<impl wgpu_core::hub::Global<G>>::command_encoder_run_render_pass (render.rs:1258)
==49791==    by 0x711D2F: wgpu::backend::direct::<impl wgpu::Context for wgpu_core::hub::Global<wgpu_core::hub::IdentityManagerFactory>>::encoder_end_render_pass (direct.rs:871)
==49791==    by 0x6B3227: <wgpu::RenderPass as core::ops::drop::Drop>::drop (lib.rs:1499)
==49791==    by 0x22DE4E: core::ptr::drop_in_place (mod.rs:174)
==49791== 

In total, it reports 164Kb lost over the course of several minutes, which are all attributed to ash::vkCreateInstance, so these are one-time initialization losses and not really a leak.

Could it be that you are seeing a leak on your side? I.e. you are recreating something every frame but not letting it drop, storing it in some persistent map forever.

[YuuriMomo]

I will check it out with the people who implemented WGPU.

[YuuriMomo]

@kvark I have memory leak in cube example.

50mb/s

[treeco123]

I'm seeing the same memory leak in the cube example as Yuri. Specs are Windows 10, Radeon R7 370 GPU.

[kvark]

@treeco123 @YuuriMomo could you clarify if

1. you both are running on Vulkan
2. you are running the master cube example, or my test branch

[YuuriMomo]

Test branch, and i'm 100% sure i'm using vulkan backend @kvark

[YuuriMomo]

[2020-05-19T20:24:16Z INFO wgpu_core::instance] Adapter Vulkan AdapterInfo { name: "GeForce GTX 1050 Ti", vendor: 4318, device: 7308, device_type: DiscreteGpu } @kvark

[Imberflur]

I originally did not see the leak on the veloren wgpu branch, but with this cube test branch mem use goes up about 20mb/s.
On linux

AdapterInfo { name: "Intel(R) UHD Graphics 620 (WHL GT2)", vendor: 32902, device: 16032, device_type: IntegratedGpu, backend: Vulkan }

After a several seconds fails with this output

thread '<unnamed>' panicked at 'Error on command buffer allocation: ERROR_OUT_OF_HOST_MEMORY', cargo/registry/src/github.com-1ecc6299db9ec823/gfx-backend-vulkan-0.5.2/src/pool.rs:44:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Any', examples/cube/main.rs:323:9
[2020-05-19T20:16:20Z ERROR gfx_memory::heaps] Heaps still have 1 types live on drop
[2020-05-19T20:16:20Z ERROR gfx_memory::allocator::general] Memory leak: SizeEntry(768) is still used
[2020-05-19T20:16:20Z ERROR gfx_memory::allocator::general] Memory leak: SizeEntry(4096) is still used
[2020-05-19T20:16:20Z ERROR gfx_memory::allocator::general] Memory leak: SizeEntry(32768) is still used
[2020-05-19T20:16:20Z ERROR gfx_memory::allocator::general] Memory leak: SizeEntry(262144) is still used
[2020-05-19T20:16:20Z ERROR gfx_memory::allocator::general] Memory leak: SizeEntry(256) is still used
[2020-05-19T20:16:20Z ERROR gfx_memory::allocator::general] Memory leak: SizeEntry(2048) is still used
[2020-05-19T20:16:20Z ERROR gfx_memory::allocator::general] Memory leak: SizeEntry(16384) is still used
[2020-05-19T20:16:20Z ERROR gfx_memory::allocator::general] Memory leak: SizeEntry(131072) is still used
[2020-05-19T20:16:20Z ERROR gfx_memory::allocator::general] Memory leak: SizeEntry(270336) is still used
[2020-05-19T20:16:20Z ERROR gfx_descriptor::allocator] DescriptorAllocator is dropped

[treeco123]

I was also running on the test branch, with Vulkan backend.
The leak in Veloren's wgpu branch seems to act differently for me and for Yuri, so that's weird.

[Imberflur]

Valgrind does not seem to report anything being leaked afaict

  valgrind --leak-check=full target/debug/examples/cube
==380774== Memcheck, a memory error detector
==380774== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==380774== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==380774== Command: target/debug/examples/cube
==380774== 
==380774== Invalid read of size 8
==380774==    at 0x401EF18: strncmp (in /usr/lib/ld-2.31.so)
==380774==    by 0x400706D: is_dst (in /usr/lib/ld-2.31.so)
==380774==    by 0x4009796: _dl_dst_count (in /usr/lib/ld-2.31.so)
==380774==    by 0x4009985: expand_dynamic_string_token (in /usr/lib/ld-2.31.so)
==380774==    by 0x4009AE1: fillin_rpath (in /usr/lib/ld-2.31.so)
==380774==    by 0x4009DF3: decompose_rpath.isra.0 (in /usr/lib/ld-2.31.so)
==380774==    by 0x400A708: _dl_map_object (in /usr/lib/ld-2.31.so)
==380774==    by 0x400EE84: openaux (in /usr/lib/ld-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774==    by 0x400F202: _dl_map_object_deps (in /usr/lib/ld-2.31.so)
==380774==    by 0x4014E2E: dl_open_worker (in /usr/lib/ld-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774==  Address 0x4fc39c9 is 9 bytes inside a block of size 15 alloc'd
==380774==    at 0x483977F: malloc (vg_replace_malloc.c:309)
==380774==    by 0x401DA6E: strdup (in /usr/lib/ld-2.31.so)
==380774==    by 0x4009D84: decompose_rpath.isra.0 (in /usr/lib/ld-2.31.so)
==380774==    by 0x400A708: _dl_map_object (in /usr/lib/ld-2.31.so)
==380774==    by 0x400EE84: openaux (in /usr/lib/ld-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774==    by 0x400F202: _dl_map_object_deps (in /usr/lib/ld-2.31.so)
==380774==    by 0x4014E2E: dl_open_worker (in /usr/lib/ld-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774==    by 0x401496D: _dl_open (in /usr/lib/ld-2.31.so)
==380774==    by 0x488E34B: ??? (in /usr/lib/libdl-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774== 
==380774== Invalid read of size 8
==380774==    at 0x401EF18: strncmp (in /usr/lib/ld-2.31.so)
==380774==    by 0x400706D: is_dst (in /usr/lib/ld-2.31.so)
==380774==    by 0x400984E: _dl_dst_substitute (in /usr/lib/ld-2.31.so)
==380774==    by 0x4009AE1: fillin_rpath (in /usr/lib/ld-2.31.so)
==380774==    by 0x4009DF3: decompose_rpath.isra.0 (in /usr/lib/ld-2.31.so)
==380774==    by 0x400A708: _dl_map_object (in /usr/lib/ld-2.31.so)
==380774==    by 0x400EE84: openaux (in /usr/lib/ld-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774==    by 0x400F202: _dl_map_object_deps (in /usr/lib/ld-2.31.so)
==380774==    by 0x4014E2E: dl_open_worker (in /usr/lib/ld-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774==    by 0x401496D: _dl_open (in /usr/lib/ld-2.31.so)
==380774==  Address 0x4fc39c9 is 9 bytes inside a block of size 15 alloc'd
==380774==    at 0x483977F: malloc (vg_replace_malloc.c:309)
==380774==    by 0x401DA6E: strdup (in /usr/lib/ld-2.31.so)
==380774==    by 0x4009D84: decompose_rpath.isra.0 (in /usr/lib/ld-2.31.so)
==380774==    by 0x400A708: _dl_map_object (in /usr/lib/ld-2.31.so)
==380774==    by 0x400EE84: openaux (in /usr/lib/ld-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774==    by 0x400F202: _dl_map_object_deps (in /usr/lib/ld-2.31.so)
==380774==    by 0x4014E2E: dl_open_worker (in /usr/lib/ld-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774==    by 0x401496D: _dl_open (in /usr/lib/ld-2.31.so)
==380774==    by 0x488E34B: ??? (in /usr/lib/libdl-2.31.so)
==380774==    by 0x4A128A7: _dl_catch_exception (in /usr/lib/libc-2.31.so)
==380774== 
==380774== 
==380774== HEAP SUMMARY:
==380774==     in use at exit: 1,028,715 bytes in 2,644 blocks
==380774==   total heap usage: 845,477 allocs, 715,768 frees, 2,541,293,779 bytes allocated
==380774== 
==380774== 32 bytes in 1 blocks are definitely lost in loss record 368 of 1,734
==380774==    at 0x483BB65: calloc (vg_replace_malloc.c:762)
==380774==    by 0xE285CEB: ???
==380774==    by 0xE4BF6D3: ???
==380774==    by 0x8400E8: ash::vk::DeviceFnV1_0::create_descriptor_pool (vk.rs:5426)
==380774==    by 0x5DE393: ash::device::DeviceV1_0::create_descriptor_pool (device.rs:999)
==380774==    by 0x63D5BB: gfx_backend_vulkan::device::<impl gfx_hal::device::Device<gfx_backend_vulkan::Backend> for gfx_backend_vulkan::Device>::create_descriptor_pool (device.rs:1551)
==380774==    by 0x6736B3: gfx_descriptor::allocator::DescriptorBucket<B>::allocate (allocator.rs:164)
==380774==    by 0x675075: gfx_descriptor::allocator::DescriptorAllocator<B>::allocate (allocator.rs:284)
==380774==    by 0x5C0B50: wgpu_core::device::<impl wgpu_core::hub::Global<G>>::device_create_bind_group (mod.rs:1241)
==380774==    by 0x6DB821: wgpu::backend::direct::<impl wgpu::Context for wgpu_core::hub::Global<wgpu_core::hub::IdentityManagerFactory>>::device_create_bind_group (direct.rs:379)
==380774==    by 0x62FEDA: wgpu::Device::create_bind_group (lib.rs:1014)
==380774==    by 0x27E939: <cube::Example as cube::framework::Example>::render::{{closure}}::{{closure}} (main.rs:327)
==380774== 
==380774== LEAK SUMMARY:
==380774==    definitely lost: 32 bytes in 1 blocks
==380774==    indirectly lost: 0 bytes in 0 blocks
==380774==      possibly lost: 0 bytes in 0 blocks
==380774==    still reachable: 1,028,683 bytes in 2,643 blocks
==380774==         suppressed: 0 bytes in 0 blocks
==380774== Reachable blocks (those to which a pointer was found) are not shown.
==380774== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==380774== 
==380774== For lists of detected and suppressed errors, rerun with: -s
==380774== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 0 from 0)

[kvark]

So something is holding on to memory. It's leaking in principle, but not at the lowest level.
I can see it as well on a windows machine, investigating now.

One experiment I made is hooking up the gfx-memory utilization statistics. There is no change in there.

[kvark]

Actually, further investigation showed that the test branch was far behind on wgpu dependency. It didn't have #649 even. I tested with latest wgpu and the memory consumption is very stable, holding around 30Mb mark. The https://github.com/kvark/wgpu-rs/tree/cube-mt-bind-group branch is now updated.

[kvark]

@YuuriMomo would you be able to test with wgpu override set to https://github.com/gfx-rs/wgpu/pull/662 ? We are about to publish this patch.

[YuuriMomo]

Okay!
edit:
No more memory leak in cube example in "cube-mt-bind-group" branch .
Stable 44,8 MB ram usage

[grovesNL]

Thanks! In that case it looks like #662 fixed this