Godot: Why Changing scenes keep using the RAM ?

[ghost]

Godot version:
Godot 3

Issue description:
When changing the scenes .. the RAM Keep raising ... it should get free if the old scene is removed and allocate new memory for the new scene .. right ?
but it's not free the memory if the old scene is deleted

Steps to reproduce:
Go to scene_changer demo ... add few meshes and lights in scene A and run

it's a simple scene .. but if it was a huge scene .. it take a lot of RAM when changing from scene to scene

Minimal reproduction project:
scene_changer.zip

[eon-s]

Looks like a leak but I cannot reproduce with 3.0.3rc2 under Linux, the Object count remains the same when returning to the main scene.

Objects: 710
Resources: 5
Nodes: 10

You can see the object count in the debugger's monitor.

[ghost]

@eon-s yes .. object counts remains the same when returning ... but go to the Task manager and see how much RAM it take every time changing the scenes

[volzhs]

I can see leaks on system monitor, not in godot editor.
it seems because of spatial material.
memory is increasing every time when changing to "Scene A"

[eon-s]

Right, and on the running game, not the editor, check if the exported (debug and release) does the same.

[volzhs]

I tested it 3.0.2-stable release template (X11).
it's same with running on editor.

[ghost]

i really don't know .. but i have a 2 big levels ... when changing between them it take almost 400 MB and the second one take about 700 ...
in this example it take only about 2 MB when changing to scene A because it contain Light and some basic meshes

[volzhs]

without material, it stays stable.
but with material, yes, it eats about 2MB on every time when changing to Scene A.

[eon-s]

I have stored preloaded scenes on an autoloaded script and change_scene_to those PackedScenes, the memory remained the same, there are unreferenced things that are not freed/reused when reloading PackedScenes.

[ghost]

@eon-s yes , exported debug and release does the same

[eon-s]

well, using preload alone with change_scene_to just throw an error but the memory problem is not there, the issue is with load, it seems.

[volzhs]

I think the problem is with unloading scene (especially spatial material thing)
memory is not going down when changing to scene B, only increasing when changing to scene A.

[swarnimarun]

Has anyone tried jumping between multiple scenes.?
Adding a viewport of Scene A to scene B.
Cause the memory to increase when going from A to B and not from B to A.
new_scene_changer
Confusing.

[bojidar-bg]

Is it possible that memory leaks somewhere within the visualserver?

[swarnimarun]

Probably @reduz can see into it. Leaks aren't good.

[volzhs]

this needs to be fixed and cherry-picked on 3.0.x ASAP before 3.1 out, I think.
@reduz would you take a look?

[reduz]

@volzhs This no longer seems to be a bug, and neither Godot internal leak checker or valgrind are reporting any leaks. Closing.

[ghost]

@reduz No , it's still exist in 3.1 Beta 1
don't close it

[reduz]

Which operating system are you using to test? On Linux I clearly see nothing.

[ghost]

@reduz i use Windows 7 x64
Godot 3.0.6 and Godot 3.1 Beta 1
run the Game (Attached in the first comment) , open the Task manager , now look at the memory of the Game process in Task manager , keep pressing change scene button in the game .. you will see the memory keep Increasing

[silverkorn]

Also using Godot 3.1 Beta (Winx64) on Windows 7 Pro 64-bit:

[silverkorn]

Just in case you are interested, the same happens with the 32-bit build on windows 7 Pro 64-bit

[ghost]

@silverkorn Really Thank you man <3
@reduz Thank you for ReOpen it again

[bruvzg]

Xcode Leaks tool doesn't detect anything either.

But memory consumption keep growing on macOS as well (goes form 75 to 200MiB in 3 minutes of constant switching).

[eon-s]

I remember an issue related to unreferenced SpatialMaterials, can somebody test this with 2 simple 2D scenes and see if memory usage changes too?

[bruvzg]

I remember an issue related to unreferenced SpatialMaterials, can somebody test this with 2 simple 2D scenes and see if memory usage changes too?

On macOS, without SpatialMaterials (same 3D project from the first post, only materials cleared) memory consumption does not change.

[bruvzg]

On Linux I clearly see nothing.

Exactly same memory change (and no memory change with cleared materials) happens on Linux too - Debian 9 (stretch), 4.9.0-8-amd64.

[bruvzg]

Here's some allocation statistics from Xcode (about 3 minutes of scene switching on 0.05s timer):

With SpatialMaterials:

Without materials:

Looks like most leaked memory allocation are done while compiling/linking material shader in the ShaderGLES3::get_current_version().

[bruvzg]

Yep ShaderGLES3::version_map hashmap keeps filling with copies of same shader (code_version is increased by one on every switch).

Old copies are not removed during run, but map is cleared on exit, that's why leak testers are silent.

[bruvzg]

Deleting old shaders fixes the leak, but it's probably better to keep it cached and detect that it's the same shader instead, code_version should not change on scene switching.

diff --git a/drivers/gles2/shader_gles2.cpp b/drivers/gles2/shader_gles2.cpp
index 65d4b63bb..d36be6aed 100644
--- a/drivers/gles2/shader_gles2.cpp
+++ b/drivers/gles2/shader_gles2.cpp
@@ -679,6 +679,16 @@ void ShaderGLES2::set_custom_shader(uint32_t p_code_id) {

 void ShaderGLES2::free_custom_shader(uint32_t p_code_id) {
    ERR_FAIL_COND(!custom_code_map.has(p_code_id));
+
+   if (version_map.has(conditional_version)) {
+       Version &v = version_map[conditional_version];
+       glDeleteShader(v.vert_id);
+       glDeleteShader(v.frag_id);
+       glDeleteProgram(v.id);
+       memdelete_arr(v.uniform_location);
+       version_map.erase(conditional_version);
+   }
+
    if (conditional_version.code_version == p_code_id)
        conditional_version.code_version = 0;

diff --git a/drivers/gles3/shader_gles3.cpp b/drivers/gles3/shader_gles3.cpp
index edc2a6c05..1a751550e 100644
--- a/drivers/gles3/shader_gles3.cpp
+++ b/drivers/gles3/shader_gles3.cpp
@@ -741,6 +741,16 @@ void ShaderGLES3::set_custom_shader(uint32_t p_code_id) {
 void ShaderGLES3::free_custom_shader(uint32_t p_code_id) {

    ERR_FAIL_COND(!custom_code_map.has(p_code_id));
+
+   if (version_map.has(conditional_version)) {
+       Version &v = version_map[conditional_version];
+       glDeleteShader(v.vert_id);
+       glDeleteShader(v.frag_id);
+       glDeleteProgram(v.id);
+       memdelete_arr(v.uniform_location);
+       version_map.erase(conditional_version);
+   }
+
    if (conditional_version.code_version == p_code_id)
        conditional_version.code_version = 0; //bye

[akien-mga]

Moving to 4.0 milestone as discussed on IRC:

16:27 <reduz> Ok then, #19300 is not really a bug, but shader caching thing
16:28 <reduz> it sucks, and I need to rewrite the whole thing, but then again, that wont happen until 3.0.
16:28 <reduz> it happens because caches are kept in memory and there is not a proper LRU system to free old shaders
16:30 <Akien> Makes sense, it's also not too critical if it's not leaked I guess? (i.e. RAM usages goes back to normal when you close Godot)
16:30 <reduz> well, nothing is actually leaked when you close godot due to modern operating systems using virtual memory, so all memory allocated disappears anyway
16:30 <reduz> but it can leak a small amount of memory during gameplay
16:31 <reduz> technically it should not be that terrible

[ghost]

@akien-mga This is really really bad news
@reduz why would you want to free the old Shaders ? just keep them and check if the new shader is exist in the Old shaders .. if yes don't add it , if no add it ... at least this will clamp the memory usage ... this is temporary solution until you find a good one ?

Edit : @reduz technically this is really really terrible , this is a small scene , what about a big scene with materials and textures ? what about a full 3d game ?
in our game , starting the game from the Intro scene -> MainMenu scene -> LoadingScreen scene -> Simple Level scene , the RAM is about 250MB
but starting the game from that simple level the RAM usage of the game is about 70MB
and this simple level contain only simple materials without any textures
reloading the level will increase the ram by ~30MB
quit the level and going to the Main menu will increase the RAM by ~50MB
loading this simple Level from the main menu will increase the RAM by ~30MB
the RAM is keep increasing and increasing each time changing scenes
so my Guess is if i create a full 3d scene with materials and textures , will increase the RAM usage with huge amount each time reloading this level or going to another level ... right ?

this will make Godot useless to make Games .. right ?

[starry-abyss]

I agree, this will make the game crash when on 32-bit Windows build the memory usage reaches 2 Gb, and probably the game/OS will also be less performant with high usage of RAM...

[eon-s]

Well, 32 bit should not be a concern since is in extinction, but procedurally generated content and open world where scene loading and unloading is constant it may result in an issue sooner than later, if at least we can have a "manual" way to clean the cache on some systems, it may be enough for the time being.

[starry-abyss]

32-bit Windows 7 is still a thing though, with about 2x more users on Steam than all Linux users together (https://store.steampowered.com/hwsurvey)...

[eon-s]

@starry-abyss Windows 7 EOL is in less than a year, so it won't be a thing for too long.

[Zireael07]

@eon-s: Win XP has EOL a year ago IIRC and it's still kicking, and a lot of people are still on 7 as @starry-abyss points out.

[silverkorn]

As shown by @bruvzg, it happens on any OS. It's the hardware limitation that might get hit by this.

[silverkorn]

I mean, if you run virtual machine(s) or a lot of Chrome tabs in the background by example, whatever which OS is used, this might affect the limitation.

[reduz]

Sorry, I totally misunderstood this issue.

[reduz]

I thought discussion was about the shader cache, that does not get erased (but should not leak), not the custom code which actually was leaking (seems it went unimplemented after porting to GLES3).

[reduz]

Feel free to test again, but I am confident it should work well now

[volzhs]

no more increasing ram on kubuntu 18.10

[capnm]

Crashes when you try to open a shader (via fs-dock or inspector button) a second time.
See https://github.com/godotengine/godot/issues/23672#issuecomment-457789666

[bruvzg]

Crashes when you try to open a shader (via fs-dock or inspector button) a second time.

Stack trace (macOS):

libGLProgrammability.dylib!BitSetFree (Unknown Source:0)
libGLProgrammability.dylib!glpDestroyLinkedProgram (Unknown Source:0)
libGLProgrammability.dylib!handleResetPost (Unknown Source:0)
libGLProgrammability.dylib!ShDestruct (Unknown Source:0)
GLEngine!gleFreeProgramObject (Unknown Source:0)
GLEngine!gleUnbindDeleteHashNameAndObject (Unknown Source:0)
GLEngine!glDeleteObjectARB_Exec (Unknown Source:0)
godot.osx.tools.64!ShaderGLES3::get_current_version() (godot/drivers/gles3/shader_gles3.cpp:206)
godot.osx.tools.64!ShaderGLES3::bind() (godot/drivers/gles3/shader_gles3.cpp:117)
godot.osx.tools.64!RasterizerSceneGLES3::_setup_material(RasterizerStorageGLES3::Material*, bool) (godot/drivers/gles3/rasterizer_scene_gles3.cpp:1177)
godot.osx.tools.64!RasterizerSceneGLES3::_render_list(RasterizerSceneGLES3::RenderList::Element**, int, Transform const&, CameraMatrix const&, unsigned int, bool, bool, bool, bool, bool) (godot/drivers/gles3/rasterizer_scene_gles3.cpp:2224)
godot.osx.tools.64!RasterizerSceneGLES3::render_scene(Transform const&, CameraMatrix const&, bool, RasterizerScene::InstanceBase**, int, RID*, int, RID*, int, RID, RID, RID, RID, int) (godot/drivers/gles3/rasterizer_scene_gles3.cpp:4408)
godot.osx.tools.64!VisualServerScene::_render_scene(Transform, CameraMatrix const&, bool, RID, RID, RID, RID, int) (godot/servers/visual/visual_server_scene.cpp:2140)
godot.osx.tools.64!VisualServerScene::render_camera(RID, RID, Vector2, RID) (godot/servers/visual/visual_server_scene.cpp:1710)
godot.osx.tools.64!VisualServerViewport::_draw_viewport(VisualServerViewport::Viewport*, ARVRInterface::Eyes) (godot/servers/visual/visual_server_viewport.cpp:70)
godot.osx.tools.64!VisualServerViewport::draw_viewports() (godot/servers/visual/visual_server_viewport.cpp:306)
godot.osx.tools.64!VisualServerRaster::draw(bool, double) (godot/servers/visual/visual_server_raster.cpp:106)
godot.osx.tools.64!VisualServerWrapMT::draw(bool, double) (godot/servers/visual/visual_server_wrap_mt.cpp:102)
godot.osx.tools.64!Main::iteration() (godot/main/main.cpp:1885)
godot.osx.tools.64!OS_OSX::run() (godot/platform/osx/os_osx.mm:2573)
godot.osx.tools.64!main (godot/platform/osx/godot_main_osx.mm:100)
libdyld.dylib!start (Unknown Source:0)

[ghost]

There is no memory increase on Windows 7 x64 !!

but the problem is Godot crash if add a Visual shader to mesh , open the Visual shader , add Color node for example , change the color node and link it to Albedo

the second problem is when add a Shader code to mesh , Run the game , Godot work fine , Edit that shader and save the scene and run the game , Godot crash , but after reopen the project again and open the scene , everything work fine until you change something in the shader code .. crash again after run the game

Spatial material is Working Great !!

[akien-mga]

Let's discuss and fix the crash in #25336.

[reduz]

Fixed (I hope)

[ghost]

it's Fixed !!!
Thank you Juan !!
Thank you all !!!
I love you guys !!! <3