Sponge: Memory leak when intensive work with chunks

[Lignium]

I am currently running

• SpongeVanilla version: spongevanilla-1.12.2-7.1.6
• Java version: OpenJDK 1.8.0_222
• Operating System: Kubuntu 18.04 LTS
• Plugins/Mods: None

Issue Description

What is the problem here? A memory leak on the Sponge side or a problem in the game itself? World#newChunkPreGenerate() is not suitable in my case. :)

Test code: https://gist.github.com/Lignium/14cfe4232eac1dc872454e37b2d3e1e9
Crash report: https://gist.github.com/Lignium/cee8d601e1373de8c605865221c89d0e
Latest log: https://gist.github.com/Lignium/780adeebbdde2fa244cdd14667eae4da

VisualVM:

[Cybermaxke]

This is because when you call unloadChunk, the unload is queued to be performed later (the next tick?). Meaning that all the chunks are kept in memory during your event handler.

[Lignium]

My plugin performs a heavy operation to generate a huge map on command, and blocks the main thread for a long time. Is there a way to force unloading of chunks here and now?

[Cybermaxke]

Currently this isn't possible. Is there a reason why you can't use a task and process the chunks in batches?

[Lignium]

Is it possible to add methods to force data dumping in API 8? I'm afraid that it will be too difficult to remake the program so as not to lose speed by skipping ticks.

[JBYoshi]

What are you trying to do?

[Lignium]

Nothing yet. I think over different ideas, but everything does not fit... I need a method that can instantly unload certain chunks in the current tick, or for example, a boolean flag for the existing World#unloadChunk() method. The question now is, is it possible to add such functionality to API 8?

[JBYoshi]

I mean, why are you trying to process so many chunks?

[gabizou]

Put it this way: What are you wanting to do, not how, not with what, but what's the objective you want to get to? Do you want to run a world pre-generator? Do you want to replace existing chunks with new chunks from a custom generator to make re-healable dungeon maps? Do you want to chunk regen only specific chunks? Because so far, you've said you wanted to generate a bunch of chunks and immediately unload them, but not explaining why, and then you talk about dumping the data/memory of them? Do you see how there's a huge chunk of idea that is missing behind what you're trying to achieve by unloading chunks?

[Lignium]

Hmm. I understood. Everything is complicated by the fact that I can not reveal the unique idea of ​​the plugin. Therefore, all I can say is that some map of a given size is generated on top of the existing world, a group of chunks is loaded as necessary, blocks are changed in them by restoring snapshots to the desired positions, then these chunks are unloaded due to uselessness and the program goes to the next group of chunks and so on. This must be done at one time, even if the process is very long. I do not know what else can be said about this. Chunks are apparently not unloading, the method of unloading the chunk is idle, the heap is clogged, the program crashes.

[Lignium]

Now the question is not how my plugin works, but how do I unload a chunk, albeit slowly, but at the time of the call. This is really an urgent need for the plugin, and it is apparently not that difficult to make it in API and implementation (I walked around your code a bit and learned a bit how it works). Unfortunately I can’t do pull-request to offer this functionality, because mixins are very complicated for me so far. I hope for your help. Such a good plugin may fail due to such a trifle, or it will be necessary to allocate a huge amount of server memory, all 128 gigabytes, to overcome the leak. Maybe I need to create an issue for this on SpongeAPI?

[Lignium]

I rewrote the plugin code, now not only have I bypassed this bug feature, but I have also acquired more advanced features. Now I can split a map into small sections (clusters) and this feature will even be in my favor, chunks will not be unloaded quickly (the time turns out to be configurable in the Sponge configuration), which means that performance between cluster generation calls will not be lost. I think this issue can be closed...