Dynmap: PLEASE! PLEASE! PLEASE! HELP FIXING THE FREAKING MEMORY LEAK!!!!!!!!!!!!! GC OVERHEAD LIMIT

[k-jiang]

Memory leak ALL THE TIME when I try to run fullrender in an independent server. Makes it impossible to use dynmap.

But first of all, I have literally done all, ALL of the possible ways I could to resolve this issue. Including:

• increase -Xmx to 3GB - failed!
• -XX:+AggressiveHeap - failed in hours!
• Parallel GC -XX:+UseParallelGC -XX:ParallelGCThreads=4 - failed in hours!
• CMS GC -XX:+UseConcMarkSweepGC -Xmn1024M - failed!
• CMS GC with Aggressive mode -XX:+UseConcMarkSweepGC -Xmn1024M -XX:+AggressiveOpts - failed!
• NO GC -XX:+DisableExplicitGC FUCK THE JAVA - failed in a short period of time!

Those are my java params I have used to run dynmap, and all failed:

#java -server -Xmx2048M -XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC -jar mapserver.jar nogui
java -Xmx2048M -XX:NewRatio=1 -XX:+AggressiveHeap -jar mapserver.jar nogui
# No GC - Fuck you java!
#java -Xmx2048M -XX:+DisableExplicitGC -jar mapserver.jar nogui
# Parallel GC - max 2048m, old/new ratio=1, gc threads=4, gc cpu time 20%, max time used on gc = 5000ms = 5sec
#java -Xmx2048M -XX:NewRatio=1 -XX:+UseParallelGC -XX:ParallelGCThreads=4 -XX:GCTimeRatio=20 -XX:MaxGCPauseMillis=5000 -jar mapserver.jar nogui
# CMS GC - max 2048m, old/new ratio=1, 1gc per compaction heap, compress at every full gc
#java -Xmx2048M -XX:+UseConcMarkSweepGC -Xmn1024M -XX:CMSFullGCsBeforeCompaction=1 -XX:+UseCMSCompactAtFullCollection -jar mapserver.jar nogui
# CMS GC Aggressive mode - old/new ratio=1, AggressiveOpts includes AggressiveHeap unstable yet fast
#java -Xmx2048M -XX:+UseConcMarkSweepGC -Xmn1024M -XX:+AggressiveOpts -jar mapserver.jar nogui

What my configuration looks like:

renderinterval: 1
renderacceleratethreshold: 60
renderaccelerateinterval: 1
tiles-rendered-at-once: 1
usenormalthreadpriority: true
saverestorepending: true
save-pending-period: 900
tileupdatedelay: 30
smooth-lighting: true
image-format: jpg-q80
disable-webserver: true
parallelrendercnt: 4
fullrenderplayerlimit: 0
updateplayerlimit: 0
per-tick-time-limit: 50
update-min-tps: 18.0
fullrender-min-tps: 10.0
zoomout-min-tps: 10.0
(and etc...)

And this is my server configuration and how I run my render:

Linux Ubuntu 14.04 LTS
4GB ram
4 x L5640 2.27GHz

A VPS but run dynmap only during
MAXIMUM CPU consumption while dynmap id running: less than 50% (200% in a 4-core platform) and no cpu limitation has been set. I have tried running some bash scripts and they can definitely eat up 100% of the cpu.
No other scripts / programs are running during rending

Here is what plugins is install into the render server

spigot 1.10
dynmap 2.3
Essentials (for player homes)
WorldGuard (for regions)
multiworld 5.2.8 (there are 2 maps need to be render)
PlotSquared and Plot2Dynmap (for plots)
SignShop
Vault
WorldEdit
It is way less plugins than what I have on my main server (53 plugins!)

Here is what happened when fullrender started:

1. At the beginning, render seems runs fast and normally, 555.66 msec/tile, 116.84 msec per render for overworld, 80.83 msec/tile, 39.67 msec per render for plots, tps 20
2. Then after few munites, render rate drops, tps drops
3. Then render stall and stops with no warning/notification/error
4. Server crashed due to the fucking java gc over head limit

So maybe I have done something wrong with my startup script or the configuration? If there is, please point it out. I would appreciate.

Also, please add a little section on your dynamp plugin thread (bukkit.org or minecraftforum.net) specify what the java params or configurations need to be on startup so the dynmap won't crash. Setting up an example would be better. I strongly believe that this not only a rare case for me but for all the people who tried to run dynmap as an independent service. Today there are less and less people would run both the map services and minecraft on the same machine. Or it would put a huge hole and performance drawback on their network otherwise. So please please please guide us or I will have no choice but forced to switch to the other map rendering mechanism.

But what I actually know, after those try-and-error, is JAVA GC IS ABSOLUTELY BULLSHIT! There is no way for dynmap handling a simple 4000x4000 full render task by simply rely on the shitty java gc without eating infinite amount of memory then crash. So I would say maybe loading the chunk with some third party method (no by bukkit or spigot) is the only way out since you can have more control on memory leakage.

1431 #1916 #499

[Psy-Virus]

Hey @k-jiang,

I don't think, it's only Dynmap causing the ML. Because: I had a few servers setup and running in the past. Mostly Bukkit, tried Spigot (both on the latest versions at the times) and now it's Forge (latest 1.7.10). Dynmap was running on all of them except the Forge Server since some weeks because of the massive amount of RAM usage caused by Dynmap, the related Chunk loading or what ever.
Dynmap, like every other Mod, doesn't make the server run better. Java is one of the, at least, most suboptimal programming languages a game could be written in. And then, Minecraft. Let's be honest: A game that looks like sh*t and eats resources like nothing good (in relation to equal games).
Then you have modded servers. People from around the world are working on APIs to get mods implemented with this absolutely not optimized base.

I think the main problem of our servers is not Dynmap, It's Minecraft and its not really optimized program code.
The strange thing is, as written before, I'm currently running a Forge server with around 60 mods. It starts and eats up to 1.9G of RAM. With and without Dynmap. But if it comes to generating new chunks or if existing chunks are loaded by more than one player, more and more RAM is used (with and without Dynmap) and as u said, the GC is sh*t.
Back to the strange thing... On the client side, this doesn't happen. I'm constantly by around 1.8G (with more mods) but gave Java the option to use up to 8G. (maybe because of no chunk generation or because the client knows how to unload a chunk(?) idk)

So to make things short, I think the problem has to be searched in the roots if the crown is dying.
Sure, Dynmap loads lots of chunks in short time, but other mods, like worldborder and other world pregen tools are doing the same and eat also much RAM in short time.
But the RAM isn't freed if the chunks are not active any more. So my guess it's either Minecraft, Bukkit, Spigot or whateverest, that causes also a big part of the problems.

My two cents
Psy

:-)

PS: Great post structure and details! U got a like for that ;-)

[jwshields]

Using caps is a great way to get help. /s

Why do you believe this is related to Dynmap? What do the crash dumps say?
Could you post your dynmap config, bukkit/spigot config, as well as crashdumps?
What versions of all your software are you running (OS, Java, Minecraft_Server, spigot, etc.)
And how is your server configured with map generation not on the server hosting the world?

[Psy-Virus]

@jwshields, did u even read the op? The most informations u request are given!

[jwshields]

@Psy-Virus
OP only had a snippet of the dynmap config- not the whole thing. nothing related to the bukkit config; no crashdump info, so people can't troubleshoot why the server actually stopped. OP could also be running a funky/old version of Java.
The information listed is incomplete and doesn't really give the whole picture

[k-jiang]

@jwshields
Tested on dynmap-2.4 dev build (I tried 2.3 etc, and crash)
Spigot 1.11 build 1105
Now with the newer version of dynmap, it is not going to crash but extremely laggy (completely stopped after just a few renders, see log below).

command used to run spigot:

java -server -Xms2048M -Xmx2048M -XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC -jar maprender.jar nogui

cpu: 4-core linux 80-120% (equvelent to 22-28% in a Windows)
TPS from last 1m, 5m, 15m: 8.67, 9.31, 10.25
gc:
log: http://pastebin.com/1pWeExYi
Timings output while server completely stalls: https://www.spigotmc.org/go/timings?url=gitaxorexi

dynmap configuration: http://pastebin.com/5f2jYBsz
server.properties: http://pastebin.com/LTFgAj0Z
bukkit.yml: http://pastebin.com/cM2LgBp3
spigot.yml: http://pastebin.com/PwCntCww

[chuushi]

Based on your timings, Dynmap-Essentials v0.91-SNAPSHOT is the problematic plugin. Have you tried using Dynmap and just Dynmap plugin alone?

It also says that it's using JsonFileClientUpdateComponent (component for an external web server) which constantly writes update data to a JSON file (to your disk) to write map update data. The writing data to disk seems to be taking a long time for Dynmap. Try using InternalClientUpdateComponent and the internal webserver.

[k-jiang]

https://timings.spigotmc.org/?url=samoqijoyu

It doesn't solve my problem. I have tried to increase my Xmx to 2GB, disable saving, reduce view distance to 4, disabled entities generation and activation. Yet it still sucks. The stupid java just uses all the CPUs to do an extremely inefficient GC FOREVER after fullrender stated few munites and stalled.

So please try to fix the memory leak somehow!!

[jwshields]

In the latest timings, dynmap isn't even causing the issue.
I would also recommend updating to 1.11.2, 1.11.1 caused a bunch of problems.

[k-jiang]

Finally! I got the reason of problem:

https://www.spigotmc.org/go/timings?url=govewaxita
1.11 log http://pastebin.com/j6BMQvdk
1.11.2 log http://pastebin.com/LjXNZ4TY

server.properties http://pastebin.com/sS3ZaSQ5
bukkit.yml http://pastebin.com/WiPGrjhD

I have basically tried every bullshit I could, including reduce view distance to 2 and force the bukkit to do chunk-gc (see settings above). But you see, the loaded chunks just keep increasing themselves BECAUSE DYNMAP IS SOMEHOW KEEP LOADING MORE CHUNKS THAN BUKKIT COULD UNLOAD. Note that I already set dynmap tps pause to 18.0.

My humble suggestions for you:

1. STOP LOADING NEW CHUNKS WHEN TPS LOW! Respect to the tps-pause setting.
2. To build a loaded chunks detection into dynmap. Dynmap should have option to pause any fullrender/stop loading new chunks itself when the number of loaded chunks is dangerously high until it returns to a safe value. For example "fullrender-pause-when-chunks-over" and "fullrender-resume-when-chunks-under". It is a little bit like the "-XX:MinHeapFreeRatio" and "-XX:MaxHeapFreeRatio" in java (http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#0.0.0.%20Total%20Heap|outline).

If you need any more testing (or tunning, even though I'm already tired of) please let me know. I would be glad to try your new fixed dev build on my server and send you feedback.

[k-jiang]

Suggestion 3:
like 2 above, but "fullrender-pause-when-free-memory-over" instead.

[k-jiang]

FYI:
https://hub.spigotmc.org/javadocs/bukkit/org/bukkit/Chunk.html#unload(boolean, boolean)

[chuushi]

The chunk unloading API itself was causing issues back when Minecraft 1.9 was introduced, so we moved away from it (Issue #1916).

[ice-fly]

2GB 🥇