Paper: Improvement of Minecraft Server Advancements Memory Leak

[PonyPC]

Is your feature request related to a problem? Please describe.
Version: 1.15.2
After about 100 players playing several hours, jvm used up all of Old Gen heap and trigger a full gc.
TPS start to drop down. I found someone also suffering this problem in papermc.org from 1.12 which mojang designed a new advancement system from : https://papermc.io/forums/t/help-needed-with-memory-leak/437

A week ago I thought it might be a paper bug. But after yesterday, I'm wrong, I realized it is caused by the Advancements in Minecraft through jvm heap dump. For some reason, the hashmap in Advancements only put in online player but never pop the player who goes offline. Also, it put same player more than one time. Finally it causes huge memory use and jvm need spend more time to gc.

Describe the solution you'd like
So I disable advancements in spigot.yml like this:

advancements:
  disable-saving: true
  disabled:
  - *

And my memory and TPS goes back! Here is the objects count differences:
BEFORE:

AFTER:

Describe alternatives you've considered
Mojang designed a bad data structure of advancement cause memory leak. Can paper fix this by patch if someone need the advancement in games ?

[Ghost-chu]

Same problem on my server, ram extremely expand after server running few hours.
Heap dump too big even crashed dump viewer.

After disable all advancements, the problem seems solved.

[electronicboy]

without a heapdump, this issue is mostly useless, the nature of there being 6.6k AdvancementDataPlayer objects in your heap also points to a leak of player objects, while mojangs impl is far from amazing, I severely doubt it is the real issue here

[ian-otto]

I can second this behavior. I have a large-ish Minecraft SMP server which requires reboots every 12 hours due to leaking memory.

I can provide a heap dump (it's 6GB). The number one consumer is AdvancementDataPlayer objects at 1.6GB's of the heap. Disabling advancements seems to alleviate the problem. Eclipse MAT describes this as the most likely source of a leak.

[aikar]

AS mentioned on the other issue, there is not actually a leak.
This is just the games really ineffecient and high volume of data being stored to implement advancements live data set.

Saving off is only way to mitigate.

[PonyPC]

@aikar So someone goes offline and the data still exists in jvm, is it normal? The mount of advancement data never reduced.

[electronicboy]

The memory will still exist while it's saved, but, if it's still strongly ref'fed after they've been fully DC'd, that's an issue; But, we've yet to see this in the server itself. it's always been caused by plugins everytime we've cared to dig into heap dumps

[PonyPC]

@electronicboy Emm, I'm confused. I do all the test without any plugin. I give up and keep disabling advancement.

[aikar]

So your saying that you logged in, logged out, without any plugins, and a heap dump still contained the advancement data with strong reachable?

I reviewed this code, and mojang is properly cleaning it up... So I don't know how that would be possible unless advancement data is loaded for a player who's disconnected during the LOGIN stage that doesn't use same disconnect method.

I will recheck where the data is loaded, but I'm pretty sure that was in the stage that the cleanup disconnect method should be used too.

[PonyPC]

So your saying that you logged in, logged out, without any plugins, and a heap dump still contained the advancement data with strong reachable?

Yes, but not always. The amount of advancement data in memory greater than online player. It increases through server running.

[electronicboy]

The amount of advancement data in memory greater than online player - that doesn't really say anything beyond advancement data being inefficient, increasing while the servers running is just typical "players are doing stuff which counts against the advancement stuff so we're saving more counters, etc"

[PonyPC]

Also in my heapdump, same players advancement data appeared several times.

[aikar]

ill look into again and see if i can reproduce that.

[Dreamurrrr]

Describe the solution you'd like
So I disable advancements in spigot.yml like this:

advancements:
  disable-saving: true
  disabled:
  - *

@PonyPC the solution you provided is wrong. If you put in - * under the disabled collum, Paper will just throw an error and stop the server from booting up. - "*" will work, not - *.

[Dreamurrrr]

I can also confirm that I have this issue too, @aikar.

[electronicboy]

We need info such as dumps, blind "same" comments literally give us 0 information

[Spottedleaf]

https://github.com/Spottedleaf/Tuinity/commit/55a069132dac018e768ef32c0751562a7ea2324e

[Dreamurrrr]

How is saying that I also have an issue a blind statement? Before I disabled advancements my server was using 10 gigs of RAM (I allocated 5 gigs). After disabling it, it went down from 10 gigs to 5.6 gigs. If it really bothers you, I can send you my dump, but it looks pretty much identical to the one posted by the issue author, @electronicboy.

[aikar]

Paper 1.16 build 104 and Paper 1.15 build 380 has the fixes for this now thanks to @Spottedleaf

[PonyPC]

happy ending