Paper: Massive performance degradation with entity ticking.

[mibby]

Paper dev 493

@electronicboy Did one of your entity commit patches recently break entity ticking? Such as possibly forever ticking spawned entities and never "forgetting" them when unloaded or outside activation ranges? I'm having massive performance degradation due to tickEntity and entityTick after updating to newer Paper builds. Performance just continues to get worse the longer the server has been up.

    entity-activation-range:
      animals: 16
      monsters: 24
      misc: 4
      water: 12
      tick-inactive-villagers: true

https://timings.aikar.co/?id=55612766e3ba430cbb0e24ac32ade4d0

Edit: Or perhaps something upstream in spigot broke? :(
https://timings.aikar.co/?id=d64f3f81a4a84ae3b4e524281d571e46
https://timings.aikar.co/?id=48466c5bc10344c4acfe6aa7b3f8444f
https://timings.aikar.co/?id=27fafa4f87094ede9ec907eb4d895f6e

[electronicboy]

Can you actually provide some insight instead of pure skepticism? While yes, there is always a potential for issues to occur, I'm not going to waste my time looking into something of which you've not provided any form of insight that there is a regression, only that your server is lagging.

All I can see is that you have a lot of entities on the server from the information that you've provided. What build does this start in? Timing comparisons of the pre-broken and the broken build? etc

[NullCase]

I too have noticed entity lag (since around paper build 480ish).

Hopper lag was much worse before i set disable-move-event: true.

There are two periods where tick loss picks up. First is from chicken entities. Second is from cows. In both cases, the number of entities shown in the timings report is about 5-10x as many entities in the game.

Chickens also cause far more lag than expected (~70 chickens causing a 15% TPS loss).

It seems that Cows (which number around 2-300) are causing 36% server load. Note that this is a 6GB server.

https://timings.aikar.co/?id=5dd9acab5fe448549db0db2b8e2c03b9

[mibby]

I wouldn't say it is necessarily pure skepticism @electronicboy, but rather indication something possibly went wrong upstream based on analytics and insight into the issue after updating Paper. I mentioned you because I thought perchance the recent entity commits you pushed live may be affecting something negatively and you may know why - it wasn't intended as any personal jab.

I do not have any timing profiles prior to updating, but reviewing my Plan analytics, TPS in December always averaged above 19.5 with 4-5k entities and the same player count. Likewise as noted by @NullCase, I'm fairly certain I was using Paper dev 480 and updated directly to 493. Since updating, entity ticking performance dropped significantly with millions of check counts.

It's a bit complicated for me to downgrade due to breaking upstream Spigot nms changes and plugins which have already updated to correct this.

[NullCase]

with millions of check counts.

This is what I'm seeing. I've observed as many as 64 million hopper checks in 8 minutes.

[chdxD1]

I am also experiencing a quite similar issue. We are seeing a large amount of lag (TPS at around 10) with not a large amount of entities.

We are running version 500 but also tried 480 (same result).

Attached is timings and picture from warmroast.
https://timings.aikar.co/?id=f4650a6388d643af95b593f81f121267

[Brokkonaut]

I can confirm this issue... There was a huge performance degradation in one of the more recent changes.

[Ghost-chu]

I can confirm this issue too.

[NullCase]

problem still exists in build 503.

[electronicboy]

Anybody planning to actually help by working out what build this regression occurred?
We only have so many people with so much time available, nor do we have test environments that we can actually load up enough to search for these issues, so far we have 13 potential builds for an issue to have been introduced in

@chdxD1 Are you sure that that is relevant to this issue, given that you supposidly had issues in a build that was pre this issue? That VoxelShape stuff is mojangs which is a known sizable overhead, but it's not really maintainable to start ripping all that stuff out

[mibby]

Not sure if related but I've been getting a lot of complaints in regards to an abundance of mob spawns with natural guardian spawns in the ocean. Possibly all mob spawn rates affected if spawn ticking is being doubled.

[NullCase]

What if we pool together some money for the person who finds the build this regression occurred in?

I'll chip in $7.53 USD.

[hurrgurl]

• Paper 488 here
• Pure vanilla
• Lots of entities (1k+)
• Powerful hardware

The only issue I have is this:
https://imgur.com/a/fqHXJqP

https://sparkprofiler.github.io/?6Konmaw1om

Don't feel too overwhelmed, been running 488 for weeks. Everything is mostly fine.

Until the issue is nailed down and reported properly - don't spend too much time on it.

[NullCase]

Tested down to Build 441. As best i can tell, its from an earlier build.

Edit: i wouldn't necessarily trust my results either. which is why i'm happy to chip in some money for someone more capable.

[dextonanderson]

This seems to still be an issue as of 1.13.2 build 632.

https://timings.aikar.co/?id=169fabd3cdb2413f9963106429530f0f

[electronicboy]

Closing this ticket, all information in here is effectively outdated, and in general, not all too relevant to the current issues surrounding entityTick, there have been various improvements have been made relating to this issue in 1.13.2, specifically cutting mojangs crazy stream usage in some areas