Galacticraft: FluidNetwork performance issues on a server

[sfxworks]

REVIEW THE GUIDELINES (LINK AT TOP OF THIS PAGE)

Minecraft version: 1.12.2
Galacticraft version: 177

Single player (SSP), Multiplayer (SMP), or SSP opened to LAN (LAN)? SMP

Galacticraft add-ons and other mods installed? ExtraPlanets

Please provide screenshots / crash report / log as needed.
https://sparkprofiler.github.io/?lG1l2EQvBF

Seems your mod is using up 60% of the server's 2 and a half Skylake cores for a fluid update with only 2 players online. Is there a way you can optimize this process?

[TheQuadShot]

I'm sorry, but I don't understand how you made this determination. The link doesn't really tell me much. Can you provide more information?

[sfxworks]

micdoodle8.mods.galacticraft.core.tick.TickHandlerServer.onServerTick()60.72%
micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.tickEnd()60.51%
micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.onUpdate()60.40%
micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.emitToAcceptors()52.15%
java.util.HashSet.iterator()0.04%
micdoodle8.mods.galacticraft.core.network.GalacticraftChannelHandler.sendToAllAround()0.01%

Per the stack trace it looks like your mod is executing a function each tick. We don't really have any anchors in our world except for a few. And while we do have advanced bases, the only other set that shows up is through mekanism. This caused lag to the point where the server needed to be restarted. It is also abnormal because it happens not on launch or on general use but at random times making it feel like there is some error in the code that would cause this over a period of time or some property it wasn't expecting. I can only theorize until I can replicate it again. I made some optimizations and increased the server's cpu and memory limits in my kubernetes cluster. Even added automatic restarts to handle this if this ever happened again. Though when I ran the spark profiler to see why the server was timing out users and causing so much tps lag, that was the result. One of your functions causing high load on my server. Sorry for the initial confusion.

[radfast]

Thanks @sfxworks

The FluidNetwork is for Fluid Pipes. I am aware that the code has some issues including performance issues and chunkloading issues, but it's not top priority for me because I did not code it ... and the problems are not normally as extreme as your profiler shows, for most people they are minor and fixable.

I will look at it some time. It's difficult (and time-consuming) to debug an issue which is only happening "at random times". I really need to see the issue actually happening on my own test game.

Please bear in mind it could be not Galacticraft lagging here, but instead something in the blocks the pipes are connected to. emitToAcceptors (52.15%) calls code in the block each pipe is connected to. So it could be another mod's block which is lagging.

If possible, it could be very helpful for me to see some screenshots of the player base which is causing this - maybe you have some idea which base because it only happens when player _X_ is online?

[sfxworks]

Thank you for your response. I have enabled a setting in sponge that logs what chunks are loaded / unloaded to investigate this further. I should have replication steps or at least reports for you within a few days.

I have been having trouble with enderIO which may be related based on your reference to emitToAcceptors. I am unsure if the two issues are related but I am by no means an expert in java programming. https://github.com/SleepyTrousers/EnderIO/issues/4725

In the meantime, I have gathered the following log as of this evening. Please look forward to a response for additional details and updates.

[06:21:54] [Server thread/ERROR] [FML]: Failed to refresh liquid acceptors
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        : java.util.ConcurrentModificationException
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at java.util.HashMap$KeyIterator.next(HashMap.java:1466)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.refreshAcceptors(FluidNetwork.java:546)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.getAcceptors(FluidNetwork.java:438)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.emitToAcceptors(FluidNetwork.java:212)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.onUpdate(FluidNetwork.java:384)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.tickEnd(FluidNetwork.java:295)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at micdoodle8.mods.galacticraft.core.tick.TickHandlerServer.onServerTick(TickHandlerServer.java:461)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at net.minecraftforge.fml.common.eventhandler.ASMEventHandler_999_TickHandlerServer_onServerTick_ServerTickEv                                                                                                                        ent.invoke(.dynamic)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at net.minecraftforge.fml.common.eventhandler.ASMEventHandler.invoke(ASMEventHandler.java:90)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at net.minecraftforge.fml.common.eventhandler.EventBus.post(EventBus.java:652)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at net.minecraftforge.fml.common.eventhandler.EventBus.post(EventBus.java:590)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at net.minecraftforge.fml.common.FMLCommonHandler.onPostServerTick(FMLCommonHandler.java:265)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:712)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:526)
[06:21:54] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]                                                                                                                        :       at java.lang.Thread.run(Thread.java:748)

[radfast]

@sfxworks my best guess is there is no link between this and the EnderIO logging issue you linked - that one is about power conduits, this one is about fluid networks.

I really do need information here about which blocks this fluid network is connected to, I appreciate that may be difficult if you have a server with many different players online.

I will probably re-write the Fluid Pipes code in Galacticraft some time, but like I hinted at already, it's not high priority as it's not a change most players would even notice, and we have so much else to do.

[sfxworks]

SGBaseBlock.neighborChanged: BlockPos{x=-7835, y=85, z=7184}
[16:25:44] [Server thread/ERROR] [FML]: Failed to refresh liquid acceptors
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]: java.util.ConcurrentModificationException
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at java.util.HashMap$KeyIterator.next(HashMap.java:1466)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.refreshAcceptors(FluidNetwork.java:546)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.getAcceptors(FluidNetwork.java:438)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.emitToAcceptors(FluidNetwork.java:212)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.onUpdate(FluidNetwork.java:384)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at micdoodle8.mods.galacticraft.core.fluid.FluidNetwork.tickEnd(FluidNetwork.java:295)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at micdoodle8.mods.galacticraft.core.tick.TickHandlerServer.onServerTick(TickHandlerServer.java:461)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at net.minecraftforge.fml.common.eventhandler.ASMEventHandler_999_TickHandlerServer_onServerTick_ServerTickEvent.invoke(.dynamic)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at net.minecraftforge.fml.common.eventhandler.ASMEventHandler.invoke(ASMEventHandler.java:90)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at net.minecraftforge.fml.common.eventhandler.EventBus.post(EventBus.java:652)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at net.minecraftforge.fml.common.eventhandler.EventBus.post(EventBus.java:590)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at net.minecraftforge.fml.common.FMLCommonHandler.onPostServerTick(FMLCommonHandler.java:265)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:712)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:526)
[16:25:44] [Server thread/INFO] [STDERR]: [micdoodle8.mods.galacticraft.core.fluid.FluidNetwork:refreshAcceptors:590]:  at java.lang.Thread.run(Thread.java:748)

Might have something soon. More logs here, but seems like those are the coordinates that started it. It's on the server at those coordinates.

[sfxworks]

I have another report if this helps. https://sparkprofiler.github.io/?SYs6DMja5Y

Looks like its calling functions to load chunks? Can't tell if its doing any saving. We're making a number of adjustments today, so our problems may decrease based on these actions. I will keep you posted on any logs I discover.

[radfast]

I see it, thank you that's an excellent profile

Not sure the cause, though it looks like mostly internal to GC code. Maybe some unexpected construction or large loop? Like I say, I did not write this part of our code, but I'll take a detailed look next time I have a day.

It appears to be fluid pipes adjacent to a Quantum Entangloporter from Mekanism - if that helps you locate which base this is, it would be great to see a screenshot.

[sfxworks]

Thanks for your analysis and input. We are using some additional arguments on the openjdk:alpine build. I have adjusted these and remove most of them as I feel that the server performs better with internal configurations. It also uses a few additional gigs of ram than it should in my case. Either way, removed all the flags so it's more standard now. Thanks for your future help in this case.

I have made an announcement regarding the blocks you mentioned. One user has come forward stating they do have a base meeting those requirements. I will follow up with you when I have that user back in the server. Thank you for looking over that profile.

[sfxworks]

Got another profiler. https://sparkprofiler.github.io/?LXt40CCj02

Identified a base. I can private message you the coordinates if you have time to take a look if you need. I am disabling mekanism chunk loaders and anchors in the mean time to see if that resolves the issue.

[sfxworks]

Yeah, I can confirm that after going into mechanism and disabling chunks loaders I am getting a solid 20 tps now.

World [world] (DIM0) TPS: 20.000, Mean: 9.535ms
World [DIM-1509] (DIM-1509) TPS: 20.000, Mean: 0.160ms
World [DIM-29] (DIM-29) TPS: 20.000, Mean: 0.120ms
World [DIM-1505] (DIM-1505) TPS: 20.000, Mean: 0.094ms
World [DIM-16] (DIM-16) TPS: 20.000, Mean: 0.090ms
World [DIM-1500] (DIM-1500) TPS: 20.000, Mean: 0.090ms
World [DIM-13] (DIM-13) TPS: 20.000, Mean: 0.082ms
World [DIM-1504] (DIM-1504) TPS: 20.000, Mean: 0.086ms
World [DIM-1501] (DIM-1501) TPS: 20.000, Mean: 0.084ms
World [DIM-30] (DIM-30) TPS: 20.000, Mean: 0.087ms
World [DIM-28] (DIM-28) TPS: 20.000, Mean: 0.157ms
World [DIM-17] (DIM-17) TPS: 20.000, Mean: 0.094ms
World [DIM-18] (DIM-18) TPS: 20.000, Mean: 0.085ms
World [DIM-20] (DIM-20) TPS: 20.000, Mean: 0.085ms
World [DIM-19] (DIM-19) TPS: 20.000, Mean: 0.083ms
World [DIM-22] (DIM-22) TPS: 20.000, Mean: 0.488ms
World [DIM-21] (DIM-21) TPS: 20.000, Mean: 0.090ms
World [DIM-1507] (DIM-1507) TPS: 20.000, Mean: 0.086ms
World [DIM-15] (DIM-15) TPS: 20.000, Mean: 0.102ms
World [DIM-1503] (DIM-1503) TPS: 20.000, Mean: 0.085ms
World [DIM-1502] (DIM-1502) TPS: 20.000, Mean: 18.341ms
World [DIM-31] (DIM-31) TPS: 20.000, Mean: 0.204ms
World [DIM1] (DIM1) TPS: 20.000, Mean: 16.218ms
World [DIM_SPACESTATION_Jupiter_3] (DIM3) TPS: 20.000, Mean: 0.183ms
World [DIM-1506] (DIM-1506) TPS: 20.000, Mean: 0.105ms
World [DIM-1510] (DIM-1510) TPS: 20.000, Mean: 0.099ms
World [DIM-1511] (DIM-1511) TPS: 20.000, Mean: 0.094ms
Overall TPS: 20.000, Mean: 49.112ms

[sfxworks]

https://timings.aikar.co/?id=06ff825928494912be429ae92a394d66

Based on this data, are we sure it's just FluidNetwork performance?

[radfast]

Doesn't seem like it can be only GC fluid network. Also looking at those screenshots, I'm not seeing any GC blocks but a huge complex build of other mods' blocks. (You are sure these guys aren't duping?)

Seems more likely now that the GC fluid network is pumping some fluid each tick into another mod's block, and that other mod's block is then causing the TPS problem by running some huge system each time.

It's extremely hard to be sure without taking over your server, tearing things apart, watching the tps and seeing exactly what happens.

If you're able to narrow this issue down to "Galacticraft fluid pipe + [ x ] block from another mod" being the cause of the problem then I'm happy to test some more. But I don't have time to test a huge modlist or a huge server build for an unspecified problem.

I'm minded to close this now. I will still be re-writing the fluid network some time for a general performance increase but it won't be targeting whatever specific problem you are having here.

[radfast]

Actually looking again at the timings it looks like EnderIO conduit bundle could be one of the causes of the lag.

I have not reviewed other mod's energy network code specifically, not for over 1 year.

But last time I looked all other mods _except_ Galacticraft had energy network code which became inefficient for _large_ builds, meaning anything over around 40 energy generators, batteries or receiving machines hooked up to the same wire network.

You might try looking at how your TPS changes if you force players to replace all the EnderIO power cables and TE fluxducts with Galacticraft Heavy Aluminum Wire instead. I have specifically coded that for good TPS on large servers like yours.

[sfxworks]

I have reached out to some players with large bases to assist with testing this. I will follow up once I have further information. Thank you for your suggestion.

[sfxworks]

Just wanted to say my server has made a significant performance increase after having major bases use Aluminum wires instead of other cables and mods. Thank you for that suggestion. After our optimizations to the server and with your great code, the server is healthy again!

[sfxworks]

https://sparkprofiler.github.io/?Jtq7xsjEv1

These are looking somewhat severe. We moved to AWS for higher clock speeds. Taking up 45% of usage of the server with 0/200 players online.

[radfast]

@sfxworks any chance you can repost that spark profiler profile? it's expired by the time I tried to look

I am happy to look at performance issues, especially if given clear direction by good profiling.

[sfxworks]

We have since banned fluid pipes from the world as well as a number of fluid items managed by GC. WE used MMCRestrict to remove them through active chunks with chunk loaders, so much of the active ones are gone. However, seems still hitting around 40% usage on the new server. I have both the old world I can provide you in private either via ssh or a .tar.gz file. Since this world is our active world and players are watching this thread, I cannot share the world map in public here at this time. I have generated this new report in the meantime.

https://sparkprofiler.github.io/?1g1Nx7aWV4

[LemADEC]

This looks like Garbage Collection lag to me.
You probably have a 'chunk-reloading' issue with AE2 and TE.

[sfxworks]

Would I advise those mod's authors' or pass in special arguments? I have some Garbage Collection args in my startup per a sponge post.

java -Xms13G -Xmx13G -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=100 -XX:+DisableExplicitGC -XX:TargetSurvivorRatio=90 -XX:G1NewSizePercent=50 -XX:G1MaxNewSizePercent=80 -XX:InitiatingHeapOccupancyPercent=10 -XX:G1MixedGCLiveThresholdPercent=50 -XX:+AggressiveOpts -XX:+AlwaysPreTouch -XX:G1MaxNewSizePercent=60 -XX:G1NewSizePercent=35 -XX:InitiatingHeapOccupancyPercent=15 -XX:+UseLargePagesInMetaspace -jar server.jar

[LemADEC]

Garbage Collection lag may come from:

• mods allocating too many small objects. A typical example is a mod scanning a large range of blocks and creating BlockPos for all positions instead of reusing a MutableBlockPos. This will fill your RAM with lots of small objects and cause CPU load.
• not enough or too much RAM allocated to the server.
• etc.

Regarding your server settings:

• 13G seems quite generous, are you sure you need that much?
• you're missing UseStringDeduplication

Regarding 'chunk-reloading', make sure you use Sponge default setting, their cache system will help.

[sfxworks]

The server crashes on anything less than 8, so yes.

Based on the profiler report, am I not to assume that "etc." could be Galacticraft in this case? +40% is +40%. Additionally, the server just crashed today when someone tried to go to space...

[LemADEC]

Switching dimension is prone to cause issues in many mods, not necessarily because of the mod who moved the player. Also, without logs, we can't say.

"etc." means simply that I gave the most common source, and the profiler you're using it too limited to know exactly what's causing the issue.
You can check how memory consumption evolve through Java GC logging options.
From a quick code review, I see bad things happening here:
https://github.com/micdoodle8/Galacticraft/blob/4c89f75e33339a9d603afce6703f161c592c5fa7/src/main/java/micdoodle8/mods/galacticraft/core/fluid/FluidNetwork.java#L325

[DarkGuardsman]

I've been talked into help fix the pipes.... If its ok I will see about rewriting the fluid network code for you or at minimal adjusting its performance. Shouldn't be that hard since its based on mekanism's fluid code which was base on my network code.