Galacticraft: Extremely high number of update calls to certain TileEntities, on a large Thermos server

[vadimyer]

Hi,

I've got a public server with ~30 people online on average, running on Thermos B55 and with GalactiCraft 3.0.12.464.

I've got a really annoying and confusing lags because of something I don't really get. Here's the WarmRoast log: https://s.vadimyer.com/WarmRoast.html

Players on our server just used to build energy networks based on wiring from GC, because it's convenient and "efficient" for them (basically no converters et al). At first I thought this is the issue, and found out that in power-GC3.conf config file you can disable energy output for different things, so I went there and disabled output for everything:

compatibility {
    D:"BuildCraft Conversion Ratio"=16.0
    B:"Disable INPUT of BuildCraft energy"=false
    B:"Disable INPUT of IC2 energy"=false
    B:"Disable INPUT of Mekanism energy"=false
    B:"Disable INPUT of RF energy"=false
    B:"Disable OUTPUT of BuildCraft energy"=true
    B:"Disable OUTPUT of IC2 energy"=true
    B:"Disable OUTPUT of Mekanism energy"=true
    B:"Disable OUTPUT of RF energy"=true
    B:"Disable old Buildcraft API (MJ) interfacing completely?"=false
    D:"IndustrialCraft Conversion Ratio"=6.557376861572266
    I:"Loss factor when converting energy as a percentage (100 = no loss, 90 = 10% loss ...)"=100
    D:"Mekanism Conversion Ratio"=0.6557376980781555
    D:"RF Conversion Ratio"=1.600000023841858
}

But that seemed to have no effect at all. The WR logs were still the same.

At last, I started to think, looking at WarmRoast logs and considering how the wiring works, that I should disable both INPUT and OUTPUT. But that will lead to all devices from GC being "isolated" since they will only accept their own energy e.g. energy from GC solar panels.

Can you please provide some info on what could I do with all of this?

[dj3520]

But that will lead to all devices from GC being "isolated" since they will only accept their own energy e.g. energy from GC solar panels.

Disabling the input means that the wires will only accept power generated by Galacticraft (solar panels, coal generator) but will still output to any device that accepts RF or EU. Disabling the output means that only Galacticraft machines can accept power from the wires. Disabling both does as you expect, "isolating" all GC energy from other mods. However that doesn't seem to be the issue, especially after looking at your warmroast report. net.minecraft.world.chunk.Chunk.getBlock() seems to be the real culprit here, taking 8 seconds on one occasion and 3 on another.

If possible I'd check if the problem continues on a forge-only server (not cauldron/kcauldron/thermos) since the issue seems to be the server itself. I've had strange issues with kcauldron before that were immediately solved by switching to forge only.

[vadimyer]

Sorry I can't find that, where did you get net.minecraft.world.chunk.Chunk.getBlock()?

[dj3520]

Sorry for large pictures
Here

And additionally here

[vadimyer]

Yeah thx, I found the 8 seconds one as well.
But how do you count the results? You say something a bit different from how I've used to read WR logs.
Overall profile time is 500 seconds, those 8 and 3 are just a small bit as it looks, as well as their % kind.

BTW, could it be because of regeneration? We wiped GC planets a week ago and the planets are 10000x10000, sometimes (but very rarely), players explore them :)

[dj3520]

While I can't say for sure that the real reason is because of the getBlock() it stands out to me because of the amount of time goes up in small increments after that. Also since you believe it's caused by the energy I looked further down the tree to see what could be causing the lags within the energy network itself.

Unless all of those chunks were loaded again then they haven't actually regenerated. Additionally if no one has chunk loaders over a wide area, those chunks aren't even loaded. The only thing I could see causing an issue here is if there are long stretches of wire connecting remote areas.

[vadimyer]

That's an interesting point as well.

We actually noticed lags because of long enough IC2 energy networks, but then again we couldn't find the real cause (also IC2 dev or someone else said that IC2 networks themselves can cause lags because of just lots of stuff connected to them, especially circularly).

We still couldn't find what's the "recommended" length and load limit of any network.

I have evaluated /ic2 debug dumpLargeGrids command and found that the top complex region is a region split into 9 chunks, with a world anchor in center, and with multiple "stages" from down to top. All of this is connected through GC wire almost completely, to machines from except any mod. But that region still wasn't the lag reason since we completely switched it off (anchors et al) and there was no visible difference on TPS.

Thanks for your help and explanations on this.

[vadimyer]

I'm suspecting there's something with our maps regeneration, because when the Nether is being regenerated I'm going down to 17 from 20 TPS and this is on an empty server.
Also the Nether world at the time of regeneration is quickly "collecting" entities (counter visible through /gc) and at the end it may become 300k of entities there until the restart.
We still didn't regenerate all maps but there's something really strange about all of that.
Excuse me for off-topic if it is.

[dj3520]

when the Nether is being regenerated I'm going down to 17 from 20 TPS and this is on an empty server.

Are you using something that loads the nether? I had a mod on my server that would load chunks on-demand so they would be generated before anyone visited them. If so then it's expected the server takes a dive in performance due to worldgen. However just deleting a dimension's region files shouldn't cause too much of a dip (I think it might do a bit of pre-gen around 0 0 though)

IC2 networks themselves can cause lags because of just lots of stuff connected to them, especially circularly

I always try to avoid wiring circles. Mainly a pet-peeve though. Not terribly sure how much that could affect performance.

[vadimyer]

Yes I'm using WorldBorder to pregenerate wiped worlds, but it wasn't running at the time I've collected WarmRoast logs, I try to run it at nights when there's nobody online, after restart entities collected in gc were always gone.

[dj3520]

If I had to guess the entities count could be the temporary chunkloaders and any mobs in the generated chunks. Using a mod to pregen can really bring the tps down since usually it works faster than someone roaming around.

I try to run it at nights when there's nobody online

The one I used (known simply as "chunkgen") actually had a setting for only generating chunks when no one was online. Because of that the server never _seemed_ to lag to players since it would only get thrashed when no one was there.

Again if possible I'd move away from Thermos since moving forward with it isn't exactly possible.

[vadimyer]

I've ran some TickProfiler profiling and got some interesting results:

https://s.vadimyer.com/a977f4d4-0ba1-47ef-b3b1-d5f4c03accbd.png
https://s.vadimyer.com/76e02cba-cea1-4703-9a5a-54217cf32f75.png

Basically something very different to me from what was shown in WarmRoast (IMO).
But still there's interesting TE called TileEntityLaunchController.
I couldn't find at least one Launch Controller in TOP 5 loaded chunks though.

All in all that there's no point to GC or even IC2.

[vadimyer]

Regarding Thermos, it's just not possible at the moment as Sponge and mods on 1.10/1.11 are still pretty much in ALPHA/BETA. I think we are running 1.7.10 on Thermos for at least the next year, sadly yeah, there's no maintained alternative anymore.

[radfast]

Please ignore some of dj3520 comments, he is not a GC developer.

My comments in this first post are about the Warmroast log posted in the first post. But please also read the posts below, as the TickProfiler issue which at first seemed unconnected is maybe actually the cause of the energy network lag you are seeing here.

I can see 4% of server time taken when IC2 systems (batteries or generators, most likely) are outputting energy into Galacticraft wiring networks. TileBaseUniversalConductor is a Galacticraft wire.

Out of that 4%, most of the time (around 90% of it) is taken by galacticraft.core.energy.grid.EnergyNetwork.refreshAcceptors(). This is when each wire is looking, every tick, to see if it has any block from any mod in any of the 6 positions next to it which can accept some energy that tick (and if so, how much energy it can accept).

On a server, the wires have to do that check, each tick, because something might have changed since the last tick - a block might have been placed or destroyed, maybe a block changed state, a piston moved something, or an explosion destroyed something, or a new chunk was loaded or unloaded. I don't see any way around doing this check.

If that is 4% of server tick time, it suggests you have some players with some large Galacticraft wiring networks, on your server. I don't know how many tiles of wires, but we must be talking thousands, here.

Even so, I would not call this "huge lags". Energy networks wired with Galacticraft wire are taking up about 1/6 of the total IC2 processing time (IC2 is using near 24% of a server tick). That seems reasonable, if a lot of your players are using Galacticraft wires instead of IC2 wires. For example, if half your players are using GC wires, and half are using IC2 wires, this would show that the GC wiring is more efficient than other wires (1/6 of time for GC wired networks, 5/6 of time for other wired networks).

Looks to me like your main problem here is IC2, not Galacticraft. In my opinion, IC2 is inefficiently, coded especially for large scale bases.

The only way to improve this would be to incentivise your players to reduce the number of individual IC2 generators / batteries which output power. So grids of like 80 low powered IC2 generators, all wired up to 1 battery, can be tough for servers. It's better to have 1 very powerful generator block, not 80.

Likewise, instead of 8 mid-powered IC2 battery blocks, 1 high-powered IC2 battery is better.

I don't see that GC is your problem here. I have spent a large amount of time in 1.7.10 optimising the GC energy systems, aiming to reduce the server load on multiplayer servers with large complex bases..

Happy to look at this further, but I'd like to have an approximate idea of how many tileEntities of each type are loaded, when you took that Warmroast snapshot? Maybe also some screenshots of the largest player bases which were online at that time, so I can see examples of the layout.

[radfast]

If you change this to true: B: Disable INPUT of IC2 energy"=true it would eliminate the 4% of server time that Galacticraft energy networks are taking - in that Warmroast snapshot - but at the cost of preventing your players' energy networks from working at all. So that's not really a fix.

[radfast]

I also looked at your TickProfiler screenshots. One of them is showing TileEntityLaunchController taking a large amount of tick time. The problem there is not the tile lagging, but the high number of calls - 118608 calls. That's hard to understand, it's a crazy large number. Normally there should be very few TileEntityLaunchController on a server, most players only build 2, if they even build any at all, so that I have rarely seen more than 20 calls for that TileEntity.

It's extremely difficult for me to debug this without being on your server, able to look around and test things in real time while watching the TickProfiler.

Does the TickProfiler issue grow worse, the longer time it has been since a server restart? So these TickProfiler screenshots were taken after the server had been running for a long time, maybe many hours?

If it's that, then I think the issue could be being caused by some kind of interaction with Forge's chunkloader. Our TileEntityLaunchController is a fairly standard implementation of a Forge chunkloader. At a guess, somehow these chunks are being reloaded when they were never fully unloaded - so you have the same TileEntity being added over and over to the server's list of tileEntities needing update ticks, until a very large number of copies of the same TileEntity are built up.

And maybe the same thing is happening with a machine.TileAssembler - I do not recognise which mod that is from? I can see that has 275,000 calls, meaning 275,000 of them are loaded?!

Actually, this could explain the energy network issue in your first post as well. If there are 400,000 tileEntities loaded on the server, then it is going to take a long time (in nanoseconds) to check 6 adjacent tileEntities, each time that a Galacticraft wire block needs to run that check. Each wire needs to iterate through 400,000 tileEntities to see if any of them is located next to itself... Minecraft is simply not designed to do this kind of thing efficiently.

If this discussion is along the right lines, then it's actually most likely a Thermos issue or one of your Bukkit plugins, because if it's a Forge chunkloader issue it should have been spotted and fixed 1 or 2 years ago. You're not the only server with 30 players online. I would start by looking at any Bukkit plugins which have any connection with chunk loading, and see if you can disable them and run the server without them for a while. Bukkit plugins are not always robustly coded - especially, they are not coded to work with Forge mods, they are mainly for vanilla Minecraft.

[EzerArch]

_(I normally don't like to be off-topic.)_

@radfast: WOW, can I say "welcome back" already? :)

[dj3520]

Please ignore some of dj3520 comments, he is not a GC developer.

Not a developer but I've contributed a few times in a really minor way. 😄

[radfast]

Hey @EzerArch :)

@dj3520 your contributions to the mod are greatly welcomed, and also the help that you give to players and server owners in the Issues list who have solvable issues (like bad downloads, bad configs, RTFM, etc). But as mentioned before it can confuse people if you post detailed developer-style explanations like this one:

While I can't say for sure that the real reason is because of the getBlock() it stands out to me because of the amount of time goes up in small increments after that. Also since you believe it's caused by the energy I looked further down the tree to see what could be causing the lags within the energy network itself.

Unless all of those chunks were loaded again then they haven't actually regenerated. Additionally if no one has chunk loaders over a wide area, those chunks aren't even loaded. The only thing I could see causing an issue here is if there are long stretches of wire connecting remote areas.

What you have written there is _not technically correct_, and it's certainly _not_ a potential cause of the issue which is being reported here. It confuses people because your _writing style_ makes you _sound_ confident but the explanation you are giving is basically taking things down the _wrong_ path.

[dj3520]

@radfast I suppose I forgot my usual disclaimer/asterisk saying I'm only guessing. If github had a signature system I would have added it there already.

also the help that you give to players and server owners in the Issues list who have solvable issues

That's mainly where I try my best to help. If I think it's too complicated I don't say anything. But with small things I feel like I can take care of I feel it's smarter for me to help and take that small little effort someone else would have to do (such as yourself) and do it myself.

[radfast]

Thank you.

[radfast]

Anyone affected by this or similar issues:

My current theory is it's related to this combination of factors:

• players leaving and returning to the dimension in rockets or other ways (teleport, nether portals, etc)
• the dimension has a chunkloader (GC Launch Controller, Railcraft Anchor, ChickenChunks Chunk Loader, Witchery shelf etc)
• when all players leave the dimension, the chunks _outside_ the chunkloaded area are _supposed_ to be unloaded
• the player has built machinery / wires / pipes / itemducts which crosses the boundary from the loaded chunks to the chunks which are _supposed_ to be unloaded
• another mod - most likely a wiring / pipes / itemducts mod, so look at Mekanism-EnderIO-Thermal Expansion-IC2 etc... - wrongly reloads, over and over, those chunks which are supposed to be unloaded
• tile entities somehow get duplicated in the chunks which are being force loaded and unloaded over and over

This explains why the issue only happens / or gets worse when the player leaves the dimension.

---

I still need help from you guys so that I can actually see this issue happening on my test systems. Especially, I need to know the smallest combination of mods and build which can make this occur.

Anyone who sees this issue in future, please:

1. Provide a full mod list, highlighting wire / cable / pipe / itemduct mods
2. Provide screenshots of the base which is lagging, with chunk boundaries shown
3. Especially in your screenshots show any chunkloaded chunks, and the connections they have with the chunks next to them
4. Try to force the problem to occur, by all players taking a rocket or teleporting to another dimension, and leaving the base unoccupied (but still loaded by the chunkloader)

[radfast]

Any update on this, please? @vadimyer? others?

[radfast]

De-prioritising as no feedback.

[radfast]

Closing, as this issue seems inactive. We can't do anything with an issue we can't reproduce, and chances seem high this is caused by something in another mod.

Anyone is welcome to comment or re-open if they have specific information to help to resolve this.