Galacticraft: Absent from Space Station causes immense TPS drops

[LokeYourLord]

So i first built a Space Station, as some can remember from my post about implementing better Oxygen Pipes, at which i ended up just using EnderIO as no one seemed to care. Anyways, that Space Station became relatively big, and started causing lag, when i was ABSENT. Yes, that's right, when i was at the Space Station, the server TPS was 20. When i was absent, down at Earth or other Planets or places, the TPS dropped so low, i had to constantly be there so the server wouldn't lag out. I noticed this first just by the fact that the server lagged out every time i left the Space Station. Then i did a check with OPIS, and it said my Space Station causes the lag, but only when i was not there. Now, i cannot provide OPIS logs, as i had to remove OPIS due to it being so heavily outdated, that MapMaker caused problems with other mods, so i removed it. So i installed TickProfiler. I ended up putting my Space Station into "Hibernation" mode, meaning that i changed its dimension ID, so that it would not be loaded by Galacticraft or any other mod. Currently it is inaccessible, and it should stay like that with the current lag it causes. But now the problem goes further: As someone else started building their Space Station, the same problem started happening. Basically, his Space Station came to just quarter the size of mine, and is already causing lag, and there are bases on Earth or other places that are way larger than the Space Stations we have. Yet, they cause at least 5 times the lag those bases do, even being far smaller than the bases on Earth or that. So i did a quick Profiling with TickProfiler, and it turns out that the Oxygen Sealers still are the cause for the lags. I disabled the edge checking (B:"Enable Sealed edge checks"=false), to reduce lag, yet it does very little difference. Anyways, once i went to the Space Station that caused lag, guess what, it stopped. TPS is again near 20, or even 20 at most times. Once i leave, the server becomes a nightmare to play on. I will give you some screenshots.

The Space Station:

The TickProfile report when being at the Space Station:

And the TickProfile report when being absent from the Space Station, basically any fairly (medium) large Space Station (images are merged, text was too long):

Smells funny, doesn't it? Anyways, that is a big problem. The Oxygen Sealers start causing lag once someone is absent from a Space Station, which is plain stupid.

[XFactHD]

Are you using the latest version of GC? If not, can you check if updating fixes this.

Offtopic: What mod are the dimension and the blocks the SS is built out of from?

[KingOfMiners]

GalaxySpace.

[LokeYourLord]

Yes, GalaxySpace. But it does not matter, my Space Station was standard Galacticraft, out of Tin Decoration Blocks, and was above the Moon. The lag caused was the exact same. I have the almost latest update, but it hasn't done any difference the latest updates, so i took a look at the changelog, nothing there, lag does still exist.

[LokeYourLord]

I see that in update 446, some Oxygen stuff seems to be fixed, i will test it, but i find it very unlikely that it will solve much, as it seems the Oxygen Pipes were fixed, but not the Sealers themselves.

[XFactHD]

Thanks @KingOfMiners.

[LokeYourLord]

Won't be able to test with 446. GalaxySpace has a problem with it, where in the Mars Space Stations cause a crash with 446. Somewhat of a paradox, but basically i wont remove GalaxySpace to test if the lag is gone with 446, as i would have no Space Station to test with currently. And since Blesse isn't the exactly fastest hotfix guy, i don't expect an update to follow anytime soon.

[radfast]

Duplicate of #2117. This issue is fixed in Galacticraft 446.

i find it very unlikely that it will solve much, as it seems the Oxygen Pipes were fixed, but not the Sealers themselves.

Please test instead of speculating.

[LokeYourLord]

Mr. radfast, read the one i just wrote directly before you, then you see.

[radfast]

You might be able to fix the issue in pre-446 Galacticraft by changing JVM settings. The issue is caused by the Java Garbage Collector, not by a bug in Galacticraft as such.

If you provide the crash report with GalaxySpace and 446 that's maybe something I will want to look at - @BlesseNtumble any idea for the reason?

[LokeYourLord]

Oh i did, here you go: https://github.com/BlesseNtumble/GalaxySpace/issues/18

My crashlog is in the bottom.

[LokeYourLord]

Would it be possible for you guys to fix the GalaxySpace issue in 447?

[radfast]

I took a look at the crash report, that one can only be fixed by @BlesseNtumble, he needs to update the Mars space station to reflect recent changes made to the Overworld space station and Galacticraft API. I can make a change in 447 to make that easier for him, but I cannot do it for him.

Seems like you will be stuck with this issue until GalaxySpace is updated. Unless maybe the JVM settings help?

Try -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:-UseAdaptiveSizePolicy

[LokeYourLord]

rad, i cannot modify my JVM settings, its a Dedicated Server with no ways to modify JVM arguments, i would if i could :/

[dj3520]

@Yanniclord I run two dedicated servers myself and have access to the JVM arguments, so unless you're using a hosting company that doesn't let you, please tell me how you start your server.

[LokeYourLord]

"so unless you're using a hosting company that doesn't let you" <--- that.

[LokeYourLord]

So, the lag comes back every so often. It is most of the time gone, but on rare occassions, the server starts lagging. We were absent for maybe 30 minutes, started importing items through Ender Chests, and after 5 minutes it started to lag.

[LokeYourLord]

@radfast There should be some kind of unload dimension command, so that the dimensions can be force unloaded. And the Java Garbage Collector you were talking about, still does not seem to work a 100%. As i said, the lag still exists. Being absent for too long makes the server lag. Oh and this crash can actually happen, for no good reason. http://pastebin.com/srLa1fSU

[dj3520]

I doubt there will be a force unload command since there isn't really any use-cases for it, and it would be more work than it's worth. Also you state that you've tested the Garbage Collector JVM arguments despite saying you cannot change them due to your hosting company.

If your problem doesn't happen without GalaxySpace, then it isn't a Galacticraft issue.

[LokeYourLord]

Well it seems to be fixed for now. I will open a new Issue in case it will return, but for now the lag is fixed.

[dj3520]

I will open a new Issue in case it will return

Why not just re-open this one?

[LokeYourLord]

Cause i can't? I didn't close it.

[dj3520]

Then keep this one open. No point in making a new report if the problem is
not solved.
On Mar 8, 2016 12:50 PM, "LokeYourLord" [email protected] wrote:

Cause i can't? I didn't close it.

—
Reply to this email directly or view it on GitHub
https://github.com/micdoodle8/Galacticraft/issues/2129#issuecomment-193888449
.

[kernelstubbs]

...I know this is closed, but try removing your Launch Controller...or Chunk Loading your base (ensuring Oxygen Network is all in) This worked for my issue...which is very similar #2134 @Yanniclord, I know you've been on that thread...but for others coming across this issue...

[EzerArch]

Indeed there is something wrong with the SS dim. I notice the tick time (/forge tps) increases 1000 ms every minute until I start to get block lag.

Screenshot 2016-03-10_00 53 33
Screenshot 2016-03-10_00 56 31
Screenshot 2016-03-10_01 06 22
Screenshot 2016-03-10_01 11 04
Screenshot 2016-03-10_01 19 26(that was a falling sand I had thrown to the sky with Gravity Gun, it took almost 1 minute to fall back)

Then I went to the space station to see what was happening. The 1st thing I noticed was, the oxygen sealer in the arrival room was about to run out of oxygen, the oxygen in its internal storage was declining.

Screenshot 2016-03-10_01 27 37
Screenshot 2016-03-10_01 27 48

Went outside, the oxygen sealer was correctly connected, directly to an oxygen storage with an infinite oxygen.
Screenshot 2016-03-10_01 28 35

Then I checked the other sealers past that problematic sealer, they all were working.
Screenshot 2016-03-10_01 29 06
Screenshot 2016-03-10_01 29 15
Screenshot 2016-03-10_01 40 17
The sealer in the arrival room was then totally out of oxygen, not of electricity, despite the oxygen was flowing though the sealable oxygen pipe next to it. Using a wrench to rotate the sealer 360 degrees fixed the problem.

@radfast, if you want, I could send you my world folder with my mods to test it.

EDIT: Changed IMG to LINK to shorten loading time.

[dj3520]

Have you tried KCauldron 1492.155? I use it without issues on both a
windows and a linux server.

[LokeYourLord]

Thats way too old. In build 15xx, they introduced a new fluid system, which if i revert to before that would result in a catastrophic fail.

[dj3520]

Couldn't hurt to try it after a backup though right? Might be better than picking between two different evils. I honestly use the old version to avoid more issues.

[dj3520]

Well since Galacticraft doesn't lag with the newest forge version, it's not exactly our problem. As for AE2 you'll have to take it up with those guys. We're not tech support for forge.

[dj3520]

If you're going to post a lot make sure it's all one message. Each one sends a new email to everyone subscribed.

[LokeYourLord]

Ok guys, these are my recent TPS tests. I am using KCauldron 1614.175, and the lag does exist in there. It does not exist in build 1614.185, but that build causes AE2 systems and other stuff to break, and also isn't it Galacticraft itself that should be fixed, and not with a different forge version? Anyways, here my TPS logs:

[ghost]

Im getting really bad block lag on my space station and i just started building there?

[ghost]

well we were on the topic of the latest version, thats what im using? and using build 46 of Thermos

[radfast]

@Yanniclord you're driving me crazy with your posts. You make too many assumptions that make it sound like you know what you are talking about, when you do not. Please just, simply, report your own issues, and describe your setup. If you report issues in a clear way, and they are reproducible, they will get fixed if you just be patient a few days.

@EzerArch

The sealer in the arrival room was then totally out of oxygen, not of electricity, despite the oxygen was flowing though the sealable oxygen pipe next to it. Using a wrench to rotate the sealer 360 degrees fixed the problem.

radfast if you want, I could send you my world folder with my mods to test it.

Yes please, that would be great. Can you comment on which chunks, containing the oxygen pipe network, are staying chunkloaded?

Please also test whether the same lag issue still occurs if you give every Oxygen Sealer its own infinite oxygen supply item.

[LokeYourLord]

@radfast Well what more do you want me to include then? I provided tons of TPS logs, no idea what more there would be. I have no crashlogs, as this is a TPS/lag issue, so nothing to provide (unless you clearly tell me to include some kind of logs). What exactly do you want me to do then?

[radfast]

@Yanniclord I want you to stop telling other Galacticraft users what to do and your theories for how to solve issues, when their issues may be unconnected with the issue you are experiencing. Just report your own issues, and everything will be good.

To everyone experiencing lag when away from dimensions:

It appears there is a lag issue connected with chunk loaders and oxygen or electricity networks. (Galacticraft Launch Controllers are 1x1 chunk loaders.) It will help me a lot if people with lag when they are away from a dimension can provide a screenshot of any chunk in that base which contains a chunk loader, showing what electricity / oxygen machinery is in the same chunk, and showing the chunk boundary.

[LokeYourLord]

Ok here you go, don't say i didn't give you enough screenshots this time. I have 2 chunkloaders (Railcraft Admin Anchors) in my entire base, and 1 Launch Controller. Also, beneath every Walkway, is a Oxygen Sealer. Images:

Generating Oxygen with Mekanism Electrolytic Seperator's.

[radfast]

I have 2 chunkloaders (Railcraft Admin Anchors) in my entire base

Do these cover every machinery-containing chunk in your base, or only some?

[LokeYourLord]

Only a few, they basically cover the Stargate room, and the AE2 system and the main machines below the AE2 system. The Draconic Evo. T7 Energy Cell is not covered, and neither is the entire Solar Evap. room. The Reactor, Farm and Laser room are also not/partially covered. If you want, i give you my ENTIRE world download, so you can play around with it if you like. I will upload it if you want to, just tell me. But you probably need all the mods, so if you want, i can put them in a folder which i upload along with the world.

[radfast]

That's likely the issue then. For now, you can fix it by adding chunk loaders so that all Galacticraft machinery is in loaded chunks. Having the machinery half in and half out of loaded chunks seems to be the cause of the lag.

"Machinery" = aluminium wire, oxygen pipes, and any machine or storage (from any mod) connected to them.

Obviously that's an issue we can look at finding a more permanent fix for, but it may be complex to fix and take time. In the meanwhile you can fix the lag issue in your world in the way I said.

[LokeYourLord]

@radfast This is what it looks like now, everything that was red, now is yellow. Everything that wasn't chunkloaded, now is. The entire Space Station is now chunkloaded. I will keep monitoring the TPS, and i hope you are right (which you very, very likely are :P ). Anyways, here:

And ignore this stupid question if you wan't to, but can you just explain me one thing? So, why is it that chunkloading in this case fixes the lag, while servers in general BAN chunkloaders because of the lag they create by keeping all machines/bases chunkloaded?. (i restrict the usage on my server, but i did not disable/ban them).

[radfast]

servers in general BAN chunkloaders

You have a server for only one player (or maybe you + your friends, I don't know). The chunkloaders you place are not loading more chunks than if a player is in the dimension - if the player is in the dimension, all chunks in a 17x17 grid around the player will be loaded. It's no problem for the server to keep the chunks in your space station loaded, that's like 1 player standing there. And all servers should be able to handle easily 50 players online together.

Now imagine a server with 10000 active players every week, each player logs on for some minutes each day to check his base. If all those players had chunk loaders keeping their base loaded when they are offline, it would be like 10000 players online together, which would crash the server.

[radfast]

Build 447 may have improved things.

Update: Build 448 replaces build 447.

[radfast]

Update: the issue is caused by the change we made to Galacticraft to add redstone control. The issue has been present in all versions since 3.0.12.432.

It's not specific to Oxygen Sealers or any other machine.

_Every_ Galacticraft machine checks (through vanilla Minecraft methods) to see whether it is receiving a redstone signal. The vanilla Minecraft redstone check is causing the unwanted chunk loading, which results in lag. Currently working on a fix...

Vanilla Minecraft blocks which do redstone checks - for example doors, pistons - will still cause this type of lag if you have them on chunk boundaries. But we will fix it so that Galacticraft machines do not cause this problem.

The lag on chunk load/unload is caused by all mods put together, it's not specifically Galacticraft - if you have 100 mods, there is always going to be some time delay when chunks load and unload.

[radfast]

Build 449 fixes the lag, or at least makes a huge improvement from our side. Other blocks (including vanilla pistons etc) can still cause similar lag.

[EzerArch]

Disable redstone control for Oxygen Sealers and Collectors, fix lag.

Does this mean all the other GC machines (still) have redstone control? If so, is this necessary? if redstone checks can cause lag, assuming "it does" for the 1st question, wouldn't it be better to restrict it to a small range of machines?

[dj3520]

Does this mean all the other GC machines (still) have redstone control?

I think it's safe to assume this, unless changed in a different commit.

If so, is this necessary?

I'd imagine not. IMO anything that would stop by "completing it's job" doesn't need redstone control. This of course applies to things like the furnaces, ingot compressors, and coal generators and the sort, but also to oxygen collectors, compressors, decompressors, gas liquifiers, etc that already (should) stop once their internal supply is full. However this should not apply to machines that can't complete their "job" such as bubble distributors, sealers, and spin thrusters. For example I'm planning to expand my space station and do not have an air lock between the section I'll be expanding and my sealers, so I expect to loose the seal. With redstone control I could toggle between using sealers or oxygen bubbles, or turn them both off if my oxygen is too low to run them constantly.

[EzerArch]

I stayed for a while in the space station, the game became unplayable and no longer recognized any keyboard input.

Screenshot 2016-04-11_02 39 13

Looks like now being absent from the overworld makes the game laggy.

EDIT:

It got worse: Screenshot 2016-04-11_02 42 04

[Dream-Master]

Yes we have this on my Server too. Ends up no one have any space station more.

[EzerArch]

There's something I might want to investigate: I notice some kind of lag that builds up when I'm standing idle for a few minutes.

I'm playing normally and then I alt-tab from the game to see something else. After I'm back in the game, I can barely move, open the chat etc. So I resort to shutting down the game.

[radfast]

@EzerArch and @Dream-Master thank you for continuing to report. The lag you describe is obviously unacceptable. I will investigate further.

@EzerArch - is there any chance you can zip up your /mods, /config and DIM2 folder for me please?

[LokeYourLord]

Yep, the issue @EzerArch and @Dream-Master are having, i am also still having it. I currently have to forbid players having Space Stations due to this. The images @EzerArch sent are pretty much what i have too, but it's variable. At one point, being absent from the Overworld causes lag, at the other point being absent from the Space Station causes lag. Either one, lag will appear. I had to delete the entire DIMx_SPACESTATION folders just to clear the server from lag. I still don't like to guess, but it must have something to do with the Oxygen Sealers, since when my two players started making their Space Station, the lag first started to appear when it became big enough and they were using Oxygen Sealers . Other than that, @EzerArch , when it is lagging, can you do me a favor? IF you are using KCauldron or any commands that let you nuke entities, can you try /killall all or /butcher ? If you do not have that, try downloading a mod called Entity Lagfix (http://www.minecraftforum.net/forums/mapping-and-modding/minecraft-mods/2161485-1-6-4-1-7-2-1-7-10-1-8-forge-entity-lagfix-2-0-2)? Then do /nukeents 100000 or so, and see if the lag clears. Also, try writing down the entity count. I badly suspect that being a thing, because when i had to clear lag, i killed like 4000 entities at once, which was impossible as the Space Stations had no mob spawners or farms or anything that could overproduce any kind of entites.

[radfast]

If you can let us know what those 4000 entities are, we may be able to do something about this.

I need to reproduce these issues on my own PC to test and fix them (if Galacticraft is the cause at all). By the sound of things, these are large modpacks, and complex builds, so to make any meaningful progress here: _please can someone who has a world with this issue, zip up into a zip file everything I need to reproduce the issue (mods, configs, and the dimension folder) and upload it somewhere for me to take a look at_.

[LokeYourLord]

@radfast Ye i can go and make a new Space Station, but it will take a little while because as i said, i wiped all existing Space Stations. And i actually tried to check what those 4000 entities are, but OPIS does not work on my server anymore, it crashes with many other mods as it is so heavily outdated, so i added a mod that lets me do /profile e 60 (for example), and track the lag in all dimensions. But when the 4000 entities were wiped and i tried to track their names, i had nothing. I will try my absolute best to see if i can find their names and if i can make a new space station and zip it all up for you.

And yes, the issue persists for me on my large modpack, and so most likely does for the others too.

[EzerArch]

I was trying to reproduce the lag, I got 130.000 ms with GalaxySpace installed, while on Pluto. Now I'm going to remove GS and see what happens.

[EzerArch]

For some uncertain reason I'm no longer getting TPS drops. Now it is around 5.0 and 8.0 ms.

What I have changed so far:

• Used /nukeents, it barely killed about 40 in the SS dim.
• Updated Galacticraft (from 460 to 462)
• Updated Optifine (now OptiFine_1.7.10_HD_U_D6)
• Removed GalaxySpace addon
• Removed Dynamic Lights mod
• Pruned my overworld with MCEdit

Now about entities building up in the space station/OW dimension, there's a glitch that I like to exploit which can be the culprit: drops can literally "fall" from space station dimension towards the Overworld if the corresponding chunks between dimensions are loaded.

• Say your SS is built around the x=0, z=0, keep the chunks around that point (0.0, 0.-1, -1.0 and -1.-1) loaded.
• Now on the Overworld, keep the same chunks (0.0, 0.-1, -1.0 and -1.-1) "beneath" the space station loaded.
• Now in the Space Station, drop some items straight down to Earth. The drops will respawn in the same coords on the Overworld at Y=32.
• Proof: https://youtu.be/wemEk8PhvMI?t=33m14s

EDIT:
I'll zip my current world and a backup and sent them to @radfast.

[dj3520]

A bit curious as to why I'm not seeing this happen on my server. Are there any reports of this occurring without GalaxySpace installed?

[radfast]

The "dropped items ending up on the Overworld at 0,0" thing is not a glitch, it's intended behaviour, has been in Galacticraft a long time. The aim is to try to reduce item loss, we don't want to make people lose all their items if they have some disaster in space like breaking a chest. It's also quirky / funny to have items end up on the Overworld instead of e.g. burning up in the atmosphere or lost in space.

I don't see how it can make lag. The dropped items are the same as any other dropped items in Minecraft. Any server load they create (which is miniscule, for a small number of dropped items, like <100 items) will be load on the Overworld update, not the space station update.

[EzerArch]

The "dropped items ending up on the Overworld at 0,0" thing is not a glitch, it's intended behaviour

Good news, I like this Easter-egg. I was afraid that could be a glitch because one day I found some drops around the OW's 0,0, in the middle of the ocean, and I guessed they could be falling from the SS, then I built this platform:

Screenshot 2016-07-03_17 24 38

But I couldn't find any drops as they were spawning on the ocean floor, beneath the platform, near y=32 (which is also the height at which you teleport from SS dim to OW). Then I reached two conclusions:

1. the items wouldn't be falling from the sky (how could they by-pass a solid platform?) but teleporting directly from a X-Y-Z-point of a DIM to the same point values of another DIM. Then I built an underwater shelter to collect items. Screenshot 2016-07-03_17 51 21

1. that items might be getting stuck "between" dimensions but some of them were spawning on overworld's loaded chunks.

If this is an intended behavior, wouldn't it be better if those drops spawned at y=128 or at the highest solid block instead of y=32?

[radfast]

Thanks, I'd better check the height they spawn at. In Galacticraft 1 they spawned at a sensible height, so you could find items scattered at y=65 for example.

[MJRLegends]

@radfast @micdoodle8 I have had a read of what has been said here and i have tried adding XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:-UseAdaptiveSizePolicy to my server start args, it helped a little.

When adding like 25 oxygen sealers, my ms tick for my space station went from like 2ms to like 7/8/9ms.

Is there anything that could be done as i did have like 80 sealers as i have a big space station and that was up to like 20/30 ms for my space station?

The problem i have is i run a proper server and say 10 people had 7/8/9ms space stations and are all online at once thats will cause a big tps drop, my aim with my server is to have no chunks to be loaded that belong to a player when there off so im needing to make sure my tps stays at 20

[radfast]

@MJRLegends :- The tick times you are reporting sound _fairly_ normal, 7/8/9ms tick time for a large base is possible. A base with 25 oxygen sealers must also have ~ 25 rooms (or one very large volume) and all the machinery and power generation equipment to match, that's blocks from many mods.

You probably know this: if you enable debug logging in Galacticraft configs it will tell you the exact time taken by the Oxygen Sealers. And the Oxygen Sealers run their sealed check only around once per 50-100 ticks. So even if you see a 15ms seal check being reported by a sealer, that is less than 0.25ms per tick on average. I agree that's still a lot for one block - if every block from every mod did that, there would be a problem - but the Oxygen Sealers are special, they have to do a complex task.

Having said that, a big goal for me is to support the people who run - as you say :) - proper servers. I think about performance in Galacticraft a lot - have a look at the number of commits marked "Performance". I have an idea that can make Oxygen Sealers sealed checks maybe 2x as fast as what they are now, but it involves some advanced coding techniques (create our own HashSet function, explained more in discussion on #2117) and will take some time, no promises for if / when it will happen.

I think the tick times you have reported are not the 'huge lags' issue which other people are seeing - which I still can't reproduce or find the cause of.

---

Like the OP in this issue report, a few people are reporting massive tps drops, on small or medium-sized space bases. Especially after the game is left running a long time. I really want to find and fix this issue, but I have not been able to see it for myself - still hoping that someone who has it will zip up their entire server, mods, configs, plugins and the dimension folder where the issue is, and send it to me for testing. Or else give me Opis output which will maybe show me the cause.

Possible causes I am thinking about:

(1) something connected with Launch Controllers and chunk loading, maybe even a bug in some version of Forge?

(2) something specific to Thermos / Cauldron servers or a plugin

(3) different Java garbage collection settings [I recommend giving Java a fairly low amount of memory, like 2MB or less, forcing it to run its garbage collection more frequently but with smaller lag spikes]

(4) a bug in another mod causing Galacticraft tile entities to multiply

Likely related issues:

2385

2249

2238

2228

2216 maybe

1803

Anyone describing this issue in future: please provide screenshots of your machinery, especially machinery / wiring used to power Launch Controllers, Oxygen Sealers, Oxygen Collectors and any other machines which are far from your main base. It is very possible that the cause of the issue is bad chunkloading strategies in electrical wiring or other conduits provided by another mod: Mekanism Thermal Expansion Ender IO IC2 etc. Unlike some others, Galacticraft's electrical system is well-designed to minimise these types of problems.

[radfast]

This report by @themechman from #2228 has useful info. Note that he seems to be saying that Mekanism v9 cables are causing the issue, and the sealers connected with Galacticraft's own heavy aluminum cables do not have the issue

After some digging around with opis (which I forgot I even had), I found three sealers outside the chunkloaded area. These are connected with Mekanism v9 cables vs another base which uses your heavy aluminum cables and has only one chunk loaded. The tick load on dim -30 is higher than overworld's.

After increasing the chunkload radius from 9 to 25 chunks, breaking the three sealers, putting them back and restarting the server, the RAM usage was no longer getting gobbled up after 15 minutes of operation.

[radfast]

It would also be a good idea for all big servers to upgrade to Galacticraft 3.0.12.474. That makes Oxygen Sealers less likely to "flip" a lot from sealed to unsealed and back again, and discourages players from making bad sealer builds.

[radfast]

Here is what I wrote on #2134:

My best guess is the issue is caused by chunks loading and unloading over and over, probably because (with the Launch Controller) you have only 1 chunk which is being chunkloaded, which has an electricity or oxygen source, and it therefore sometimes loads the neighbouring chunk when it is pumping electricity or oxygen to pipes or machines in that chunk.

and KernelStubbs there wrote:

I configured the Chicken Chunk loaders to fully encompass all cabling, machines and contents (sealed areas) of my bases - but I've removed them and as I mentioned above, the issue is gone.

[radfast]

Anyone affected by this or similar issues:

My current theory is it's related to this combination of factors:

• players leaving and returning to the dimension in rockets or other ways (teleport, nether portals, etc)
• the dimension has a chunkloader (GC Launch Controller, Railcraft Anchor, ChickenChunks Chunk Loader, Witchery shelf etc)
• when all players leave the dimension, the chunks _outside_ the chunkloaded area are _supposed_ to be unloaded
• the player has built machinery / wires / pipes / itemducts which crosses the boundary from the loaded chunks to the chunks which are _supposed_ to be unloaded
• another mod - most likely a wiring / pipes / itemducts mod, so look at Mekanism-EnderIO-Thermal Expansion-IC2 etc... - wrongly reloads, over and over, those chunks which are supposed to be unloaded
• tile entities somehow get duplicated in the chunks which are being force loaded and unloaded over and over

This explains why the issue only happens / or gets worse when the player leaves the dimension.

---

I still need help from you guys so that I can actually see this issue happening on my test systems. Especially, I need to know the smallest combination of mods and build which can make this occur.

Anyone who sees this issue in future, please:

1. Provide a full mod list, highlighting wire / cable / pipe / itemduct mods
2. Provide screenshots of the base which is lagging, with chunk boundaries shown
3. Especially in your screenshots show any chunkloaded chunks, and the connections they have with the chunks next to them
4. Try to force the problem to occur, by all players taking a rocket or teleporting to another dimension, and leaving the base unoccupied (but still loaded by the chunkloader)

[dj3520]

@radfast I seem to have joined the party of laggers. Details, of course, are at #2421
Will try to update to 480 soon. Until then I'm stuck in space for the server's sake.

[dj3520]

@radfast Opis reads that the entirety of my base is "game loaded" though I'm not sure if that's true. I believe it's thinking that an area around 0, 0 is automatically chunkloaded by the game because of vanilla behavior, but I'm not sure if space stations do the same. If this were true then technically I wouldn't need chunkloaders so long as the dimension stays loaded.

Blue chunks are force loaded by chunkloader. Green are "game loaded." Red outline is basic area of my base.

EDIT: Installed tickdynamic mod on server. Issue is much less of an issue now.

[SharkBaitDLS]

If you're still looking for a server configuration where this is reproducible, I've got two here to start from. One with an older version of Galacticraft where the issue is not present, and one with a newer version where it is. Otherwise same world, same mods.

The overworld chunk loader needs to be broken and re-placed every time the server starts before it starts actually loading things properly, but after that everything should be fully working.

The space station in question is Dimension 2, and none of it is being manually chunk loaded but some background thread periodically loads/unloads the dimension when nobody's present. I haven't tracked down that root cause but regardless whatever triggers it to briefly load is enough to then cause the TPS bug to surface with the newer GC version.

[MJRLegends]

This issue has been fixed :D @SharkBaitDLS

[radfast]

@MJRLegends your issue might have been fixed by the Oxygen Sealer performance increase but I think the original report in this thread is about a different issue which I've been wanting to lock down for a long time (which is why this issue was left open)

@SharkBaitDLS thank you that seems super helpful, especially as you've identified a change in Galacticraft versions as the cause, that should help us to track it down... I'll take a look at these worlds, and post here if I have any questions.

EDITED ok questions already:

(1) do I really need both downloads or can I just get one and swap Galacticraft versions over?

(2) it is possible for you to reduce the size of the download to something less than 3.5GB, otherwise no-one in my house will be able to watch video for the next few hours! I probably don't need dimensions apart from the space station and the overworld, I'm assuming the overworld is where you go to trigger the "background thread" you mentioned. And even on the overworld, we probably only need spawn and the region the player is going to be in (and the immediately adjacent regions) when the player goes there from the space station. (If you're telling me I need 20 players online to trigger this, that's something I can't test ...)

EDITED 2: any chance of a binary search to identify the exact Galacticraft version update which triggered this? I've realised I won't have much time for Galacticraft development this week due to some other obligations :(

[dj3520]

@radfast Need a tester?

[radfast]

yes please, @dj3520, if you have time to have a look that would be very much appreciated - especially if a binary search technique can tell us which exact Galacticraft version update led to this - I've now downloaded one of the two world downloads which @SharkBaitDLS has kindly offered us, but again I'll need to wait until later to grab that other one, otherwise I'm going to annoy the other people in my house and their Amazon Prime / Netflix / BBC viewing schedules...

[dj3520]

@radfast I live alone so this isn't a problem for me. Additionally this is one bug I'd like to see squashed before 1.7 is left alone.

[radfast]

great, thank you. I can see that the "no tps issue" world is using Galacticraft 378 (wow that's old) and the "tps issue" world is using Galacticaft 497.

It would be good now to take the next steps:
1) identify a sequence of steps that can reproducibly cause the TPS issue to start (if the issue is present) for consistent testing
2) reduce the number of other mods in the modlist, but still have the TPS issue occur - this will reduce loading time and start to narrow down possible causes
3) use binary search techniques to find the Galacticraft version between 378 and 497 causing this...

Any of these you can help with would be really appreciated: but the first needs to be number 1 in the list.

[LokeYourLord]

I can't believe that after all this time this issue still exists, and the many times this issue has been closed and reopened, there are still posts being made here XD Never mind me though...

Also, apparently this was more than 1 year ago. Time for the anniversary celebrations :D

[dj3520]

he "no tps issue" world is using Galacticraft 378 (wow that's old) and the "tps issue" world is using Galacticaft 497

497 - 378 = 119 possible builds where it all went wrong. Got my work cut out for me, haha.

[radfast]

119 < 2 ^ 7

@LokeYourLord we tend to keep issues open if they're serious, until a proper fix is found. This one has been specially hard to fix because it's not clear how to reproduce it, and it's likely caused by an interaction with one or more other mods. I won't pretend we've been working full time on only this issue for 12 months, though :p Thanks to @SharkBaitDLS and @dj3520 I feel we may finally be getting towards an answer.

[SharkBaitDLS]

I think I already started doing a little binary searching myself on versions. I want to say I was able to see the issue as early as .429 -- also to confirm. The two worlds are identical just with Galacticraft swapped out. I can report back later tonight on what I was finding.

[radfast]

Thank you that would be great.

What steps should I take to see the issue? Please bear in mind that I'll log in as 'radfast' and I won't have any /home or base set up when I first log in. I can op myself so that I can get to Dimension 2. Do I literally just go there and then go back to the Overworld or do I need to go to specific coordinates on the Overworld?

If you know a bit about Java - or even if you don't but you're good generally, as I guess you are (Amazon?) - you can also use tools like the VisualVM (part of the JDK) or Opis or WarmRoast to profile the server while the TPS issue is happening, and identify which precise methods are taking up disproportionate CPU time.

[dj3520]

After disabling FTB*.jars (my test server didn't like them) and keeping both server and client version in sync, I've used the no-tps-bug.zip and changed GC versions manually to generate this report on the matter.

Confirmed 378 does not have an issue.
Tested 438, no issue.
Tested 468, after leaving space station tps 1~2
Tested 453, persists
Tested 446, persists
Tested 442, persists
Tested 439, persists (huh)
Tested 438 again, persists (WHAT?)
Attempt auto rollback, [21:23:48] [Client thread/ERROR] [FML]: There was a critical exception handling a packet on channel GalacticraftCore java.lang.NullPointerException on space station visit. Snip of full error here
Delete and re-upload world (still 438) Same issue.
Nukes server. Re-uploads. Same error. Gotta love old versions.

[SharkBaitDLS]

I'll test on my setup as well starting on 438 to see if I can reliably reproduce when it changed. Once I do that I'll start profiling on the first known version that produces this behavior.

@radfast in terms of what you need to do on logging in, go to the coordinates in the Overworld from my screenshot and break/replace that chunk loader and set it to radius 2. In my experience whenever I start the server the chunk loader doesn't work properly unless I do that. With that loader active in the Overworld it'll ensure that the space station has power supplied to all the oxygen machines on the space station. Other than that, just head on up to dimension 2. I've been leaving the station not by rocket but by jumping back to the Overworld. Maybe worth doing to reproduce, maybe not.

[SharkBaitDLS]

@dj3520 perhaps your results are explained by the fact that station isn't chunk loaded, but the dimension loads once every 90 seconds (by some mechanism I do not yet understand) and then stays loaded for about ~45 seconds. So it can take up to 90 seconds after leaving the station to see the issue start happening. May be why 438 initially appeared to be fine but then subsequently was producing the behavior.

[SharkBaitDLS]

Confirmed the presence of the issue in 438 from my end. Continuing search.

[dj3520]

@SharkBaitDLS I see. Thank you I forgot about the chunkloader. However for right now the crash I posted is making testing this a bit of a pain. I may try with just Galacticraft in the hopes that nothing else terrible goes wrong with my setup and methodology.

[SharkBaitDLS]

Yeah, not sure how to help there. My only idea would be to try to run it as a complete package using the JVM command in StartServer.sh rather than trying to run it in your own server instance.

In the meantime, I'll continue an approximation of binary searching through versions.

[dj3520]

Issue is I'm not using the FTB jar and my test server can only use 2GB of ram

[SharkBaitDLS]

Well, hopefully I can track down the version diff that introduced the bug and then we can scope down to a simpler test setup from there.

[SharkBaitDLS]

Ok, I think I've found the change that introduced the bug. It was from 410->411.

Unfortunately, 411 also introduced a bug that causes my space station to throw an NPE in the Oxygen Sealer logic:

 [00:22:31] [Server thread/ERROR]: Encountered an unexpected exception
net.minecraft.util.ReportedException: Ticking block entity
        at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:642) ~[MinecraftServer.class:?]
        at net.minecraft.server.dedicated.DedicatedServer.func_71190_q(DedicatedServer.java:334) ~[lt.class:?]
        at net.minecraft.server.MinecraftServer.func_71217_p(MinecraftServer.java:547) ~[MinecraftServer.class:?]
        at net.minecraft.server.MinecraftServer.run(MinecraftServer.java:396) [MinecraftServer.class:?]
        at net.minecraft.server.MinecraftServer$2.run(MinecraftServer.java:685) [li.class:?]
Caused by: java.lang.NullPointerException
        at micdoodle8.mods.galacticraft.core.oxygen.ThreadFindSeal.check(ThreadFindSeal.java:359) ~[ThreadFindSeal.class:?]
        at micdoodle8.mods.galacticraft.core.oxygen.ThreadFindSeal.<init>(ThreadFindSeal.java:92) ~[ThreadFindSeal.class:?]
        at micdoodle8.mods.galacticraft.core.oxygen.ThreadFindSeal.<init>(ThreadFindSeal.java:47) ~[ThreadFindSeal.class:?]
        at micdoodle8.mods.galacticraft.core.oxygen.OxygenPressureProtocol.updateSealerStatus(OxygenPressureProtocol.java:64) ~[OxygenPressureProtocol.class:?]
        at micdoodle8.mods.galacticraft.core.tile.TileEntityOxygenSealer.func_145845_h(TileEntityOxygenSealer.java:163) ~[TileEntityOxygenSealer.class:?]
        at net.minecraft.world.World.func_72939_s(World.java:1939) ~[ahb.class:?]
        at net.minecraft.world.WorldServer.func_72939_s(WorldServer.java:489) ~[mt.class:?]
        at net.minecraft.server.MinecraftServer.func_71190_q(MinecraftServer.java:636) ~[MinecraftServer.class:?]
        ... 4 more

That bug was fixed in version 419, so that's the earliest version that the bug is present and the world is fully usable. It may be possible to reproduce the bug with a simpler test bed on version 411 that doesn't cause the crash.

[SharkBaitDLS]

Initial thread profile results of when the bug is active. Interestingly it looks like immersive engineering's redstone wires are taking a big chunk of the load.

The more interesting thing to me is that most of the other entities appear to be stuck in the onChunkUnload method.

I'm going to try stripping all the immersive engineering blocks out of the station and then re-profile.

[SharkBaitDLS]

Next profile. Looks like impact is split evenly with AE2 stuff now. Going to strip that next.

[dj3520]

I'd make a new server and just do Galacticraft

[SharkBaitDLS]

The challenge is then figuring out how to get a space station that reproduces the issue with just Galacticraft. I'm going to cull all the spatial pylons for now because I suspect AE2's quantum bridge is what's causing the dimension to load at all in the first place.

[dj3520]

Issue should persist with just Galacticraft if it's a Galacticraft issue.
Btw each message here sends an email to anyone who follows the repo, so I'd suggest not using this like a chat room.

[SharkBaitDLS]

Apologies. Is there a more appropriate location to discuss root cause investigation? GitHub is not something I use regularly.

[dj3520]

Unfortunately I don't think there's anywhere else besides the forum that serves as an official "chat" for Galacticraft. Times like these (live debugging, website outages, small issues) are when discord servers come in handy, but Galacticraft doesn't have one sadly.

[SharkBaitDLS]

Well I need to call it a night on this anyway, if we want to try to convene for more in-depth testing I'm happy to try to coordinate and make my server available if helpful.

Last thread profile for the night.

[radfast]

Hey guys

First off thank you hugely for this and the time involved, seems like serious strides forward

I think this is the best place to discuss the issue, as it allows for a 3 or 4-way conversation and anyone else who has the same issue can also jump in. Anyone who doesn't want emails can just unsubscribe from this or from Github emails generally.

In the profiles I can see some long tick times from tiles in other mods, mainly AE2 cable bus and Spatial Pylons, also Immersive Engineering Redstone Wire connector. There are also some ticks with long updates for Galacticraft Oxygen Sealer (but that's something we have fixed more recently) and Oxygen Collector (but that should only be slow something like 1 tick out of 100).

More fundamentally, like you say, there seem to be a lot of onChunkUnload() calls, suggesting that the server is over and over loading a chunk and then unloading it soon afterwards.

In builds 411 and 413 we made changes to _reduce_ the amount of forced dimension loading from Galacticraft - for example this - so it's possible that something we were doing in builds before then was masking this issue, by keeping things loaded.

Minecraft server chunk loading involves some alchemy, but broadly the principles are like this:

1. a server always keeps loaded the Overworld spawn chunks, and chunks in a radius around each online player
2. many mods cause neighbor chunk loading to occur when updating tiles - for example, Thermal Dynamics or another mod with wiring might need to check the tile for every connected wire in a grid, when some electricity goes into the grid, and checking the tiles will cause all chunks containing those tiles to be loaded whenever a check occurs (which can be each tick) - and the same for redstone wiring and similar, including vanilla redstone
3. even simpler types of block checks a mod can do, like scanning a 3x3x3 cube of blocks to see if it makes a complete structure, will cause chunk loading, my guess is the AE2 Spatial Pylon is doing something of this kind - a lot of mods are coded to check surrounding blocks over and over each tick, basically because blocks in Minecraft can change
4. the vanilla server maintains a pool of loaded chunks, and each tick the server will pick some of those loaded chunks to unload - the algorithm aims to pick the chunks which were not updated recently and the chunks which are furthest from players
5. Thermos/Bukkit, and Bukkit plugins, and other server-side mods, can change the algorithm for chunk unloading, generally making it unload chunks more aggressively
6. Chunk loaders will re-load chunks each tick (before the tick starts) if they got unloaded, though in some cases the chunk loader is able to keep the chunk from being unloaded at step 4 (I think, but not completely sure about this).

Importantly, if a tile in a chunk-loaded chunk is positioned near the chunk boundary and makes a neighbor block or tile check - see (2) and (3) above - then it can cause neighboring chunks to be loaded even though the chunkloader isn't set to load them. I believe that those neighboring chunks will be quickly unloaded - see (4) above - maybe even next tick, if there is no player nearby.

This neighbor chunk loading is obviously affected by how players build - for example if everything was built in a 14x14 square inside the chunk boundaries then there would probably be no issue - but most players don't even think about chunk boundaries in their machinery and wiring builds.

In summary, a lot of mods - the majority, probably - are not coded with Minecraft chunkloading and performance in mind. Each mod does its own thing, if the chunk gets loaded then each mod's tiles will activate and do their checks, so loading neighboring chunks, recursively. (Probably for some mods if you built a grid of wires covering a huge area like 10000 x 10000 blocks then all those chunks would get loaded each tick.)

Much of the time this does not matter, because players tend to build bases which are only maybe 8 x 8 chunks in dimension, and players tend to stay inside their bases, so the chunks would be loaded anyhow because they are in the server's chunkloading radius around the player.

But if you start to mix in a chunk loader forcing certain chunks to stay loaded, but not their neighbors, and also player absence from the base (e.g. because the player leaves his Space Station to go to the Overworld) then problems of the kind described here can occur.

Which is a long way of saying, maybe this is not Galacticraft's fault at all.

Especially because unlike most other mods, Galacticraft is careful not to cause neighbor chunk loading unless necessary - so Galacticraft Aluminium Wire, for example, is higher performance than wires from other mods, it might be possible even to fix some of these issues just by replacing your cryogenic flux ducts with Heavy Aluminium Wire.

---

Further testing

I'm still a bit confused about whether it's the space station base or the Overworld base which is causing the slow tile updates and .onChunkUnload()

Some checks I would be interested in seeing at this stage:

1. Do the issues occur if the chunk loader is set to a larger radius so that all built chunks in the Overworld base are kept loaded all the time

2. Do the issues also occur when the player leaves the space station base to travel to other places in the same dimension, like travelling 1000 blocks away in the same dimension?

3. Is it possible to reproduce the issue by constructing a similar base - with chunk loader, AE2 wiring, spatial pylons, Immersive Engineering redstone etc - make it a similar size at least, doesn't matter so much whether it really makes sense - which is purely on the Overworld, and the player teleports away from that base e.g. to his other base, so that the server can unload the chunks. (Or maybe you can get all fancy here and use the Spatial Pylons to copy the whole spacestation base to the Overworld?)

If the issue still occurs when the base is on the Overworld, that will show this probably isn't a Galacticraft issue, but a vanilla / Forge chunkloading issue or just poor practices in some of these other mods, and that - as I currently suspect - the issue was probably being masked by Galacticraft taking a heavy handed approach to dimension loading up to build 410.

I will confess that I didn't yet succeed to get SharkBaitDLS' world to load properly for me, so I am kind of relying on you guys to do the tests for now. Thanks for the suggestions @dj3520 about deleting the FTB jars, I will try that.

[radfast]

station isn't chunk loaded, but the dimension loads once every 90 seconds (by some mechanism I do not yet understand) and then stays loaded for about ~45 seconds. So it can take up to 90 seconds after leaving the station to see the issue start happening.

This is interesting. 90 seconds is a long interval, that's not anything Galacticraft is doing. My guess is this is most likely something in AE2 but I really can't be sure.

[themechman]

Discord seems to be the newest thing and a lot of other modders are using it for collaboration.

While this is more an AE team issue here, it may be relevant to your testing.

I can definitely say without hesitation that AE2's Quantum bridge does some crazy chunk loading.

If you need to test with AE2 being present, try these suggestions:

Connect a few of AE2's dense power cells against the quantum ring, and connect with a cable for good measure. This should provide enough power buffer for dim changes and keeping the that part of the ME network alive long enough for the server to recalculate the network.

Next observation is that you must keep the ME network 100% loaded. If you're not using the dim, yank the singularity orb. Disconnect any section not used. Like you radfast, I found out that my spatial pylon network did increase the server load because only a few blocks were in an adjoining chunk that was not loaded. It wasnt until I chunk loaded the whole thing that the load decreased.

Dont try to use an energy acceptor and creative source (Mek Cube or TE Cell) to power remote quantum bridge rings. This will unload the power feeds and potentially cause insane problems on the dim where the linked ME controller is placed. (Use first suggestion or creative power block from AE).

[SharkBaitDLS]

I can definitely confirm it's the space station not the Overworld base that's ticking heavily. The issue is present when I'm standing in the Overworld base and it's fully loaded. The theory that Galacticraft stopped fully loading the chunks aggressively which subsequently causes problems when the quantum bridge loads a subset of the chunks makes sense to me -- I tried for some time to find documentation on the quantum bridge indicating what, if any, chunk loading it does but came up dry, but if that is what's it's doing that seems to make sense for me. Things I will try tonight:

• Flying a far distance from the station while staying in the dimension to see what does or doesn't load
• Completely removing all the AE2 cabling and just leaving the quantum bridge at the center of the station alone

I think you're on the right track with that, but if it doesn't work then I'll also try copying the whole base into the Overworld.

[radfast]

Your time spent on this is very much appreciated, and if it shows something we can fix in 1.7.10 then obviously we'll make that fix.

When I get some time (tonight I hope) I'm going to start at the other end, trying a reduced mod list to see if I can reproduce the issue in your world with a smaller number of mods. Might need some rebuilding I guess to fill in the sealed area, but I'll see.

@themechman I read your post with interest, sounds like there's a lot AE2 could do better in chunkloading terms. Maybe they already improved things in 1.10.2+ versions. We have a library/API class BlockVec3 with methods for getting blocks and tile entities without causing chunk loading. Therefore by design our power network - for example - will simply not attempt to power blocks or sub-networks in unloaded chunks. Nobody has ever complained about this. I can see it could be harder for AE2 to achieve, because an ME system usually only works properly if everything is online.

[SharkBaitDLS]

I think the only non-GC mods that make up the Space station's sealed area are chiseled variants of cobblestone and EnderIO's glass.

[SharkBaitDLS]

I think your original assertion was correct @radfast -- removing all the AE2 equipment sans the quantum bridge has mitigated the issue. Now I just see a small CPU utilization bump when the dimension loads which to my understanding is to be expected as this version predates the optimizations for the oxygen network. No loss of TPS.

My guess is that explicitly chunk-loading the whole space station at all times is the correct long-term fix so that the AE2 cables don't go haywire.

My second guess is that I can reproduce this without needing a space station or even Galacticraft at all, just a quantum bridge and a network attached to it that spans a few chunks. That would validate the theory that Galacticraft was saving me from this issue due to all the machines loading their chunks.

[SharkBaitDLS]

Struggling to reproduce this in the Overworld with a quantum bridge a kilometer out from my base and a bunch of devices a few chunks out from the bridge. There must still be an additional factor to this I'm not including in the reproduction attempt. I still think that the chunk-load boundary theory holds merit.

[SharkBaitDLS]

Confirmed that chunk loading the whole space station mitigates the issue even on v497. If I have spare cycles this weekend I'll try to figure out the minimum reproduction case still.

[radfast]

Confirmed that chunk loading the whole space station mitigates the issue even on v497

OK this is good news. Seems like the "missing link" is what is causing chunk loading on the Space Station once every ~90 seconds. Maybe the AE2 Quantum Bridge functions differently if the other end of the bridge is in a different dimension?

From our side, I'm going to take a look at the Oxygen Collectors and anything else that we can do to reduce CPU load if there is a lot of chunk loading and unloading. Quite a few of the slower tile entities in Galacticraft have code that runs a check once every N ticks where N could be anything from 4 to 100. For some of these, the tick counter will be resetting to 0 if the chunk unloads and reloads, which is going to have an impact on performance because the code tests to see if (tick counter mod N) == 0.

There's nothing more I think we can do about AE2 and Immersive Engineering performance. Changing Galacticraft to keep all built space station chunks permanently loaded would not be the right solution to this.

[dj3520]

I have neither AE2 nor Immersive Engineering on my server and still notice this problem. I've also noticed that it seems right after the problematic chunks are forced loaded, the server doesn't skip ticks when trying to catch up, meaning an actual tick rate above 20TPS.