Mindustry: Game crashes in multiplayer with too many mods

Created on 22 Aug 2020 · 11Comments · Source: Anuken/Mindustry

Platform: Windows

Build: 104.10

Issue: It looks like that when a critical mass of items added by mods is reached, the game crashes when mod items first time start being transported by a conveyour and enter the Core or a Router. This only happens in multiplayer and sometimes it happens for both players, sometimes only for the joined ones.

Steps to reproduce:

For two players: install the critical mass of mods (see the mod list below for a possible example)
Create a sandbox game, host by one player and join by the other
Create an Item Source and make it produce any modded item. In our case we were mostly testing it with Voidore from the Opore Mod, but other modded items showed the similar behavior. The Voidore is the black thing produced by the Voidore Forge factory building, just in case.
Build a conveyor from that Item Source to nowhere, observe how the item starts flowing, everything is OK
Build the conveyor now to make it reach the Core or build a Router on it.
The joined player crashes (see Crash Report 2)
Try disabling some mods that add items and see that it's not a particular mod that's causing this, but the amount of mods (or maybe items added). For example, we tried the following tests:
a) only disable MicroAdd -- no error,
b) only disable Minfactory -- no error,
c) enable both MicroAdd and Minfactory -- the error is there.
d) only disable Heavy Industries, Heavy Armaments and Hypertech Mod -- no error

These tests clearly indicate that when a certain amount of modded items is added to the game, it crashes. The limit at which it starts crashing is about 140-150 items listed in the Item Source list.

Another variation of this problem is a bit different crash that happens in the same circumstances but the exception is different, and it happens for the host too (see Crash Report 1). Originally we have caught that kind of error on a map where we were already playing for about 2.5 hours and got a lot of infrastructure. Upon the attempts to reproduce that error (Crash Report 1) to report it, we stumbled upon the situation where only the joined player get the error now (Crash Report 2). It is yet unclear what circumstances determine which variation of the crash is going to happen but the variation 2 seems to be prevailing now.

Link to mod(s) used, if applicable:

Below is the list of the mods we were testing on. But it could be probably replicated with any set of mods that adds up to the similar amount of modded items:

Crash report:

Crash Report 1 (happened both on the host and on the client sides)

java.lang.NullPointerException
    at mindustry.world.modules.ItemModule.add(ItemModule.java:104)
    at mindustry.world.blocks.distribution.Conveyor.handleItem(Conveyor.java:298)
    at mindustry.world.blocks.distribution.Junction.update(Junction.java:64)
    at mindustry.entities.type.TileEntity.update(TileEntity.java:324)
    at mindustry.entities.EntityGroup.update(EntityGroup.java:51)
    at mindustry.core.Logic.update(Logic.java:241)
    at arc.ApplicationCore.update(ApplicationCore.java:36)
    at mindustry.ClientLauncher.update(ClientLauncher.java:138)
    at arc.backend.sdl.SdlApplication.listen(SdlApplication.java:158)
    at arc.backend.sdl.SdlApplication.loop(SdlApplication.java:146)
    at arc.backend.sdl.SdlApplication.<init>(SdlApplication.java:52)
    at mindustry.desktop.DesktopLauncher.main(DesktopLauncher.java:46)

Crash Report 2 (client-only crash)

java.lang.RuntimeException: Failed to to read remote method 'onStateSnapshot'!
    at mindustry.gen.RemoteReadClient.readPacket(RemoteReadClient.java:303)
    at mindustry.core.NetClient.lambda$new$4(NetClient.java:134)
    at mindustry.net.Net.handleClientReceived(Net.java:247)
    at mindustry.desktop.steam.SNet$1.update(SNet.java:83)
    at arc.backend.sdl.SdlApplication.listen(SdlApplication.java:158)
    at arc.backend.sdl.SdlApplication.loop(SdlApplication.java:146)
    at arc.backend.sdl.SdlApplication.<init>(SdlApplication.java:52)
    at mindustry.desktop.DesktopLauncher.main(DesktopLauncher.java:46)
Caused by: java.lang.NullPointerException
    at mindustry.world.modules.ItemModule.read(ItemModule.java:158)
    at mindustry.core.NetClient.onStateSnapshot(NetClient.java:452)
    at mindustry.gen.RemoteReadClient.readPacket(RemoteReadClient.java:301)
    ... 7 more

Place an X (no spaces) between the brackets to confirm that you have read the line below.

[X] I have searched the closed and open issues to make sure that this problem has not already been reported.

bug

Source

noncom

Most helpful comment

Thinking like this, I should never need to play games at all.

Sure, you don't need to play anything. You don't need a ridiculous amount of items for a good modded experience.

switch to an "extended protocol"

How would this switch be done in a clean way that would be backwards compatible? Items are written as bytes in several different places as binary, and these places have no way of knowing what the save 'version' is or which bytes were written. Sure, it's possible to rewrite IO and add a bunch of compatibility code, but I don't want that.

that means no fun with a lot of complex machinery and combinations

How is 127 (or, more accurately, 255, since unsigned bytes are used) items too little? The base game has only 16, and even with that number you can do plenty - in fact, some items are still underutilized in the base game.

Anuken on 23 Aug 2020

👍2

All 11 comments

well the mods may conflict or smth

imreallybadatnames on 22 Aug 2020

Something like this has already been reported. In the future, v6 will not allow mods with >127 items to be loaded.

Anuken on 22 Aug 2020

Something like this has already been reported. In the future, v6 will not allow mods with >127 items to be loaded.

@Anuken Oh, this is sad :( that means no fun with a lot of complex machinery and combinations... 127 is... too little. Look at Minecraft, for example.. Are you sure that even extending that to one more byte is going to be that punishing?

noncom on 22 Aug 2020

127, for Mindustry, is a lot of items. You should never need that many; this isn't Minecraft.

Anuken on 22 Aug 2020

👎2 👍1

Well, this leads nowhere. Thinking like this, I should never need to play games at all.

I get it, one byte is ok for vanilla and why sacrifice bandwidth if vanilla or a game with 1-2 mods won't need that. But this can be treated by allowing the games to switch to an "extended protocol" in case they need more addressing space and the players are aware that it would cost more because mods... Other than that, I might be missing something, but looking at the sources for v6 I'm currently not seeing any reason why the addressing space can't be extended.

noncom on 23 Aug 2020

Thinking like this, I should never need to play games at all.

Sure, you don't need to play anything. You don't need a ridiculous amount of items for a good modded experience.

switch to an "extended protocol"

that means no fun with a lot of complex machinery and combinations

Anuken on 23 Aug 2020

👍2

Apologizing for a bit of a text, but I just wanted to clarify some thoughts on this.

How would this switch be done in a clean way that would be backwards compatible? Items are written as bytes in several different places as binary, and these places have no way of knowing what the save 'version' is or which bytes were written. Sure, it's possible to rewrite IO and add a bunch of compatibility code, but I don't want that.

Ok, I understand, this is valid, but if that theoretical "extended protocol" is a thing, then:

The games of vanilla protocol don't have to be compatible with extended, neither in the saved game, nor in the network form. You can't play the same game with different mods.
The game always knows automatically if it's in the extended mode because of the number of items that's added by the current modpack. The clean way would be to abstract the IO codecs and to chosoe the appropriate implementation (lense) at runtime.
When loading a vanilla map the game already knows it's in the heavy-modded state, so it loads the map with that and when it serializes anything, it already writes the data with the extended protocol.
In terms of compatibility: no compatibility code needs to be added. Even if for some reason you would want to only convert game data to extended format without actually playing it and converting it automatically with a CLI app, it would require a very simple one-way migration. But all that might be unnecessary because of the above points.

How is 127 (or, more accurately, 255, since unsigned bytes are used) items too little? The base game has only 16, and even with that number you can do plenty - in fact, some items are still underutilized in the base game.

Yes, only the key point here is not that we exactly need over 9000 items. The key point is that we want to play with a certain modpack. And the effect that these mods cumulatively add that many items is collateral. Why should the user care if this would pose a problem or not? Consider the following situations:

In a 4-player game each player chooses 2 mods to represent his/her race, and 2 other mods for common QoL improvemens are available for everyone. So, 2 + 4 * 2 = 10 mods in total. The fact that these mods might be item-heavy and overflow the total limit is irrelevant, the players want to play with this set of mods because of their other qualities. The players like the mods, not exactly the resources. And it's less resources per player because players don't use the mods that don't belong to their race.
A player or a group of players having a favourite mod pack of 6 mods and playing it, let's say, for a half of a year already and immensely enjoying it. Everybody is super happy. Mindustry is flourishing, new features keep arriving, and the mods are following the suit. Each mod author cares about his mod and adds stuff. And suddenly the modpack stops working because someone added that one final extra ore. Boom! Pick a mod that's going to leave your favourite modpack now. Have you watched the movie "Circle"?
And that's going to touch a lot of mod combinations, sooner or later. Either that or the mod authors will have to start constraining themselves paranoically without even knowing if their mod is going to be used in a tight mod pack or alone. Because there would be no guarantee that your mod is going to work for everyone if you add that one extra production line that your think would have been super-awesome for your mod.
The authors would care for that because getting in a modpack equals popularity of your mod. And that's exactly why you are creating a mod -- so that people would play it. The limitation of blind competition for a place in a modpack would have a very negative effect on creativity here, ultimately leading to lockdowns.
Situations are possible where authors might want to add complicated production lines. Not every resource has to be used in a thousand ways. It's alright to have some rare stuff that's used only in a limited amount of applications.
It's alright to only use certain techs from the mods. You don't necessarily have to use the whole mod. However, the item amount penalty applies to the whole mod at once. Need a couple of buildings from a mod but that mod adds 15 extra items? Bad for you, player.
And speaking about the amount of items itself.... even that can be useful: for example in speedruns about building a factory that keeps up a certain production rate of hundreds of resource types and in puzzles.

I don't know what you think, but if you keep developing the game, then most likely the modding is going to get more and more sophisticated. That's simply natural if you plan to keep developing the game and the community. You have opened that box already, and you have already changed the game a lot to allow modding. Improving mod support (and that's what it is), is only a logical step forward.

An alternative way of dealing with the situation would have been to forbid modding resources and to pre-create a vocabulary of 128 (or 256) resources and force all the mods to only ever use these. But this is fishy. And if you'd happen to pull that one off today, that's what would bring some actual destruction.

Otherwise you simply delegate this problem to the end users. Turns out the final customer has to think about technical problems.

And let's say that you can ignore me now, I can perfectly accept it. I can fork, spend a bit of time, change some lines of code and make a build for me and my friends -- a piece of cake. But I can't imagine all this stuff going backwards ever. If not me, then others are going to come and question this over and over again. And they'd be right...

noncom on 25 Aug 2020

👍1

The games of vanilla protocol don't have to be compatible with extended, neither in the saved game, nor in the network form. You can't play the same game with different mods.

This is problematic.

Let's say I install several mods and start using them in my saves, but at some point decide that one of the mods should be removed. Maybe the mod doesn't function well, maybe it's not well balanced, or maybe I don't like the art - the reason doesn't matter. I go and delete this mod, and now the game may decide that it's no longer in this "heavy modded" state - this causes a switch back to the original codec.

Suddenly, all my saves written with the extended format get broken, even if I never used any content from the removed mod. Since the switch was never recorded anywhere, the user can't even see any meaningful errors. This isn't acceptable behavior.

If I'm being honest, there's no need for an extended format at all. Adding an extra compressed byte per item is not going to noticeable.

And that's going to touch a lot of mod combinations, sooner or later.

I disagree. The overwhelming majority of users are not going to be installing as many mods as you are - and even in your case, with many of the largest mods available being used, you still didn't hit the [unsigned] byte item cap.

It comes down to what you think is reasonable. If items were capped at INT_MAX total, you probably wouldn't care, because that's an absurdly high number that no one should be able to reach in normal circumstances. The same applies for Mindustry's 256 item cap, in my opinion - unless you're deliberately trying to turn your game into an unplayable mess, you should never reach that amount.

It's also worth noting that Mindustry, in its current state, simply does not support large amounts of items well, even outside of IO. Scrolling through 256+ item/block icons in a sorter or placement menu is tedious. Sure, you can argue that Mindustry could have better UI support for large amounts of content in future updates, but the fact is, it doesn't.

[...] mod authors will have to start constraining themselves paranoically without even knowing if their mod is going to be used in a tight mod pack or alone.

If mod authors are forced to think carefully about whether they want to add an item instead of adding new ones left and right, that's fine by me. However, I really don't think this would happen - as said above, most player will not be using massive modpacks, and those that do will be very unlikely to run both into the hard item cap and bother specific mod authors about decreasing their item count.

The only notable exception for mod authors getting constrained in some way would be auto-generated items for holding metadata - these are fundamentally unsupported and would cause massive problems outside of IO anyway.

Situations are possible where authors might want to add complicated production lines. [...]
It's alright to only use certain techs from the mods. [...]
And speaking about the amount of items itself.... even that can be useful [...]

Your main argument here seems to come down to, "Mods in the future will be using more different items and hitting the cap frequently" or "Running into the 256 item cap will be a common occurrence for many players", and I disagree. In the unlikely event that the item cap becomes a common problem, I'll see if I can change the IO to accommodate for it, but otherwise, I don't see a good reason to do so.

[...] pre-create a vocabulary of 128 (or 256) resources and force all the mods to only ever use these [...]

Yeah, no. I don't see how this solves any problems.

Otherwise you simply delegate this problem to the end users. Turns out the final customer has to think about technical problems.

If the technical problem is "I can't install a million mods at once", that's fine by me. It's impossible to isolate all technical problems from users, and presumptuous to try.

Anuken on 26 Aug 2020

👍1

Thanks for the detailed answers and explanation!

I agree with all that, actually, you are right. There of course come forth the GUI questions and game dynamics and many more where quantity starts turning into new qualities that require consideration. I think that the cases I described can be valid but they are very hypothetical for the current state of things, and that Mindustry, as a product used by a huge amount of users, should focus on great experience for everyone. I am glad and very pleased that you have such a balanced and practical outlook on it! Thanks to that we have what we have now, let's remember that :)

Also good to know that by your estimation there should be no serious technical issues with the memory/bandwidth consumption in case if a need to increase the cap should arise some time in future, or in a private fork if I get to implementing it.

However! A few questions from this discussion still remain unclear for me:

I have encoutered the bug on 140+ items, it did not happen after 128 items, so it's not that we hit the signed byte positive range cap, obviously it uses the sign bit as well. So it is using 256 ids now, but why it crashed on 140+ ids? Do you maybe have an idea? Could it be about those auto-generated metadata items that occupy extra place?
You have mentioned that the limit is going to be 127 items for v6. So, in the end, are you going to limit it by 128 or 256 ids? Or do you think that it's going to remain to be 256?
In the discussion we were using 127/128 and 255/256 interchangibly, so is there a special value, like 0, that's reserved, or was it just a figure of speech and we actually were talking about amounts of 128 and 256 everywhere?

So what's the actual situation here, could you please clarify this last bit?

noncom on 1 Sep 2020

the item limit has recently been increased: https://github.com/Anuken/Mindustry/commit/58182fe0401455a52325120ea1158a4facadc3e9

Quezler on 1 Sep 2020

O_O

I have been so deeply busy on my job for these days, so that after I read Anukens answer back then I have not even had time to check in, I did not know!

I hope the things are going to keep well, then! Let's see what exciting changes come out of it!

noncom on 1 Sep 2020

Was this page helpful?

0 / 5 - 0 ratings