Refinedstorage: [1.11.2] Disks Disappearing

[dannydjdk]

Issue description:

We are having issues with Disks disappearing from our drives. It tends to occur when the server crashes for any reason. When the server comes back up, everyone's drives are gone. Crafting patterns and upgrades are also disappearing, though sporadically. We've also been seeing items disappearing in the Solderer. For instance, when placing 16, we'll end up with 4 soldered, and the rest will disappear. Not sure if that's related to the other issue though.

Unfortunately, the problem is quite intermittent and I haven't come up with a way to reliably replicate it. I'm still working on this, but, considering the severity of the problem, I thought I should let you know about it asap.

What happens:

On server crash (or possibly any restart), disks, patterns and upgrades will intermittently disappear.

What you expected to happen:

Of course, we expect disks, patterns and upgrades to persist when server crashes/restarts.

Steps to reproduce:

1. Have RS system with disks and crafters with patterns, etc.
2. Crash server with some other buggy mod.
3. Restart and possibly freak out cuz all your stuff is missing. ;)

Version (Make sure you are on the latest version before reporting):

• Minecraft: 1.11.2
• Forge: 13.20.0.2227
• Refined Storage: 1.4

Custom modpack/mod list: http://dannyandson.com/unreal-tech-modpack/

Does this issue occur on a server? Yes

If a (crash)log is relevant for this issue, link it here:

No crash log to report. Logs not available at the moment.

I apologize for the lack of info at this time. I realize that this will be quite difficult to pinpoint with the information I've provided, but again, considering the severity, I didn't want to delay the report. Let me know if there's any further info I can provide or troubleshooting steps I can take. My time for troubleshooting is limited, but I'll do what I can.

[raoulvdberge]

I did know about this, and it should be fixed for 1.4.1. I gave a (serverside) testbuild to a few people and they didn't report any more of this happening. This is due for release today or tomorrow.

[MrManNo1]

In case you're still searching for a solution to this, I've noticed that whatever is causing the glitch seems to be stripping placed blocks of all their NBT data. Not only are the actual "physical" items lost, but any settings applied to the blocks are also lost (so, for example, settings applied to importers and exporters).
Not sure if you knew this already, but it might help.

[kreezxil]

Is the update being pushed for this or not, I see there is nothing pushed since February 7th, 2017.

We are having the same issues on our server in 1.11.2, nobody noticed until we upgraded to Forge 2232.

[raoulvdberge]

My fix apparently didn't work, so I still don't know what causes this.

[digghair]

Refined storage drives gone again from machines including speed upgrades and filter item list in export card no crash report from client side I just log in after a few hours and its all gone this is second time in the two days after updating this mod that this is happening.

[raoulvdberge]

@digghair Yes, that is nothing more than we didn't know yet. I'm still investigating the issue, I don't know what causes it.

[raoulvdberge]

Is this still an issue in RS 1.4.1?

[dannydjdk]

Sorry, we assumed it still was an issue based on your comment above, so we've been using external storage. We'll test it over the next few days and let you know what happens.

[vikespear]

I still get this issue. I play on version 1.4.1 in All the mods 2 modpack version 0.50.
I have 2-3 independant RS systems on the server. One of them is chunkloaded for getting items out of the quarry and the other two (meanwhile only one other) isn't chunkloaded. The one that isn't chunkloaded always lose everything when I log in the next day and no one was on the server. I tried using storage blocks instead of disks, because i thought blocks in the world maybe don't disappear like the disks in the drive. The storage blocks are still there but the items on it are gone completely. Also upgrades in the wireless transmitter disappeared.

Everything I described happened just in the non-chunkloaded system, it never happened to the chunkloaded one (I don't know if that is important, but as I use more than one rs system and it happend only to that one I thought it would be worth mentioning it)

[MrManNo1]

Can also confirm this is still happening on dannydjdk's server on 1.4.1. Just happened again to us earlier today.

[raoulvdberge]

When this happens, are all disks gone throughout the server or is it only limited to a few people?

[dannydjdk]

Usually it's limited to a few people, but it's not always the same people. It seems random (that's always helpful to know, right?). There was at least one instance where it appeared that everyone lost their items (disks, crafting patterns, upgrades and settings) at the same time. It corresponded with an unrelated server crash.

Our server is configured to allow chunk loaders to operate while the player is online and for an hour after disconnection. This seems to support @vikespear's suspicion that chunk unloading and reloading triggers the problem. In our case, it seems like the disappearances were only occurring after a crash or reboot, but it may just be that that's when we happened to notice because of the mass unloading and reloading of all players chunks after a reboot.

[kreezxil]

I wonder if this is related to the issue in minecolonies where chunk loaders cause the npcs to stop interacting with the player. Suspiciously it sounds as if a world saving function is being bypassed for loaded chunks. But then why isn't there world corruption?

-----Original Message-----
From: "Danny" [e-mail]
Sent: ‎2/‎25/‎2017 10:03 AM
To: "raoulvdberge/refinedstorage" [e-mail]
Cc: "kreezxil" [e-mail]; "Comment" [e-mail]
Subject: Re: [raoulvdberge/refinedstorage] [1.11.2] Disks Disappearing (#986)

Usually it's limited to a few people, but it's not always the same people. It seems random (that's always helpful to know, right?). There was at least one instance where it appeared that everyone lost their items (disks, crafting patterns, upgrades and settings) at the same time. It corresponded with an unrelated server crash.
Our server is configured to allow chunk loaders to operate while the player is online and for an hour after disconnection. This seems to support @vikespear's suspicion that chunk unloading and reloading triggers the problem. In our case, it seems like the disappearances were only occurring after a crash or reboot, but it may just be that that's when we happened to notice because of the mass unloading and reloading of all players chunks after a reboot.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[raoulvdberge]

The only place where a node is removed is in BlockNode#breakBlock which is only called when the block is broken:

    @Override
    public void breakBlock(World world, BlockPos pos, IBlockState state) {
        super.breakBlock(world, pos, state);

        INetworkNodeManager manager = API.instance().getNetworkNodeManager(world.provider.getDimension());

        INetworkNode node = manager.getNode(pos);

        manager.removeNode(pos, true);

        API.instance().markNetworkNodesDirty(world);

        if (node.getNetwork() != null) {
            node.getNetwork().getNodeGraph().rebuild();
        }
    }

[VT-14]

are any server add-ons like Sponge being used?

[dannydjdk]

No plug-ins; strictly Forge. And, apparently it has occurred in single player for some as well.

[IchigoGames]

This is Still Happening as of refinedstorage-1.4.1 Forge 2233 1.11.2 on The New Fronter Server

[TwitchtvCarbs]

I have noticed that the data files are saved/overwritten normally when the server shuts down (Figured this out because I have tried copying the data files while server is running and restarting and the files are not the ones I copied over, they get overwritten, of course shutting down the server then copying the files over and starting back up the server would _sometimes_ work and restore the data I had from a backup) could it be that the data is not saving all the way or the save for the data is canceled when a crash is occurring?

[ProsperCraft]

We just had a user loose every one of his drives. forge-1.11.2-13.20.0.2251

One of Squared pack.

He says he had just been intereacting with his RS system then was disconnected.
Here is a crash report around that time - http://pastebin.com/WcWAAk5c

Those are the coords of his base.

[Cntrldfusion]

Still happening. It's happening with my mod pack as well. RS 1.4.1 doesn't seem to matter which version of forge we are using on our server. It has happened to people with and without chunk loading. If they have drives the drives disappear and if they use the storage blocks the inventory disappears. So far the only common occurrence seems to be when we have a server crash, not a scheduled restart or random reboot. And the crashes we have had were not related to RS. We do use Minecolonies but the problem has happened to people with and without a colony. Not sure if any of this info helps.

[D4rknessKing]

Still happening here too! People are getting discouraged by having to recreate their pattern every 2 or 3 times a week... We need a fix.
P.S: It happens with /stop too! Not only with crashs.

[raoulvdberge]

I cant reproduce this bug! People who want this fixed should provide me with 100% reproduceable steps.

[kreezxil]

Raoul,

When you try to reproduce it, what mods are you using in your test
environment? Are you testing it in a dedicated server vs the one in the
build suite as there its a difference?

Are you simulating a server crash and also forced restarts?

Is it possible you have too many exception handlers in place that would
otherwise allow crash reports to be generated?

I've got a lot of time on my hands with the recent "stability" of 1.11.2,
knowing those answers will allow me to setup and force the issue on a
server thus producing steps.

On Mar 26, 2017 12:07 AM, "Raoul" [e-mail] wrote:

I cant reproduce this bug! People who want this fixed should provide me
with 100% reproduceable steps.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/raoulvdberge/refinedstorage/issues/986#issuecomment-289258962,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADs0iDBy0IMutr3SrmKLPSKT5K8vVK0Yks5rpfJ-gaJpZM4L_YCe
.

[Cntrldfusion]

I will also help as much as possible to collect information since this is one of the favorite mods on our server. I can say that I was not able to reproduce the problem by doing forced restarts of the server which is why I think it is related to when a crash occurs. We also do a scheduled nightly reboot on our server and it's never happened after that either. I have set up some storage systems in different places on our server and will try to provide more info when it happens again.

[raoulvdberge]

@kreezxil

When you try to reproduce it, what mods are you using in your test environment? Are you testing it in a dedicated server vs the one in the build suite as there its a difference?

I tried this on a dedicated server, yes.

Are you simulating a server crash and also forced restarts?

I'm simulating one by throwing an exception.

Is it possible you have too many exception handlers in place that would otherwise allow crash reports to be generated?

No.

[Cntrldfusion]

Did anyone that provided you with crash reports have this problem WITHOUT the Minecolonies mod? I removed Minecolonies from my modpack because it was interfering with other mods but I have not had this problem since I did that. Even when we updated to 1.4.2 I created another test setup on our server and have basically been using it as a server wide trash can to see what happens. We have only had a couple of crashes so far, but RS is still intact.

[OrionDevelopment]

@Cntrldfusion. Yes the original author of this Issue. His modpack does not contain any Minecolonies.
As the writer of the new IItemHandler system for Minecolonies, I can tell you that unless you are giving the disks, patterns etc to you Minecolonies Citizen, Minecolonies is not interacting with them in any way.

We have no compatibility interaction with RS (it is not needed, and provided by default thanks to MC Forge)

We donnot interact with your inventory nor with any other player. We only interact with vanilla chests, to make minecolonies feel as vanilla as possible.

[OrionDevelopment]

I dug a bit into this issue:
The only thing that might cause this from minecolonies side is its Colony Grief protection. However, the grief protection is based on normal events that forge fires (like BlockPlaceEvent, PlayerInteractEvent etc)

To handle the protection, events are cancelled when a player is not allowed to interact in a given region.

@raoulvdberge are you using a fake player anywere? If so it might be that your fakeplayer is blocked by Minecolony. This also caused issues with BC Builders etc, we discussed this issue with LexManos to see how to proceed, and he told us that the implementation for a FakePlayer should return the UUID of the Player who placed it.

If you already are doing this, then interaction with the colony is guaranteed to work.

[raoulvdberge]

How can a FakePlayer cause inventory data to be lost?

[raoulvdberge]

Are you messing with WorldSavedData stuff?

[Cntrldfusion]

I'm not sure if they do or not, but I think it's possible since also around the time this issue started they introduced a Warehouse for inventory storage and a Deliveryman NPC to interact with it.

[raoulvdberge]

RS stores every node in WorldSavedData. They are indepentent from the Tile Entity, but if you modify the inventory through the Tile Entity, the WorldSavedData instance will be changed as well.

[OrionDevelopment]

No we are not, all colony data is stored in a seperate file.

[OrionDevelopment]

And we are always asking for the fakeplayer thingy since a lot of mods seem to implement it wrong and then do not check for events that might be cancelled, it caused a lot of issues with for example EnderIO.
If you are not using it to place a block or something that might hold the disk data, then we are not interacting with it.

[Mifesto]

We do not use Minecolonies and we also experience this problem, do not fixate on this mod.

[raoulvdberge]

I wasn't.

[elbim]

Today I saw disappearing items and patterns for the first time in my world (MC 10.10.2, Forge 12.18.3.2185, RS 1.2.25) and I could reproduce it:
Lived quite well with the chunk loading ward of Extra Utils 2 for a long time and wanted to change to Chicken Chunks (v2.2.0.52), so that I didn't have to build that many wards. After placing the chunk loader, half of my patterns disappeard and also a whole lot off other items.
Restoring my backup, everything was fine again and I could reproduce this behaviour (place chunk loader -> save/exit world -> reload world -> items/patterns gone). No errors, no crashs.
could there be a correlation to this topic?

[raoulvdberge]

No. This is an external mod causing your chunk to corrupt.

[raoulvdberge]

Some technical information, in hopes that anyone with a technical background can help out:

RS network nodes (aka machines) are stored in a WorldSavedData (see com.raoulvdberge.refinedstorage.apiimpl.network.node.WorldSavedDataNetworkNode).
On world load event (see com.raoulvdberge.refinedstorage.apiimpl.network.NetworkNodeListener) it reads the world saved data NBT and puts it in a NetworkNodeManager.. next up: in TileNode#getNode, it gets the node from the NetworkNodeManager.

WHAT I THINK THAT IS HAPPENING:
TileNode#getNode is called somehow before the WorldSavedDataNetworkNode is read, causing it to reset the machine data.
I'm not sure about this though, it can be something else too.

WHY CAN'T THIS HAPPEN:
During read, it would just overwrite the earlier set value in the NetworkNodeManager, working around this bug... but I'm not sure.

WEIRD BEHAVIOR THAT CONFUSES ME:
This bug only occurs on some nodes, not on all nodes at once. This means that the corruption isn't world-wide, it's just limited to a few positions. This is possibly related to chunkloading / loading a chunk when a player joins?

WHAT IT ALSO CAN BE:
There is some code that reads out legacy NBT tags from the tile entity, and reads it to the new decoupled NetworkNode (for the 1.10 -> 1.11 conversion).

https://github.com/raoulvdberge/refinedstorage/blob/mc1.11/src/main/java/com/raoulvdberge/refinedstorage/tile/TileNode.java#L114-L135

What could happen is that the NBT tag count doesn't match, and it reads out the TE data again, while it shouldn't, and thus voiding essentially all machine data.

This still doesn't make sense to me, as this cause wouldn't occur randomly like it's described here (during server starts / restarts, but not always).

[kreezxil]

As awesome as you are, have you presented this to the team at Minecraft
Forge?

On May 12, 2017 3:04 AM, "Raoul" [e-mail] wrote:

Some technical information, in hopes that anyone with a technical
background can help out:

RS network nodes (aka machines) are stored in a WorldSavedData (see
com.raoulvdberge.refinedstorage.apiimpl.network.node.
WorldSavedDataNetworkNode).
On world load event (see com.raoulvdberge.refinedstorage.apiimpl.
network.NetworkNodeListener) it reads the world saved data NBT and puts it
in a NetworkNodeManager.. next up: in TileNode#getNode, it gets the node
from the NetworkNodeManager.

WHAT I THINK THAT IS HAPPENING:
TileNode#getNode is called somehow before the WorldSavedDataNetworkNode is
read, causing it to reset the machine data.
I'm not sure about this though, it can be something else too.

WHY CAN'T THIS HAPPEN:
During read, it would just overwrite the earlier set value in the
NetworkNodeManager, working around this bug... but I'm not sure.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raoulvdberge/refinedstorage/issues/986#issuecomment-301011488,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADs0iDCnsAtYPAXz79ILAxy3QJYyDWacks5r5BKDgaJpZM4L_YCe
.

[raoulvdberge]

Ok, I reworked the code a little and it should fix this bug.

Can you guys try this build:

https://drive.google.com/file/d/0B2PrXMobPWvWbUVabHhRcDQzQ1k/view?usp=sharing

(Warning: Clients need this build as well, you can't just use 1.4.4 on the client and this build on the server. The debug build is technically 1.4.5)

There is some debug output. If you're having problems, link the logs to me.

[Tithonius]

I seem to be catching a bit of an exception with the new build. my client happens to be crashing before title screen with the crash log found here: https://pastebin.com/93E0DFTt

[Tithonius]

The server is crashing as well. that crash log can be found here: https://pastebin.com/LNNjje9v

[raoulvdberge]

Ah, that is the OC integration crashing if OC isn't installed. It's 5am here, I can't fix it now. Can you just install OpenComputers for testing purposes and retry?

[Tithonius]

yeah will do

[Tithonius]

Okay, both server and client are up and running, I will be testing to see how this runs over the next few days. Will keep you posted

[raoulvdberge]

The server log, with the console output

[raoulvdberge]

Oh, you removed your comment. Are the drives gone or not?

[Tithonius]

no they arent, just had another issue, i fixed it your all good now :)

[Tithonius]

Ill be doing testing over the next few days and ill get back to you, but its all looking good for now :)

[Tithonius]

Would it at all be possible to get a version released on curse forge so i can push an update to the pack im using?

[raoulvdberge]

I first want to make sure it works before uploading anything

[Tithonius]

alright, no worries, ill have it on a test server for the next couple weeks or so, so ill keep ya posted

[raoulvdberge]

Does that test server get any users?

[Tithonius]

yeah anywhere from 5 to 15 or so depending on time of day

[raoulvdberge]

Cool. In what time intervals did this issue occur? That way, we can put a deadline and know sure the fix worked.

Make sure to restart and update the server as well at times, to stress test and simulate the cause of this problem.

[Tithonius]

yeah im going to be doing restart tests for sure, and ill be pushing pack updates during that time as mods update and become available.
but to be honest most of the times we had it happen was when we updated Refined Storage. It didnt happen when i updated to the debug build which is a good sign, but i will be keeping an eye on it for sure

[raoulvdberge]

@Tithonius

1.4.5 will be coming out on Curse in a bit. I decided to push it anyway.

[Tithonius]

Sweetness, i assume its the new version without the debug?

[raoulvdberge]

It still has the debug output, see my reasoning here:

https://github.com/raoulvdberge/refinedstorage/issues/1220#issuecomment-301374542

[Tithonius]

Coolio

[raoulvdberge]

Any updates @Tithonius?

[Tithonius]

So far its working beautifully. through all the updates and restarts we have had no isssues so far

[raoulvdberge]

I'm assuming this bug is fixed! Please still keep me up to date though. I'm closing it for now.

[whatthedrunk]

I just lost my drive again. No crash just upgraded forge and some mods. Not eveyone lost drives and mine was being chunkloaded with Chicken Chunks and was in my claimed land. Not sure if that's part of it but wanted to note it.

Edit: Other users who were chunkloaded with chickenchunks in claimed lane were fine so that was probably not it.

[kreezxil]

Holy smokes, so it's a good idea then to use the external storage
interface? For instance put everything on a diamond colossal chest ...
wondering if that'll safeguard us.

On May 21, 2017 8:42 PM, "whatthedrunk" [e-mail] wrote:

I just lost my drive again. No crash just upgraded forge and some mods.
Not eveyone lost drives and mine was being chunkloaded with Chicken Chunks.
Not sure if that's part of it but wanted to note it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raoulvdberge/refinedstorage/issues/986#issuecomment-302979646,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADs0iEVFi7J6HJs0nVdnNqJOL6jHMLG9ks5r8Of5gaJpZM4L_YCe
.

[whatthedrunk]

What I am doing for now is connecting a ton of storage drawers to a drawer controller connected to the external storage. @kreezxil

[kreezxil]

The bulk of my items are in hyper storage in an ez storage system as a
precaution. Will probably dismantle my drive so as not to have to endure
misery.

On May 21, 2017 10:47 PM, "whatthedrunk" [e-mail] wrote:

What I am doing for now is connecting a ton of storage drawers to a drawer
controller connected to the external storage. @kreezxil
https://github.com/kreezxil

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raoulvdberge/refinedstorage/issues/986#issuecomment-302992686,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADs0iBiNrwcJhNmMnZ8ELWTD9zs2GX_pks5r8QVHgaJpZM4L_YCe
.

[raoulvdberge]

I published version 1.4.7 on Curse.

It has a new config option "debugLog" that generates a .log file with diagnostic info for me, to help fix this bug.

If you are experiencing this bug, please enable it on your server.

If the bug happens again, send me the relevant log file.

[raoulvdberge]

This bug should be fixed in 1.4.9, please try it out and let me know!

[phit]

Just updated my server after having drives disappear for multiple players today, crossing my fingers this solves the issue. Thanks for the hard work!

[raoulvdberge]

No problem! I hope it's fixed now... please keep me up to date.

[phit]

Seems to be working, so far no new reports of missing drives! I'll post here if this issue comes up again.

[artillect]

I've had several disks empty themselves in a similar way. The disks are labeled as full and show as full in the disk drive but nothing shows up in the grid.

[kreezxil]

riley you need to tell us exactly which version of the mod you are using.
if you are using an older version provided by a pack that is unmaintained,
then you'll either have to update that pack yourself or get the maintainer
to do it, or move on to a pack that is actively maintained. I say all that
because raoul fixed this issue in the 1.11.2 branch.

On Sat, Apr 21, 2018 at 1:16 AM, Riley [e-mail] wrote:

I've had several disks empty themselves in a similar way. The disks are
labeled as full and show as full in the disk drive but nothing shows up in
the grid.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raoulvdberge/refinedstorage/issues/986#issuecomment-383271155,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADs0iG41VZe_SMmFbuPYeN4YWKlPZY8bks5tqs7EgaJpZM4L_YCe
.

[artillect]

I'm on version 1.5.32 on 1.12.2. This issue definitely isn't fixed on 1.12, sorry that I posted this in (probably) the wrong thread, but I'd rather not open a new issue when one already exists. I'm gonna go reply to the 1.12 one

[kreezxil]

I would say it counts as a new issue because it is for the 1.12.2 branch of
the mod as this bug chain was for the 1.11.2 branch. I was shocked to a see
a necro post. The must've done something to bring the bug back because it
was fixed for about 9 months at least.

While I'm not on his team.

Can you reproduce the bug reliably?
Same steps?
Can you provide the steps.

keep your client and dedicated server consoles open.
obvserve the activity when the drives disappear.
pastebin those parts of log where it shows something evil happen to
refinedstorage in regards to those drives.
finally pastebin a copy of the latest log from your client and the server.
(latest log is not a crash report log)

On Sat, Apr 21, 2018 at 12:26 PM, Riley [e-mail] wrote:

I'm on version 1.5.32 on 1.12.2. This issue definitely isn't fixed on
1.12, sorry that I posted this in (probably) the wrong thread, but I'd
rather not open a new issue when one already exists. I'm gonna go reply to
the 1.12 one

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raoulvdberge/refinedstorage/issues/986#issuecomment-383314552,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADs0iLxycgJPggN8awq5ULjJoTspn9emks5tq2vSgaJpZM4L_YCe
.

[artillect]

I haven't been able to reproduce it yet but I'll keep trying. Let's hope I don't replicate it again on my survival server.

I'll definitely give it a shot to replicate it though.

[yazzledore]

I am having a similar issue, but it doesn't just happen when the server restarts (single player world). Playing Skyfactory 3 and the crafting upgrade I'm trying to put in an exporter keeps disappearing about a second after I put it in the slot, as if it's being exported as well (but obviously doesn't go anywhere).

Actually, just found them somewhere and realized they actually were being exported, not sure if this is the same issue anymore.

[raoulvdberge]

Don't come to the bug tracker reporting stuff for 1.10.2