Urbit: after moving pier, checkpoint and log disagree! las:133297 old:133311

[thunderrabbit]

I caused trouble for my planet ~rabsef-linmyn when moving it from my local machine to AWS (and back).

• I had my planet ~rabsef-linmyn running fine on my local machine (rebooted after recent breach)
• I tared and moved it via ssh to AWS.
• ~rabsef-linmyn ran fine on AWS.
• I ctrl-d stopped ~rabsef-linmyn on AWS
• _Without tarring the AWS version_, I tried starting the (previous) local version to see what would happen.
• It did not work, so I tared and moved it via ssh from AWS to my local machine.
• Started the latest-from-AWS pier on local machine and it still did not work.
• (I do not know if I ctrl-d stopped my planet on my local machine at that time, but I probably did)
• Restarted ~rabsef-linmyn on AWS and got the error below.

$ urbit rabsef-linmyn
~
urbit 0.7.4
boot: home is /home/ec2-user/rabsef-linmyn
loom: mapped 2048MB
boot: protected loom
live: loaded: MB/269.418.496
boot: installed 258 jets
arvo: time: ~2019.4.18..12.38.27..5411
rest: checkpoint at event 133.310
rest: old 0v1p.blipn, new 0va.dftqi
boot: loaded passcode from rabsef-linmyn/.urb/code.~nisbex-dinnec
---------------- playback starting----------------
checkpoint and log disagree! las:133297 old:133311
Some events appear to be missing from the log.
Please contact the authorities, and do not delete your pier!

• I emailed Morgan via [email protected].
• here's what Ted had to say (edits for clarity):

This is dead code crashing, so we probably shouldn't be too aggressive in debugging it, but some effort might be worthwhile.

My guess is those last fifteen events didn't get flushed to disk despite being written. It's notoriously difficult to write a file on Unix and know it's actually been written ... that's the world we live in, and it's why we're switching to a database that has hopefully already solved this issue.

For more details on fsync(), if you're that kind of nerd: http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/

[Because] he booted a new ship after the breach, maybe there's a recovery process.

Joe or Philip, do you know if his ship would just boot off the snapshot if he renamed his event log file to something else? I suspect his snapshot is valid, so if Vere just started off that it would work.

[ohAitch]

Hm, fsync seems unlikely to be the culprit, since the AWS subsequence of events was

• moved tarball to AWS
• it ran fine
• stopped with ctrl-d
• [tar read]
• restarted ctrl-d pier and got error

which does not a involve a reboot between the stop and start(let alone hard shutdown). Are there ways the file write could have been misplaced on the way to the _RAM filesystem cache_ somehow?

[jtobin]

@thunderrabbit Did you happen to get anywhere with this out-of-band afterwards?

[thunderrabbit]

No one else has replied regarding this issue.

I have simply crossed my fingers that this is solvable, perhaps just by waiting for the next breach.

[joemfb]

@thunderrabbit, sorry for the lack of follow up on this issue. We have breached, and this new release includes an entirely new event persistence system. Among many other things, this new system enforces the necessary invariants to prevent ships from getting into this broken state.

[ohAitch]

Are both the log and snapshot in lldb now?

On Mon, Jul 22, 2019 at 21:46, Joe Bryan [e-mail] wrote:

@thunderrabbit https://github.com/thunderrabbit, sorry for the lack of
follow up on this issue. We have breached, and this new release includes an
entirely new event persistence system. Among many other things, this new
system enforces the necessary invariants to prevent ships from getting into
this broken state.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/urbit/urbit/issues/1250?email_source=notifications&email_token=AAOFPBUNB4VN6QC24BBIS2TQA2EJXA5CNFSM4HIIODF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2R5ASQ#issuecomment-514052170,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOFPBREBDNV4PYTC27QXLTQA2EJXANCNFSM4HIIODFQ
.

[joemfb]

No, the log is in LMDB, while the snapshot is unchanged (a memory image, in the pier at .urb/chk/*.bin). But there's only one code path for creating a snapshot, snapshots are only ever requested at the most recent computed and committed event (we block further events to synchronize), and snapshot creation itself is guarded by an assertion that we are snapshotting the state derived from the event at which we expect to be.