Opentrons: bug: deck calibration doesn't cleanly shutdown the robot

[nickcrider]

Overview

There's something funky about how deck calibration is rebooting the robot. Perhaps the filesystem isn't getting flushed?

Steps to reproduce

1. On 3.19.0 open up a SSH terminal to the robot and do journalctl -f

2. Start the deck calibration process in the app. You'll see a whole bunch of serial and api messages printing like they should.

3. Do it kind of quick and lazy. Complete the deck calibration, and command the robot to reboot.

4. You'll see a websocket error and the SSH host will disconnect

5. Wait for the robot to reboot and the API server to start, now go look at journalctl -b -1 -n 50

6. The log messages you saw when finishing deck calibration won't be there.

Expected behavior

The system should shut down cleanly and flush logs sent to journald to disk.

Furthermore, does re-calibrating the deck _really_ require a full reboot of the pi, thus bringing the network adapters down/up (we all know there is some pain there)? I haven't tested, as of this writing, but I have a theory that a simple systemctl restart opentrons-robot-server would incorporate any manual modifications to deck_calibration.json. More to come on that...

[SyntaxColoring]

Wait for the robot to reboot and the API server to start, now go look at journalctl -b -1 -n 50

Because of journalctl badness that I've expounded upon elsewhere, I don't think this will work.

-b -1 should be "the boot before last," according to the journalctl docs. But the way it orders those boots is problematic. It orders them by something like the timestamp of the first message logged in each boot. Since the OT-2 doesn't have a real-time clock, and since that first message will be logged before the OT-2 has a chance to synchronize with a time server, the boot time will be wrong. Which means the boot ordering will be wrong. Which means this command might not show logs from the boot you expected it to.

[nickcrider]

Ah good point, I think that's the case. I knew it would affect the time based sorting (eg. journalctl --since=yesterday) but I naively assumed that the boot order would be logically auto-incremented, given that the boot_id is is consistent. From this 5 year old issue on the systemd repo it appears as if I was not the only one to (wrongly) assume it worked this way.

I think to prove this case we'll have to get the /proc/sys/kernel/random/boot_id before starting the deck cal and explicitly ask journalctl for that boot by ID once we've rebooted to see what exactly got written to disk before shutting down the system.

I will play around with this and re-write the procedure. Thanks for the comment and helping to pull the pieces together @SyntaxColoring !

[sfoster1]

Deck calibration (in the app at least) won't restart the robot after it completes in 3.20.