Umbrel: Stop Umbrel if external storage is unmounted on Umbrel OS

[mayankchhabra]

A few people have reported that the sync wrote on their SD card, instead of the SSD. Ideally, this can not happen because the umbrel-startup.service waits for umbrel-external-storage.service to execute successfully before starting Umbrel, but there appears to be a case wherein the drive gets unmounted _after_ being mounted first (and might potentially get re-mounted as a read-only filesystem).

I'm not sure about the exact circumstances that cause this; it could be a human error (manually unplugging the drive), under-voltage, bad drives, etc. So all we need to do here is to just stop umbrel-startup.service as soon as the drive is either unmounted or remounted as read-only.

[vindard]

Happy to work on this one ... I'll take a look and get back to you guys here with how I intend to tackle it (over next few days). Appreciate any hints/suggestions as well if you guys have any 😅

[lukechilds]

I believe the issue is if the Pi is powered by a power supply that isn't powerful enough to also provide full power to the external storage device, the device, and therefore the filesystem is unstable. If systemd detects this, it unmounts the filesystem and remounts it as read-only to prevent data loss.

This results in our bind mounts being removed so now $UMBREL_ROOT is on the SD card and not bind mounted to the external storage. IDB will continue running and eventually fill up the SD card.

I haven't confirmed this is exactly what's happening, but this is my assumption based on the reports we've had.

Underpowered power supply seems to be the likely cause, but the same issue will probably also be caused by faulty hardware or a corrupt filesystem.

We should listen for an unmount on /mnt/data or the $UMBREL_ROOT bind mount and stop Umbrel as soon as this happens.

Ideally we'd have some sort error web page explaining to the user what's going on, although that could maybe come as an additional fix in the future, the important thing right now is that we just stop Umbrel from running.

[lukechilds]

Also, just as an aside re the error page, it might be useful if we just have like an umbrel-error Docker image. It would just be a generic HTML error page that accepts a text argument from STDIN to display on the page.

If anything goes wrong, not just this specific issue, but any other issue where we need to stop Umbrel, we can run this directly from Docker (no need for compose) in a script.

e.g:

docker run getumbrel/umbrel-error "Umbrel has shutdown because your external storage device is unstable, this is likely due to a hardware error."

[mayankchhabra]

docker run getumbrel/umbrel-error "Umbrel has shutdown because your external storage device is unstable, this is likely due to a hardware error."

Ingenious 🤯

[vindard]

Noting down some useful logs/feedback from the Telegram chat on this

• Logs for undervoltage and incompatible SSD adapter errors https://t.me/getumbrel/5284

[nolim1t]

I think also monitoring for voltage throttling would help mitigate this. Maybe pop up with a warning on the dashboard like RPI does if you plug it into the monitor

vcgencmd get_throttled