Umbrel: OTA update stuck at "downloading"

[mayankchhabra]

This has, in all cases, been reported by users on manual Umbrel installations.

[jgmontoya]

I'm having this issue as well. Also manual installation. How should one go about updating without losing information?

[snapo]

same here, manual installation (ubuntu 20.04) upgrade from 26 or so to 28 stuck at downloading ... as i have no funds yet on it i play a bit around. maybe i figure it out

[mayankchhabra]

As a temporary fix, if your update is stuck at downloading, you can run cd into the umbrel directory and run the following to reset the update and get back into dashboard:

rm statuses/update-status.json && echo "{\"state\": \"success\", \"progress\": 100, \"description\": \"\", \"updateTo\": \"\"}" >> statuses/update-status.json

[mayankchhabra]

Logs from a stuck update from @jgmontoya

=======================================
=============== UPDATE ================
=======================================
========== Stage: Download ============
=======================================

Creating lock
Cleaning up any previous mess
Downloading Umbrel 0.2.5
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   123  100   123    0     0    200      0 --:--:-- --:--:-- --:--:--   199
100    14  100    14    0     0     11      0  0:00:01  0:00:01 --:--:--    11

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Running update install scripts of the new release
./scripts/update/update: line 109: cd: scripts/update: No such file or directory
ls: cannot access '*-run.sh': No such file or directory
Deleting cloned repository
Removing lock

[jgmontoya]

If anyone want's to update on the meantime this worked for me:
sudo ./scripts/update/update --repo getumbrel/umbrel#v0.2.8

I don't think it matters but at least I had my instance stopped at the time of updating.

EDIT: There are some risks of doing this, please read this before you chose to upgrade using this method: https://github.com/getumbrel/umbrel/issues/205#issuecomment-691475831

[snapo]

If anyone want's to update on the meantime this worked for me:
sudo ./scripts/update/update --repo getumbrel/umbrel#v0.2.8

I don't think it matters but at least I had my instance stopped at the time of updating.

That worked absolutely superb...

[jgmontoya]

I could update using the dashboard normally from v0.2.8 to v0.2.9 without any problem whatsoever.

EDIT: Just updated to v0.2.10 OTA without any problems as well

I think this issue can be closed :tada:

[mayankchhabra]

Well, that makes it more complex because I can replicate the issue on http://testnet.getumbrel.com lol (which runs a manual Umbrel installation on Ubuntu 18.04).

An interesting discovery is that when the OTA update gets stuck, it's not logging to syslog for some reason. And if you try to reset the update state, redirect the log output to another file and re-trigger the process, the update doesn't get stuck anymore. Crazy.

[mayankchhabra]

Still the same issue on https://testnet.getumbrel.com when trying to update to v0.2.10:

Update: Added exec &>> /home/ubuntu/ota.log to the top of update script and it worked again. 🤦‍♂️

[lukechilds]

If anyone want's to update on the meantime this worked for me:
sudo ./scripts/update/update --repo getumbrel/umbrel#v0.2.8

I don't think it matters but at least I had my instance stopped at the time of updating.

Just a quick word of warning, this is something we use internally for development. It bypasses all semver range checks.

e.g If you're doing a big jump, say going from v0.1.2 to v1.2.3 there might be some incremental steps you need to do, like update to v0.5.0 first, then update to v1.2.3. This is all handled automatically when updating via the dashboard, but it isn't if you update directly with the update command.

It will just directly install whatever version/branch/commit you specify over your current installation.

It's fine to do in this case, and in the future if you understand the risks and check that you're updating between supported ranges.

But do be careful!

[vindard]

Great news, I was able to replicate the issue as well and found that it was a quirk with karen trying to run the update script multiple times and mistakenly updating update-status.json even after the update is finished if the update file is left in the triggers dir.

The fix should be to simply make sure the signals/update file gets deleted as part of the update process (and maybe a little cleanup of the update trigger logic for good measure).

Before I do that explicitly though, do you all already have a way of cleaning up files in the signals/ folder after each successful trigger for other triggers? I couldn't seem to find any in the code.

---

As an aside, running umbrel locally is tricky because there's no easy way to kill persistent processes kicked off in start script (karen, monitor, decoy-trigger). We could probably include killing those somehow as part of the stop script as well.

[lukechilds]

Awesome, thanks for looking into this @vindard!

Before I do that explicitly though, do you all already have a way of cleaning up files in the signals/ folder after each successful trigger for other triggers? I couldn't seem to find any in the code.

You mean statuses/, right? signals/ are just touched to trigger an event.

No, currently the status files are just removed when the scripts finish executing which is problematic. Really we need some sort of trap condition that correctly handles any way a script can fail and ensures status files are cleaned up. See this comment for more details: https://github.com/getumbrel/umbrel/pull/188#discussion_r479740992

As an aside, running umbrel locally is tricky because there's no easy way to kill persistent processes kicked off in start script (karen, monitor, decoy-trigger). We could probably include killing those somehow as part of the stop script as well.

Yes, ideally those services should be stopped as part of stop. However the update process is spawned by karen and calls stop, so we need to make sure that's implemented in a way that OTA updates don't kill themselves halfway through updating.

[mayankchhabra]

It was a quirk with karen trying to run the update script multiple times and mistakenly updating update-status.json even after the update is finished if the update file is left in the triggers dir

@vindard Are you sure that this is indeed the case? Because:

• events/signals/update file shouldn't cause karen to trigger if it's just sitting there. It needs to be touched again (or updated/modified in some way). That is also why there is no benefit of cleaning up the events/signals/ directory.

• update-status.json reflecting incorrect update status means that in reality the update went through, but is just stuck in the UI. However, that has not been the case in my testing as well as with people who've encountered this issue. The update never went through (as verified by version checks in info.json).

[vindard]

Hmm, ok this is what I was able to replicate so far. Let me know how consistent this is with you all's experiences.

My steps from my testing were:

1. Have the repo open in VS code and a terminal running where I ran sudo ./scripts/start
   I was able to see karen logs from that terminal

2. Kick off an update from the UI (I did 0.2.9 to 0.2.10)
   My update completed successfully and I was _briefly_ able to get back into the dashboard and observe everything looking ok

3. Leave everything open to observe. Notice the update script get triggered again
   In my terminal I started to see 3 types of fswatch events just after the update happened: '', '.gitkeep', 'update'. The first two did nothing, but the 3rd one re-ran the update script.

   I suspect this may be something to do with VSCode being open and creating fswatch events somehow but when the update trigger got re-run I saw:

   • the exact error from comment above (https://github.com/getumbrel/umbrel/issues/205#issuecomment-687167339)
   • the update-status.json file get erroneously changed to be in a downloading state (_*see below_)
   • the Downloading... screen (after the update-status.json change) from this comment (https://github.com/getumbrel/umbrel/issues/205#issuecomment-689741078) when I try to log back in (I assume because the update-status.json file is now wrong)

The thing I was able to replicate was I was able to get the update to happen, but then quickly after, the update was triggered again which produced the error messages in the logs and put the dashboard into that inconsistent Downloading... state by changing the update-status.json file incorrectly (even though the update had already happened and was fine at the fs level).

Since this has only been happening on non-Pi installs, I was wondering if there may be something other than the initial touch that could trigger signals/update fswatch events and lead to the triggers/update script being re-run illegitimately and producing that error state I described :thinking:

---

My initial ideas to try some fixes were to :

1. Remove the update file from signals dir during the update process so that if the underlying filesystem was doing anything else weird or someone was browsing the repo in the IDE while Umbrel is running there wasn't any danger of re-triggering the update process

2. Take a look at the triggers/update script again and see if I can put in any checks/safeguards to make sure we can't get to that erroneous update-status.json state even if the update scripts gets re-run somehow wrongly after an update

---

* _Wrong update-status.json state:_

{"state": "installing", "progress": 10, "description": "Downloading Umbrel ", "updateTo": ""}

[vindard]

I submitted a partial fix for this issue for the cases that I was able to replicate that lead to these types of errors.

The fix should help _some cases_ where the update-status.json can get to the "10% Downloading" inconsistent state from the update script, as described by some of the logs/comments^ above

Partial fix details

• ^ update-status.json reflecting incorrect update status means that in reality the update went through, but is just stuck in the UI. However, that has not been the case in my testing as well as with people who've encountered this issue. The update never went through (as verified by version checks in info.json).

I haven't been able to replicate the update entirely not going through as yet on my end, but I'll keep poking at it to see if I can prompt this scenario

.

^ An interesting discovery is that when the OTA update gets stuck, it's not logging to syslog for some reason. And if you try to reset the update state, redirect the log output to another file and re-trigger the process, the update doesn't get stuck anymore. Crazy.

Update: Added exec &>> /home/ubuntu/ota.log to the top of update script and it worked again. man_facepalming

This behaviour is also so strange. I haven't been able to replicate that one at all on my end as yet.

---

_Edit 1: I still haven't been able to replicate the in-flight error, but I tried my hand at a "try-catch" for the update script to see if that would help to at least not have the dashboard stuck (details here)_