Umbrel: `Loading LND...` -> can't access dashboard because lnd not running

[kilrau]

Every x days I can't access the dashboard, because it gets stuck at this screen:

SSHing into the Pi4 (umbrel 0.3.7) then reveals the lnd container is not running. LND logs show a shutdown was triggered due to Health check: chain backend failed after 3 calls. LND log:
lnd.log

Starting lnd with docker start lnd works and everything is back to normal until lnd shuts down again a couple of days later. This started after upgrade to umbrel 0.3.7/lnd 0.12.1.

Possible solution: umbrel to monitor the backend and start lnd automatically once bitcoind is ready again. What made bitcoind unresponsive for lnd in the first place: no idea.

[mapecode]

I have the same problem

[dermichal]

Same for me.

[exitfiat]

Same

[cedracine]

"You are not alone..." - Same here, since the first version installed ~4 months ago (Raspi4b-4Go)
It happens to be "Loading Bitcoin Core" sometimes too...

[djyoungcity]

Same issue here. Never saw dashboard, this is a fresh install. Running umbrel-os-v0.3.7-patch-1.img

lnd | 2021-03-10 22:04:00.164 [INF] LTND: Version: 0.12.1-beta commit=v0.12.1-beta, build=production, logging=default, debuglevel=info
lnd | 2021-03-10 22:04:00.164 [INF] LTND: Active chain: Bitcoin (network=mainnet)
lnd | 2021-03-10 22:04:00.165 [INF] LTND: Opening the main database, this might take a few minutes...
lnd | 2021-03-10 22:04:00.165 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false
lnd | 2021-03-10 22:04:00.207 [INF] CHDB: Checking for schema update: latest_version=20, db_version=20
lnd | 2021-03-10 22:04:00.208 [INF] LTND: Database now open (time_to_open=42.650539ms)!
lnd | 2021-03-10 22:04:00.746 [ERR] LTND: unable to initialize neutrino backend: unable to create neutrino light client: EOF
lnd | 2021-03-10 22:04:00.746 [INF] LTND: Shutdown complete
lnd | unable to initialize neutrino backend: unable to create neutrino light client: EOF

https://umbrel-paste.vercel.app/g610esa

[BTCFranky]

I have the same issue after reboot. umbrel 0.3.7
Is there a workaround?

[Michael-5th]

same issue for me, stuck at Loading LND...

[louneskmt]

Hey guys!

Can you all get the logs and send them here?

Step 1 - In a terminal, execute this to SSH:

ssh -t [email protected]
Password: <your own dashboard password>

Step 2 - Once you are connected, run the debug script with the following command:

~/umbrel/scripts/debug --upload

Step 3 - Copy the Umbrel Paste link and send it here.

NB : On Windows, you might need to execute the following command instead of the one above.

ssh umbrel@umbrel

[louneskmt]

@djyoungcity So there is no funds yet on your Umbrel?

If so, please SSH into your Pi and execute the following to reset your Umbrel and LND:

sudo systemctl stop umbrel-startup && sudo rm -rf ~/umbrel/lnd/!(lnd.conf) && sudo rm ~/umbrel/db/user.json && sudo rm ~/umbrel/db/umbrel-seed/seed && sudo systemctl start umbrel-startup

EDIT: be careful with this command, it will reset your Umbrel and you will lose your funds. Save your seed. Do not use this if you have channels opened.

[Michael-5th]

@louneskmt I ran the script but I am not seeing anything resembling an Umbrel Paste Link.

I am on Windows and had to use: ssh umbrel@umbrel

I did get the result: The temperature of your Raspberry Pi is too high. We recommend either using a case that lowers the temperature of your Pi or a case with a builtin fan.

Could that be causing the issue? My Pi is slightly warm to the touch but not hot by any means. I took the top lid of the case off and will run a fan near it to try & improve the temp.

Thanks!

[louneskmt]

@Michael-5th Instead, you can copy the output to a Pastebin (pastebin.com) and share the link here 😁

[Michael-5th]

@louneskmt Thanks for the tip!

Link: https://pastebin.com/6E5F8Ri2
Password: mFPPW2Ykww

[exitfiat]

https://umbrel-paste.vercel.app/l310hb4
Still stuck at Loading LND...

[louneskmt]

@Michael-5th Can you reflash the SD card with v0.3.7-patch-1?

@kaschmirrr What if you run cd ~/umbrel && docker-compose restart lnd?

[kilrau]

Loading screens getting stuck can have a whole bunch of different reasons, so please help the umbrel team by digging into lnd logs before saying "same" here :pray: like e.g. @djyoungcity did here - it shows that the Neutrino backend has issues causing lnd to go down.

Anyhow, I figured out why my lnd did go down with Health check: chain backend failed after 3 calls. Whenever bitcoind loses Internet connection for a couple of minutes, lnd realizes via failed health-check calls (should find out which) and shuts down after 3 failed calls. Looks like a safety measure. I now programmed my router to go down every 24h and stay off for 15 minutes, then come back online. So umbrel's bitcoind is without connectivity for this period and lnd now realiably shuts down every day at this point.

Question is how to properly fix this: I think this health check should be configurable within lnd, since even a watchdog from umbrel side that automatically restarts lnd once bitcoind is back online would result in a locked lnd...

[exitfiat]

@louneskmt nope didnt help, still stuck there
@kilrau dont know how to get to the lnd log, i am not that technical person. But i can share it if i get instructions on how to access it.

[Michael-5th]

@louneskmt Hi - I re-flashed but cannot connect over Tor using the onion address or onion.local

Here is the updated logs: https://pastebin.com/4BhTmRkj

It may just need more time, so I will check again later as well and comment back if it's fixed. Thanks for the support!

EDIT: I'm back in today - looks like the re-flash worked. Thanks!!

[mwager]

@louneskmt your commands fixed it for me. thx!

[henrique-diniz]

I fixed this problem using this command:

sudo systemctl stop umbrel-startup && sudo rm -rf ~/umbrel/lnd/!(lnd.conf) && sudo rm ~/umbrel/db/user.json && sudo rm ~/umbrel/db/umbrel-seed/seed && sudo systemctl start umbrel-startup

and after that, this code:

sudo ./scripts/start

[louneskmt]

@henrique-diniz Be careful to not start manually Umbrel (using your last command). It skips SSD checks.

sudo systemctl start umbrel-startup already starts Umbrel and is part of the long command.

For people reading this: be careful with the above command (sent by @henrique-diniz). It resets your Umbrel, do not use it if you don't have a backup of your seed or if you have lightning channels opened.

[henrique-diniz]

@louneskmt, of course. It is just an alternative to solve this problem in my case 🤠

[lestephane]

@louneskmt will your reset command reset only LND? What's going to happen to bitcoind and, say, samourai server, if I had it installed prior to running your command.

[lestephane]

This is the result of the debug upload for me:

...
================
==== Result ====
================
This script could not automatically detect an issue with your Umbrel.
Please share the following links and paste it in the Umbrel Telegram group (https://t.me/getumbrel) so we can help you with your problem.
https://umbrel-paste.vercel.app/58041423-5eb9-4bb6-ad20-800c34e863a0

[bbenko]

I had the same problem, and it looks like Raspberry Pi 4 doesn't like my preowned external SSD (Seagate STGX2000400 2TB).

With the recommended SSD (Crucial MX500 1TB) and enclosure (ORICO 2.5) from https://getumbrel.com/#start, Umbrel now works excellent.

I did a quick dd test, duplicating blk00001.dat file on both hard drives connected to Raspberry Pi 4 and Dell XPS 13 laptop.

| | Raspberry Pi 4 | Dell XPS 13 |
| ------------- | ------------- | ------------- |
| Crucial + Orico | 3.7887 s, 35.4 MB/s | 0.574575 s, 234 MB/s |
| Seagate | 36.3014 s, 3.7 MB/s | 1.35346 s, 99.2 MB/s |

Example command:

umbrel@umbrel:~/umbrel/bitcoin/blocks $ dd if=blk00001.dat of=blk00001.dat.copy
262127+1 records in
262127+1 records out
134209213 bytes (134 MB, 128 MiB) copied, 3.7887 s, 35.4 MB/s

[lukechilds]

Thanks for reporting your results @bbenko!