Raspiblitz: LND service is not running

[RestartingAdmin]

Update to version 1.2 successful, system ran for some days now, no problems
Then LND stopped working, rebooting, re-flashing - no change
So here is debug output after reflashing and restarting the system:

https://termbin.com/ox25

[RestartingAdmin]

On the hardware issues: I use the power supply from hardware list, problems arose with heat, so I dismantled the case and use the raspi without. Temp in acceptable range now.

[lebowski36]

I got the same problem but for now had no time to reflash the sd.
I think it should be no problem with temp, because I have a fairly big heat sink plus open setup.

Just updated because of the new backup feature but in the future I will wait for the newer version to settle before updating :)

[fluidvoice]

I got the same problem but for now had no time to reflash the sd.
I think it should be no problem with temp, because I have a fairly big heat sink plus open setup.

Just updated because of the new backup feature but in the future I will wait for the newer version to settle before updating :)

FYI, "I had the same problem" comments will tend to get ignored. If you want help open a new issue and provide details and logs.

[fluidvoice]

Update to version 1.2 successful, system ran for some days now, no problems
Then LND stopped working, rebooting, re-flashing - no change
So here is debug output after reflashing and restarting the system:

https://termbin.com/ox25

In your log I saw an LND "edge not found" error. This problem I also hit and opened an issue with LND dev's but so far have not discovered the cause. The current theory is it's possibly a data corruption issue but we're not sure. Please provide links to these logs:

sudo cat /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log | nc termbin.com 9999
sudo cat /var/log/syslog | nc termbin.com 9999

[PatrickScheich]

A quite similar phenomenon here.
Updated from v1.1 to v1.2 (today) but LND never managed to start.

It seems to be caught in a loop.
...
2019-05-13 06:51:57.058 [INF] CRTR: FilteredChainView starting
2019-05-13 06:51:57.309 [ERR] SRVR: unable to start server: edge not found

as asked here the two logs
syslog: https://termbin.com/oufo (file has been too big so I did a sudo tail -n 1000 /var/log/syslog | nc termbin.com 9999)
lnd.log: https://termbin.com/441c

[rootzoll]

@fluidvoice From your experience - does lnd 0.6.1 fixes this "edge not found" error?

[rootzoll]

@fluidvoice To fix the "edge not found" - have you tried deleting the /mnt/hdd/lnd/data/graph and let LND rebuild it? I think that should be safe, because all wallet/channel data is in another directory.

EDIT: NO STOP - channel.db is that directory.

[RestartingAdmin]

Update to version 1.2 successful, system ran for some days now, no problems
Then LND stopped working, rebooting, re-flashing - no change
So here is debug output after reflashing and restarting the system:
https://termbin.com/ox25

In your log I saw an LND "edge not found" error. This problem I also hit and opened an issue with LND dev's but so far have not discovered the cause. The current theory is it's _possibly_ a data corruption issue but we're not sure. Please provide links to these logs:

sudo cat /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log | nc termbin.com 9999

https://termbin.com/xnko

sudo cat /var/log/syslog | nc termbin.com 9999

https://termbin.com/krfd

[bjadel]

@rootzoll

@fluidvoice To fix the "edge not found" - have you tried deleting the /mnt/hdd/lnd/data/graph and let LND rebuild it? I think that should be safe, because all wallet/channel data is in another directory.

EDIT: NO STOP - channel.db is that directory.

I have the same error "edge not found". Currently I am using the LND version 0.6.1 and the bug remains. Which file should I delete? I just want to make sure that I don't delete the wrong file. :-)

[rootzoll]

@bjadel DO NOT DELETE FILES WITHOUT MAKING A BACKUP. By deleting that directory I got LND to run again - showing your on-chain funds, but haveing no access to channel data/funds anymore. This this is not the fix you were looking for.

[fluidvoice]

@fluidvoice From your experience - does lnd 0.6.1 fixes this "edge not found" error?

@rootzoll No it does not.
@PatrickScheich and @RestartingAdmin were you ever running TOR?
My node was working upon upgrade to v1.2 then stopped some time after activating TOR but could be just a coincidence. Turning off TOR did not fix it.

[fluidvoice]

I have the same error "edge not found". Currently I am using the LND version 0.6.1 and the bug remains.

@bjadel are you, or were you, running TOR?
There is currently no known fix or workaround for this problem besides perhaps doing a complete re-install (after backing up of course)

[bjadel]

@fluidvoice Yeah, my node uses TOR. My channel.db had 137 errors. Apparently fixed it with "bbolt compact", but it still doesn't work. Should I deactivate the TOR service?

[PatrickScheich]

@fluidvoice To fix the "edge not found" - have you tried deleting the /mnt/hdd/lnd/data/graph and let LND rebuild it? I think that should be safe, because all wallet/channel data is in another directory.

EDIT: NO STOP - channel.db is that directory.

Well, I had have been too quick , and deleted

• /mnt/hdd/lnd/data/graph/mainnet/channel.db
• /mnt/hdd/lnd/data/graph/mainnet/sphinxreplay.db

after a reboot, lnd started to rebuild both files (which took around 12 hours)
as the rebuild had finished (nearly) everything works fine..

• bitcoind comes up fine
• lnd comes up fine

except that all my channels are gone (I guess I have to recover them from the seed words)

[PatrickScheich]

@fluidvoice From your experience - does lnd 0.6.1 fixes this "edge not found" error?

@rootzoll No it does not.
@PatrickScheich and @RestartingAdmin were you ever running TOR?
My node was working upon upgrade to v1.2 then stopped some time after activating TOR but could be just a coincidence. Turning off TOR did not fix it.

Yes, I once tried to switch my no to TOR (which worked fine)
then I switched it back to "non TOR" (which also works fine)
I used the "services" menu from raspiblitz for this

the non TOR version works for about 2..3 weeks, then I updated to v1.2 and lnd got stuck from the first moment

[PatrickScheich]

@bjadel DO NOT DELETE FILES WITHOUT MAKING A BACKUP. By deleting that directory I got LND to run again - showing your on-chain funds, but haveing no access to channel data/funds anymore. This this is not the fix you were looking for.

@rootzoll As I had done exactly that "deleting those files without making a backup upfront" I'm wondering if there is a procedure that will "bring back" my open channels? If not, I'm ok with that and will write off the approx. 200k sats in that channels.
So if I'm willing to accept at loss, would it be ok to to carry on with funding the lnd-wallet and creation new channels?

[rootzoll]

@PatrickScheich you really were quick following my bad advice - I think I had that idea just only for 15min without that "STOP" warning. Do you still have the channel.backup file? Just from seed there seems not to be chance to recover those channels.

[fluidvoice]

@fluidvoice From your experience - does lnd 0.6.1 fixes this "edge not found" error?

@rootzoll No it does not.
@PatrickScheich and @RestartingAdmin were you ever running TOR?
My node was working upon upgrade to v1.2 then stopped some time after activating TOR but could be just a coincidence. Turning off TOR did not fix it.

Yes, I once tried to switch my no to TOR (which worked fine)
then I switched it back to "non TOR" (which also works fine)
I used the "services" menu from raspiblitz for this

the non TOR version works for about 2..3 weeks, then I updated to v1.2 and lnd got stuck from the first moment

My experience was the same except I upgraded to v1.2 first. I was working. Then turned on TOR. Still working. Then turned off TOR and some time later it stopped working with the "edge not found" error in the log.

[fluidvoice]

Do you still have the channel.backup file?

I managed to create a channel.backup file. The problem is I cannot, so far, find where I stashed my seed phrase. So no way to recover my sat's without that right?

[rootzoll]

@fluidvoice your channel.backup file is encrypted with your private key from your seed phrase. So yes - you will need the seed phrase to make use of the channel.backup file.

[PatrickScheich]

@PatrickScheich you really were quick following my bad advice - I think I had that idea just only for 15min without that "STOP" warning.

Yes, I'm able of a very precise timing when it come to such things

Do you still have the channel.backup file? Just from seed there seems not to be chance to recover those channels.

sure there are channel.backup files at

• path:/mnt/hdd/lnd/data/chain/bitcoin/mainnet/channel.backup
  size:45 bytes
  last accesstime:Wed May 15 14:45:38.5320188400 2019

• path:/home/admin/.lnd/data/chain/bitcoin/mainnet/channel.backup
  size:45 bytes
  last accesstime:Wed May 15 14:45:48.4420175350 2019

but may they have been overwritten by the recreation of channel.db?
nevertheless I will give it a try, can you point me to the HowTo for the channel recovery procedure

[PatrickScheich]

@fluidvoice your channel.backup file is encrypted with your private key from your seed phrase. So yes - you will need the seed phrase to make use of the channel.backup file.

@rootzoll is there a way to get the content of channel.backup in "clear text", to see if a given channel.backup contain a particular set of channels?

[fluidvoice]

@rootzoll is there a way to get the content of channel.backup in "clear text", to see if a given channel.backup contain a particular set of channels?

see help for the LND command line utility lncli --help and the exportchanbackup --all and verifychanbackup commands shown there.
Otherwise I'd ask this question on a LND forum.

[fluidvoice]

I think there is around 7 nodes now reporting this "edge not found" error in the log file and where LND keeps stopping.
How many of you guys are running with TOR turned on, or once had it turned on, but now it's off?

[Z-and]

Hi @rootzoll ,

I seem to be one of the many also experiencing this "edge not found" problem and no longer am able to run certain commands on my node. However, I still have my backup seed and would like to somehow retrieve my funds (both onchain and offchain). What would be the best way to do this? Do I need to attempt to create the channel.backup file or was this automatically done when I upgraded to v1.2? Any help would be much appreciate since I have over 1+ BTC at stake >.>

[fluidvoice]

Any help would be much appreciate since I have over 1+ BTC at stake

yowza!

first verify if you already have a channel.backup file or not:
sudo ls -al /home/admin/.lnd/data/chain/bitcoin/mainnet/channel.backup

open one SSH connection to the node to monitor what LND is doing while you run commands, run:
sudo tail -f /mnt/hdd/lnd/logs/bitcoin/mainnet/lnd.log

open another SSH connection run LND commands:
lncli --help to get a list of commands and help on running them.
the exportchanbackup --all and verifychanbackup commands shown there.

run lncli command to create the channel.backup (I don't remember the exact syntax of the command but it's listed in the --help output) then save the channel.backup to a USB flash drive for safe keeping.

The key with doing this is that because LND stops running when it hits this "edge not found" error, you may have to restart LND several times to create the channel.backup before LND stops running (it has to be running to perform lncli commands). You can restart LND with the command sudo systemctl start lnd.service or sudo systemctl restart lnd.service and stop it with sudo systemctl stop lnd.service. What I saw in my case was that when the node has been running for a while the bitcoind server/daemon will keep syncing and so will be far ahead of the stopped LND syncing. So when you startup LND it will start syncing and will only hit the "edge not found" and stop running at the end of syncing. During this time (before it finishes syncing, and stops) you should might be able to run lncli to create the channel.backup and you'll be able to see when LND is syncing or has stopped by keeping that SSH window open running tail on it.
There might be other ways to create it. I'm not an expert. You might need to ask LND developers or other Raspiblitz users on the Telegram channel. @openoms @rootzoll any help?

[fluidvoice]

@Z-and had you been running TOR on the node?

[openoms]

more docs on the channel.backup https://github.com/lightningnetwork/lnd/pull/2313

syntax:
lncli --network=mainnet exportchanbackup --all --output_file=channel.backup

@Z-and
How many channels do you have?
(all will be force closed when restoring from the SCB)
Are there significant funds in the channels?
If not you can attempt to restore without the SCB only from the mnemonic seed with the desktop Lightning App. That gives you access to your on-chain funds.
https://github.com/rootzoll/raspiblitz/blob/master/FAQ.md#2-recover-from-wallet-seed

If you know your peers you can contact them to force close the channels, that way the funds are returned on-chain (unless they cheat).

[openoms]

@fluidvoice you mentioned that patch with which you could ignore the "edge not found" error when building from source.
That could be also attempted to use the existing LND dir, especially if there is no channel.backup or @Z-and wants to avoid force closing his channels.

[fluidvoice]

@openoms now that I have Raspiblitz running on x86 I'm hoping the LND dev's will be more active in giving me patches to try out to debug this problem: https://github.com/lightningnetwork/lnd/issues/3025#issuecomment-495587663
I'm also thinking about trying to build LND with the patch in my VM, but my guess is that this patch (ignoring the error) will not fix the problem, so I'd like the dev's to be engaged if possible.

[Z-and]

@fluidvoice No I wasn’t using Tor at any point

@opennoms I have 25 channels open and around 1 BTC funded throughout these channels, another .19 BTC on the onchain wallet.

I was able to retrieve the channel.backup file so I don’t think it’s necessary to try and generate a new one. I also have the backup seed. I’m not sure if I should just wait for a work around on the “edge not found” error or attempt to restore from backup. I’d prefer to take the safest route possible.

[openoms]

@fluidvoice it would worth a try trying to build from source with the patch

@Z-and
You are not in a bad position having both the seed and the channel backup. I would act rather soon either restoring (resulting to force closes) or trying to build LND from source with the patch (do you need help with that? - clue: the source build script is in the build_sdcard.sh, just commented out)
Your peers can can cheat you by attempting to close with an old state while you are offline.

[Z-and]

@fluidvoice it would worth a try trying to build from source with the patch

@Z-and
You are not in a bad position having both the seed and the channel backup. I would act rather soon either restoring (resulting to force closes) or trying to build LND from source with the patch (do you need help with that? - clue: the source build script is in the build_sdcard.sh, just commented out)
Your peers can can cheat you by attempting to close with an old state while you are offline.

Yeah I would definitely need some help if I were to build LND from source. What do you think is the safer route? Building from source or setting up a new node with mnemonic seed/channel.backup? I have an extra Pi setup so I can unplug my existing one and setup a new one.

[openoms]

I think both options are fairly safe in terms of getting your funds.
Restoring from seed + SBC will result in force closing all the channels.
Trying the LND version built with a patch (ignoring the edge not found error) would not introduce any new risk, but might not work (ignoring the error might not be enough). If it works you might be able to keep your channels open and regain control over them.

I am putting together a helper script which will update your LND with the version built form source+ patch. Just need to test it first.

[openoms]

Build from the source of my fork where the patch to ignore the "edge not found" error is commited.(https://github.com/openoms/lnd/commit/1a59596df3e8ae9e95a858cda33c329d4736d1bd).

Have a look at the script:
https://github.com/openoms/bitcoin-tutorials/blob/master/lnd.from.source.patched.sh

To be safe as possible make a backup of the lnd directory (https://github.com/rootzoll/raspiblitz/blob/master/FAQ.md#how-to-backup-my-lightning-node)
run:
/home/admin/config.scripts/lnd.rescue.sh backup

To update lnd with the patch run this line on the RaspiBlitz:
wget https://raw.githubusercontent.com/openoms/bitcoin-tutorials/master/lnd.from.source.patched.sh && sudo bash lnd.from.source.patched.sh

If successful should get this message on the end:

Fingers crossed it gets your channels back!

[fluidvoice]

Build from the source of my fork where the patch to ignore the "edge not found" error is commited.

Nice thanks for doing that. Will try it on my Atomic-Pi. Wait, this will work for Arm only or also x86?
Will try it. I built a patch before for Arm but did not lock the steps down in a script.
My VM setup keeps breaking due to UUID error when the external HDD /dev/sdx changes - internal details of how the .vmdk is handled.

[openoms]

Build from the source of my fork where the patch to ignore the "edge not found" error is commited.

Nice thanks for doing that. Will try it on my Atomic-Pi. Wait, this will work for Arm only or also x86?
Will try it. I built a patch before for Arm but did not lock the steps down in a script.
My VM setup keeps breaking due to UUID error when the external HDD /dev/sdx changes - internal details of how the .vmdk is handled.

I have tested it in the VM so will work on X86. Building from source is the same on all platforms. Go ahead and try it. It has similar affect to the update script to lnd v0.6.1 had which is in short: stop lnd, update, start .

[fluidvoice]

Build from the source of my fork where the patch to ignore the "edge not found" error is commited.

Nice thanks for doing that. Will try it on my Atomic-Pi. Wait, this will work for Arm only or also x86?
Will try it. I built a patch before for Arm but did not lock the steps down in a script.
My VM setup keeps breaking due to UUID error when the external HDD /dev/sdx changes - internal details of how the .vmdk is handled.

I have tested it in the VM so will work on X86. Building from source is the same on all platforms. Go ahead and try it. It has similar affect to the update script to lnd v0.6.1 had which is in short: stop lnd, update, start .

I keep breaking my VM setup. Need get it working then stop messing with it. Atm, I cannot SSH into it. Getting "connection refused" and cannot figure out why. Plus with the VM I cannot connect a keyboard and go to a terminal prompt with CTRL+ALT+F2 like you can with a physical device.

[openoms]

@fluidvoice please share the VM specific problems here: https://github.com/openoms/raspiblitz/issues/46

[bjadel]

@openoms thanks for the script
I got the following error:
channeldb/graph.go:3029:6: syntax error: unexpected return, expecting expression
Maybe a go version problem?
/usr/local/go/bin/go version go version go1.11 linux/arm

UPDATE: It's working now. I'll write more later.

[openoms]

Maybe a go version problem?
/usr/local/go/bin/go version go version go1.11 linux/arm

That is possible. I am running on Go v1.12.5. Coming with the the next v1.3 RaspiBlitz version or manual SDcard build.

[fluidvoice]

sudo bash lnd.from.source.patched.sh

It didn't seem to work, so I tried it by hand.
Note this is the same PITA stuff I had the first time I built a LND patch the dev's gave me:

$ go version
go version go1.12.5 linux/amd64
$ go get -d github.com/openoms/lnd

. cd .; git clone https://github.com/openoms/lnd /usr/local/gocode/src/github.com/openoms/lnd

fatal: could not create work tree dir '/usr/local/gocode/src/github.com/openoms/lnd': Permission denied
package github.com/openoms/lnd: exit status 128
$ sudo chown -R admin:admin /usr/local/gocode/src/
$ go get -d github.com/openoms/lnd

. cd .; git clone https://github.com/btcsuite/btclog /usr/local/gocode/src/github.com/btcsuite/btclog

Cloning into '/usr/local/gocode/src/github.com/btcsuite/btclog'...
fatal: unable to access 'https://github.com/btcsuite/btclog/': Failed to connect to github.com port 443: Connection timed out
package github.com/btcsuite/btclog: exit status 128

. cd .; git clone https://github.com/btcsuite/websocket /usr/local/gocode/src/github.com/btcsuite/websocket

Cloning into '/usr/local/gocode/src/github.com/btcsuite/websocket'...
fatal: unable to access 'https://github.com/btcsuite/websocket/': Failed to connect to github.com port 443: Connection timed out
package github.com/btcsuite/websocket: exit status 128

. cd .; git clone https://github.com/btcsuite/btcwallet /usr/local/gocode/src/github.com/btcsuite/btcwallet

Cloning into '/usr/local/gocode/src/github.com/btcsuite/btcwallet'...
fatal: unable to access 'https://github.com/btcsuite/btcwallet/': Failed to connect to github.com port 443: Connection timed out
package github.com/btcsuite/btcwallet/chain: exit status 128
package github.com/btcsuite/btcwallet/waddrmgr: cannot find package "github.com/btcsuite/btcwallet/waddrmgr" in any of:
/usr/local/go/src/github.com/btcsuite/btcwallet/waddrmgr (from $GOROOT)
/usr/local/gocode/src/github.com/btcsuite/btcwallet/waddrmgr (from $GOPATH)
package github.com/btcsuite/btcwallet/wallet: cannot find package "github.com/btcsuite/btcwallet/wallet" in any of:
/usr/local/go/src/github.com/btcsuite/btcwallet/wallet (from $GOROOT)
/usr/local/gocode/src/github.com/btcsuite/btcwallet/wallet (from $GOPATH)
package github.com/btcsuite/btcwallet/wallet/txauthor: cannot find package "github.com/btcsuite/btcwallet/wallet/txauthor" in any of:
/usr/local/go/src/github.com/btcsuite/btcwallet/wallet/txauthor (from $GOROOT)
/usr/local/gocode/src/github.com/btcsuite/btcwallet/wallet/txauthor (from $GOPATH)
package github.com/btcsuite/btcwallet/walletdb: cannot find package "github.com/btcsuite/btcwallet/walletdb" in any of:
/usr/local/go/src/github.com/btcsuite/btcwallet/walletdb (from $GOROOT)
/usr/local/gocode/src/github.com/btcsuite/btcwallet/walletdb (from $GOPATH)
$ sudo make build
/bin/sh: 1: go: not found
/bin/sh: 1: go: not found
Building debug lnd and lncli.
GO111MODULE=on go build -v -tags="dev" -o lnd-debug -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.6.1-beta-99-g1a59596df3e8ae9e95a858cda33c329d4736d1bd" github.com/lightningnetwork/lnd/cmd/lnd
/bin/sh: 1: go: not found
Makefile:94: recipe for target 'build' failed
make: * [build] Error 127

admin@atomicpi:/usr/local/gocode/src/github.com/openoms/lnd$ /usr/local/go/bin/go build -v -tags="dev" -o lnd-debug -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.6.1-beta-99-g1a59596df3e8ae9e95a858cda33c329d4736d1bd" github.com/lightningnetwork/lnd/cmd/lnd
go: finding github.com/golang/protobuf v1.3.1
go: finding github.com/btcsuite/btcd v0.0.0-20190426011420-63f50db2f70a
go: finding github.com/coreos/bbolt v1.3.2
go: finding github.com/NebulousLabs/go-upnp v0.0.0-20180202185039-29b680b06c82
go: finding github.com/NebulousLabs/fastrand v0.0.0-20180208210444-3cf7173006a0
go: finding github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af
go: finding github.com/davecgh/go-spew v1.1.1
go: finding github.com/btcsuite/btcwallet v0.0.0-20190424224017-9d95f76e99a7
go: finding github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
go: finding git.schwanenlied.me/yawning/bsaes.git v0.0.0-20180720073208-c0276d75487e
go: finding github.com/juju/utils v0.0.0-20180820210520-bf9cc5bdd62d
go: finding github.com/kkdai/bstream v0.0.0-20181106074824-b3251f7901ec
go: github.com/btcsuite/[email protected]: git fetch -f origin refs/heads/:refs/heads/ refs/tags/:refs/tags/ in /home/admin/go/pkg/mod/cache/vcs/470589a49e6ebd7f1d95b67163a08e62c5d4731d935adb870df00f1c13052788: exit status 128:
fatal: unable to access 'https://github.com/btcsuite/btcwallet/': Failed to connect to github.com port 443: Connection timed out
go: finding github.com/miekg/dns v0.0.0-20171125082028-79bfde677fa8
go: finding github.com/juju/testing v0.0.0-20180920084828-472a3e8b2073
go: finding github.com/jessevdk/go-flags v1.4.0
go: github.com/kkdai/[email protected]: git fetch -f origin refs/heads/:refs/heads/ refs/tags/:refs/tags/ in /home/admin/go/pkg/mod/cache/vcs/7e5aebdccb149c9a56d8bd4d8479e5bf8d2b3fd47ffcb6558fb613504c69cacf: exit status 128:
fatal: unable to access 'https://github.com/kkdai/bstream/': Failed to connect to github.com port 443: Connection timed out
go: finding github.com/juju/retry v0.0.0-20180821225755-9058e192b216
go: finding github.com/btcsuite/fastsha256 v0.0.0-20160815193821-637e65642941
go: github.com/coreos/[email protected]: git fetch -f origin refs/heads/:refs/heads/ refs/tags/:refs/tags/ in /home/admin/go/pkg/mod/cache/vcs/a694344f44d9867839bcf500215b37dfe9c765a4a307a88fe6690821014a4d84: exit status 128:
fatal: unable to access 'https://github.com/coreos/bbolt/': Failed to connect to github.com port 443: Connection timed out
go: github.com/jessevdk/[email protected]: unknown revision v1.4.0
go: github.com/NebulousLabs/[email protected]: git fetch -f origin refs/heads/:refs/heads/ refs/tags/:refs/tags/ in /home/admin/go/pkg/mod/cache/vcs/c94b6db2eeb78cb24c6f6836c92cf43dabb86d7966c7be56e28eca613b3bfd9e: exit status 128:
fatal: unable to access 'https://github.com/NebulousLabs/fastrand/': Failed to connect to github.com port 443: Connection timed out
Fetching https://gopkg.in/errgo.v1?go-get=1
go: github.com/btcsuite/[email protected]: git fetch -f origin refs/heads/:refs/heads/ refs/tags/:refs/tags/ in /home/admin/go/pkg/mod/cache/vcs/e68333caf77fed837042d7dec72ca9ca0d1002dd65e1ef2f13acd9bbac3cd01b: exit status 128:
fatal: unable to access 'https://github.com/btcsuite/btcd/': Failed to connect to github.com port 443: Connection timed out
go: github.com/davecgh/[email protected]: git fetch -f origin refs/heads/:refs/heads/ refs/tags/:refs/tags/ in /home/admin/go/pkg/mod/cache/vcs/b9a4b9bbdb4a59723f2348415ad7ffda91568455a1cfd92e97976132bdfbaf57: exit status 128:
fatal: unable to access 'https://github.com/davecgh/go-spew/': Failed to connect to github.com port 443: Connection timed out
Fetching https://gopkg.in/macaroon-bakery.v2?go-get=1
go: github.com/rogpeppe/[email protected]: git fetch -f origin refs/heads/:refs/heads/ refs/tags/:refs/tags/ in /home/admin/go/pkg/mod/cache/vcs/afcc5aad51842604d2269ffe093be1298f7a42f42a582c2a61a24530bb334a9a: exit status 128:
fatal: unable to access 'https://github.com/rogpeppe/fastuuid/': Failed to connect to github.com port 443: Connection timed out
go: finding github.com/urfave/cli v1.18.0
go: finding github.com/lightninglabs/neutrino v0.0.0-20190521105038-03deaa76ddc2
go: finding github.com/juju/clock v0.0.0-20180808021310-bab88fc67299
go: finding github.com/jackpal/go-nat-pmp v0.0.0-20170405195558-28a68d0c24ad
Parsing meta tags from https://gopkg.in/macaroon-bakery.v2?go-get=1 (status code 200)
get "gopkg.in/macaroon-bakery.v2": found meta tag get.metaImport{Prefix:"gopkg.in/macaroon-bakery.v2", VCS:"git", RepoRoot:"https://gopkg.in/macaroon-bakery.v2"} at https://gopkg.in/macaroon-bakery.v2?go-get=1
Parsing meta tags from https://gopkg.in/errgo.v1?go-get=1 (status code 200)
get "gopkg.in/errgo.v1": found meta tag get.metaImport{Prefix:"gopkg.in/errgo.v1", VCS:"git", RepoRoot:"https://gopkg.in/errgo.v1"} at https://gopkg.in/errgo.v1?go-get=1
go: finding gopkg.in/macaroon-bakery.v2 v2.0.1
go: finding gopkg.in/errgo.v1 v1.0.0
go: finding github.com/juju/version v0.0.0-20180108022336-b64dbd566305
go: finding github.com/ltcsuite/ltcd v0.0.0-20190101042124-f37f8bf35796
Fetching https://google.golang.org/grpc?go-get=1
Parsing meta tags from https://google.golang.org/grpc?go-get=1 (status code 200)
get "google.golang.org/grpc": found meta tag get.metaImport{Prefix:"google.golang.org/grpc", VCS:"git", RepoRoot:"https://github.com/grpc/grpc-go"} at https://google.golang.org/grpc?go-get=1
go: finding google.golang.org/grpc v1.18.0
go: finding github.com/lightningnetwork/lightning-onion v0.0.0-20190430041606-751fb4dd8b72
go: finding github.com/go-errors/errors v1.0.1
Fetching https://google.golang.org/genproto?go-get=1
go: finding github.com/grpc-ecosystem/grpc-gateway v0.0.0-20170724004829-f2862b476edc
Parsing meta tags from https://google.golang.org/genproto?go-get=1 (status code 200)
get "google.golang.org/genproto": found meta tag get.metaImport{Prefix:"google.golang.org/genproto", VCS:"git", RepoRoot:"https://github.com/google/go-genproto"} at https://google.golang.org/genproto?go-get=1
go: finding google.golang.org/genproto v0.0.0-20190201180003-4b09977fb922
go: finding github.com/juju/errors v0.0.0-20181118221551-089d3ea4e4d5
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
get "golang.org/x/net": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net?go-get=1
go: finding golang.org/x/net v0.0.0-20190206173232-65e2d4e15006
go: finding github.com/jackpal/gateway v1.0.5

.... in the end it fails

[fluidvoice]

UPDATE: It's working now. I'll write more later.

@bjadel how? And more importantly did it allow you to get past the "edge not found" error?

[fluidvoice]

tried a 2nd time, still errors:

$ sudo bash lnd.from.source.patched.sh
* Build LND from Source *
repo=github.com/openoms/lnd
up to the commit=1a59596df3e8ae9e95a858cda33c329d4736d1bd

. cd .; git clone https://github.com/davecgh/go-spew /usr/local/gocode/src/github.com/davecgh/go-spew

Cloning into '/usr/local/gocode/src/github.com/davecgh/go-spew'...
fatal: unable to access 'https://github.com/davecgh/go-spew/': Failed to connect to github.com port 443: Connection timed out
package github.com/davecgh/go-spew/spew: exit status 128

[openoms]

tried a 2nd time, still errors:

$ sudo bash lnd.from.source.patched.sh
* Build LND from Source *
repo=github.com/openoms/lnd
up to the commit=1a59596df3e8ae9e95a858cda33c329d4736d1bd

. cd .; git clone https://github.com/davecgh/go-spew /usr/local/gocode/src/github.com/davecgh/go-spew

Cloning into '/usr/local/gocode/src/github.com/davecgh/go-spew'...
fatal: unable to access 'https://github.com/davecgh/go-spew/': Failed to connect to github.com port 443: Connection timed out
package github.com/davecgh/go-spew/spew: exit status 128

well done for solving the file permission problem. but you still clearly have internet connection issues.
If you are in a VM try setting your network adapter to NAT.
Port forward for ssh (set up on the host machine):
$ VBoxManage modifyvm --natpf1 "ssh,tcp,,3022,,22"
Connect from a host terminal:
$ ssh [email protected] -p 3022

You can update lnd before connecting a HDD to avoid the problems with .vmdk.

[fluidvoice]

If you are in a VM try setting your network adapter to NAT.
Port forward for ssh (set up on the host machine):
$ VBoxManage modifyvm --natpf1 "ssh,tcp,,3022,,22"
Connect from a host terminal:
$ ssh [email protected] -p 3022

You can update lnd before connecting a HDD to avoid the problems with .vmdk.

I'm using the Atomic Pi with a USB wifi dongle. I don't have a long enough ethernet cable to reach my router. The connection is good enough to keep bitcoind synced. I also shut down the bitcoind service to reduce network traffic during the build. Still doesn't work.

[fluidvoice]

Trying again with explicit/manual command pulled from the Makefile.
Clearly the wifi network connection is working sufficiently - for at least part of the build:

Edit: I wanna see if the errors always happen in the same place?
$ grep Failed errors.txt

fatal: unable to access 'https://github.com/davecgh/go-spew/': Failed to connect to github.com port 443: Connection timed out
fatal: unable to access 'https://github.com/btcsuite/btcutil/': Failed to connect to github.com port 443: Connection timed out
fatal: unable to access 'https://github.com/ltcsuite/ltcd/': Failed to connect to github.com port 443: Connection timed out
fatal: unable to access 'https://github.com/kr/pretty/': Failed to connect to github.com port 443: Connection timed out
fatal: unable to access 'https://github.com/rogpeppe/fastuuid/': Failed to connect to github.com port 443: Connection timed out

admin@atomicpi:/usr/local/gocode/src/github.com/openoms/lnd$ sudo GO111MODULE=on /usr/local/go/bin/go build -v -tags="dev-patch-edgenotfound" -o lnd-debug -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.6.1-beta-99-g1a59596df3e8ae9e95a858cda33c329d4736d1bd" github.com/lightningnetwork/lnd/cmd/lnd
go: finding github.com/NebulousLabs/fastrand v0.0.0-20180208210444-3cf7173006a0
go: finding github.com/ltcsuite/ltcd v0.0.0-20190101042124-f37f8bf35796
go: finding github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d
go: finding github.com/btcsuite/btcwallet v0.0.0-20190424224017-9d95f76e99a7
go: finding github.com/Yawning/aez v0.0.0-20180114000226-4dad034d9db2
Fetching https://golang.org/x/crypto?go-get=1
go: finding github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495
go: finding github.com/btcsuite/btcd v0.0.0-20181130015935-7d2daa5bfef2
Fetching https://google.golang.org/grpc?go-get=1
go: finding github.com/coreos/bbolt v1.3.2
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
get "golang.org/x/crypto": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://google.golang.org/grpc?go-get=1 (status code 200)
go: finding golang.org/x/crypto v0.0.0-20190103213133-ff983b9c42bc
get "google.golang.org/grpc": found meta tag get.metaImport{Prefix:"google.golang.org/grpc", VCS:"git", RepoRoot:"https://github.com/grpc/grpc-go"} at https://google.golang.org/grpc?go-get=1
go: finding google.golang.org/grpc v1.18.0
go: finding github.com/kr/pretty v0.1.0
go: finding github.com/juju/clock v0.0.0-20180808021310-bab88fc67299
go: finding github.com/btcsuite/btcutil v0.0.0-20180706230648-ab6388e0c60a
Fetching https://golang.org/x/exp?go-get=1
Parsing meta tags from https://golang.org/x/exp?go-get=1 (status code 200)
get "golang.org/x/exp": found meta tag get.metaImport{Prefix:"golang.org/x/exp", VCS:"git", RepoRoot:"https://go.googlesource.com/exp"} at https://golang.org/x/exp?go-get=1
go: finding golang.org/x/exp v0.0.0-20190121172915-509febef88a4
Fetching https://golang.org/x/lint?go-get=1
Parsing meta tags from https://golang.org/x/lint?go-get=1 (status code 200)
get "golang.org/x/lint": found meta tag get.metaImport{Prefix:"golang.org/x/lint", VCS:"git", RepoRoot:"https://go.googlesource.com/lint"} at https://golang.org/x/lint?go-get=1
go: finding golang.org/x/lint v0.0.0-20180702182130-06c8688daad7
go: finding github.com/lightninglabs/gozmq v0.0.0-20180324010646-462a8a753885
go: finding github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792
go: github.com/davecgh/[email protected]: git fetch -f https://github.com/davecgh/go-spew refs/heads/:refs/heads/ refs/tags/:refs/tags/ in /home/admin/go/pkg/mod/cache/vcs/b9a4b9bbdb4a59723f2348415ad7ffda91568455a1cfd92e97976132bdfbaf57: exit status 128:
fatal: unable to access 'https://github.com/davecgh/go-spew/': Failed to connect to github.com port 443: Connection timed out
Fetching https://golang.org/x/sync?go-get=1
Parsing meta tags from https://golang.org/x/sync?go-get=1 (status code 200)
get "golang.org/x/sync": found meta tag get.metaImport{Prefix:"golang.org/x/sync", VCS:"git", RepoRoot:"https://go.googlesource.com/sync"} at https://golang.org/x/sync?go-get=1
go: finding golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: finding github.com/davecgh/go-spew v1.1.1
go: github.com/davecgh/[email protected]: unknown revision v1.1.1
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
get "golang.org/x/sys": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys?go-get=1
go: finding golang.org/x/sys v0.0.0-20190102155601-82a175fd1598
go: github.com/btcsuite/[email protected]: git -c protocol.version=0 fetch --unshallow -f https://github.com/btcsuite/btcutil refs/heads/:refs/heads/ refs/tags/:refs/tags/ in /home/admin/go/pkg/mod/cache/vcs/32d165082fc1fd887ffb8b2f499b1b568f548c19a985f5cf7838726192e3df1d: exit status 128:
fatal: unable to access 'https://github.com/btcsuite/btcutil/': Failed to connect to github.com port 443: Connection timed out
go: finding github.com/golang/protobuf v1.2.0
Fetching https://gopkg.in/check.v1?go-get=1
go: finding github.com/tv42/zbase32 v0.0.0-20160707012821-501572607d02
go: finding github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da
Parsing meta tags from https://gopkg.in/check.v1?go-get=1 (status code 200)
get "gopkg.in/check.v1": found meta tag get.metaImport{Prefix:"gopkg.in/check.v1", VCS:"git", RepoRoot:"https://gopkg.in/check.v1"} at https://gopkg.in/check.v1?go-get=1
go: finding gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
Fetching https://honnef.co/go/tools?go-get=1
go: finding github.com/btcsuite/golangcrypto v0.0.0-20150304025918-53f62d9b43e8
Parsing meta tags from https://honnef.co/go/tools?go-get=1 (status code 200)
get "honnef.co/go/tools": found meta tag get.metaImport{Prefix:"honnef.co/go/tools", VCS:"git", RepoRoot:"https://github.com/dominikh/go-tools"} at https://honnef.co/go/tools?go-get=1
go: finding honnef.co/go/tools v0.0.0-20180728063816-88497007e858
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
get "golang.org/x/net": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net?go-get=1
go: finding golang.org/x/net v0.0.0-20181106065722-10aee1819953
go: finding github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af
go: finding github.com/lightninglabs/neutrino v0.0.0-20190313035638-e1ad4c33fb18
go: finding google.golang.org/grpc v1.16.0
go: finding github.com/btcsuite/btcutil v0.0.0-20190207003914-4c204d697803
go: github.com/btcsuite/[email protected]: unknown revision 4c204d697803
go: finding github.com/btcsuite/btcd v0.0.0-20190213025234-306aecffea32
go: finding github.com/btcsuite/btcd v0.0.0-20190426011420-63f50db2f70a
Fetching https://golang.org/x/tools?go-get=1
go: finding github.com/jessevdk/go-flags v1.4.0
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
get "golang.org/x/tools": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools?go-get=1
go: finding golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52
Fetching https://gopkg.in/yaml.v2?go-get=1
go: finding golang.org/x/sys v0.0.0-20190209173611-3b5209105503
go: finding github.com/kkdai/bstream v0.0.0-20181106074824-b3251f7901ec
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
go: finding gopkg.in/yaml.v2 v2.2.2

[fluidvoice]

To remove potential wifi problems of the Atomic-Pi, tried building on my notebook (though still WiFi as it does not have ethernet) with just env changes to the build script. It still gets "port 443: Connection timed out" errors.

. export GOROOT=/usr/local/go

export GOROOT=/usr/bin

. export PATH=$PATH:$GOROOT/bin

. export GOPATH=/usr/local/gocode

export GOPATH=/home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found
export PATH=$PATH:$GOPATH/bin

$ which go
/usr/bin/go
$ go version
go version go1.12.4 linux/amd64

$ sudo bash lnd.from.source.patched.sh

* Build LND from Source *

repo=github.com/openoms/lnd
up to the commit=1a59596df3e8ae9e95a858cda33c329d4736d1bd
package bytes: unrecognized import path "bytes" (import path does not begin with hostname)
package container/list: unrecognized import path "container/list" (import path does not begin with hostname)
package context: unrecognized import path "context" (import path does not begin with hostname)
package crypto/ecdsa: unrecognized import path "crypto/ecdsa" (import path does not begin with hostname)
package crypto/elliptic: unrecognized import path "crypto/elliptic" (import path does not begin with hostname)
package crypto/rand: unrecognized import path "crypto/rand" (import path does not begin with hostname)
package crypto/sha256: unrecognized import path "crypto/sha256" (import path does not begin with hostname)
package crypto/tls: unrecognized import path "crypto/tls" (import path does not begin with hostname)
package crypto/x509: unrecognized import path "crypto/x509" (import path does not begin with hostname)
package crypto/x509/pkix: unrecognized import path "crypto/x509/pkix" (import path does not begin with hostname)
package encoding/binary: unrecognized import path "encoding/binary" (import path does not begin with hostname)
package encoding/hex: unrecognized import path "encoding/hex" (import path does not begin with hostname)
package encoding/pem: unrecognized import path "encoding/pem" (import path does not begin with hostname)
package errors: unrecognized import path "errors" (import path does not begin with hostname)
package fmt: unrecognized import path "fmt" (import path does not begin with hostname)

. cd .; git clone https://github.com/btcsuite/btcd /home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found/src/github.com/btcsuite/btcd

Cloning into '/home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found/src/github.com/btcsuite/btcd'...
fatal: unable to access 'https://github.com/btcsuite/btcd/': Failed to connect to github.com port 443: Connection timed out
package github.com/btcsuite/btcd/blockchain: exit status 128
package github.com/btcsuite/btcd/btcec: cannot find package "github.com/btcsuite/btcd/btcec" in any of:
/usr/bin/src/github.com/btcsuite/btcd/btcec (from $GOROOT)
/home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found/src/github.com/btcsuite/btcd/btcec (from $GOPATH)
package github.com/btcsuite/btcd/chaincfg: cannot find package "github.com/btcsuite/btcd/chaincfg" in any of:
/usr/bin/src/github.com/btcsuite/btcd/chaincfg (from $GOROOT)
/home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found/src/github.com/btcsuite/btcd/chaincfg (from $GOPATH)
package github.com/btcsuite/btcd/chaincfg/chainhash: cannot find package "github.com/btcsuite/btcd/chaincfg/chainhash" in any of:
/usr/bin/src/github.com/btcsuite/btcd/chaincfg/chainhash (from $GOROOT)
/home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found/src/github.com/btcsuite/btcd/chaincfg/chainhash (from $GOPATH)
package github.com/btcsuite/btcd/connmgr: cannot find package "github.com/btcsuite/btcd/connmgr" in any of:
/usr/bin/src/github.com/btcsuite/btcd/connmgr (from $GOROOT)
/home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found/src/github.com/btcsuite/btcd/connmgr (from $GOPATH)
package github.com/btcsuite/btcd/rpcclient: cannot find package "github.com/btcsuite/btcd/rpcclient" in any of:
/usr/bin/src/github.com/btcsuite/btcd/rpcclient (from $GOROOT)
/home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found/src/github.com/btcsuite/btcd/rpcclient (from $GOPATH)
package github.com/btcsuite/btcd/txscript: cannot find package "github.com/btcsuite/btcd/txscript" in any of:
/usr/bin/src/github.com/btcsuite/btcd/txscript (from $GOROOT)
/home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found/src/github.com/btcsuite/btcd/txscript (from $GOPATH)
package github.com/btcsuite/btcd/wire: cannot find package "github.com/btcsuite/btcd/wire" in any of:
/usr/bin/src/github.com/btcsuite/btcd/wire (from $GOROOT)
/home/brad/working/raspiblitz/dev/lnd/patches/edge-not-found/src/github.com/btcsuite/btcd/wire (from $GOPATH)

[fluidvoice]

Build from the source of my fork where the patch to ignore the "edge not found" error is commited.

Nice thanks for doing that. Will try it on my Atomic-Pi. Wait, this will work for Arm only or also x86?
Will try it. I built a patch before for Arm but did not lock the steps down in a script.
My VM setup keeps breaking due to UUID error when the external HDD /dev/sdx changes - internal details of how the .vmdk is handled.

I have tested it in the VM so will work on X86. Building from source is the same on all platforms. Go ahead and try it. It has similar affect to the update script to lnd v0.6.1 had which is in short: stop lnd, update, start .

I am putting together a helper script which will update your LND with the version built form source+ patch. Just need to test it first.

@openoms you tested running the script, but did you actually run the resulting binaries? I don't even see where they are. It seems it built only *.go files? Where are the lnd and lncli binaries?

[Z-and]

@openoms thanks for the script
I got the following error:
channeldb/graph.go:3029:6: syntax error: unexpected return, expecting expression
Maybe a go version problem?
/usr/local/go/bin/go version go version go1.11 linux/arm

UPDATE: It's working now. I'll write more later.

Please share when you get a chance, I'd like to give this a shot.

[fluidvoice]

@openoms thanks for the script
I got the following error:
channeldb/graph.go:3029:6: syntax error: unexpected return, expecting expression
Maybe a go version problem?
/usr/local/go/bin/go version go version go1.11 linux/arm
UPDATE: It's working now. I'll write more later.

Please share when you get a chance, I'd like to give this a shot.

update on this problem: https://github.com/rootzoll/raspiblitz/issues/562#issuecomment-497939315
tl;dr fix is to avoid the problem but does not fix nodes already with this error. Running the above patch might work or not. It would be nice to try if anyone can get it to build. Otherwise reinstall/restore is in order.

[openoms]

Build from the source of my fork where the patch to ignore the "edge not found" error is commited.(openoms/lnd@1a59596).

Have a look at the script:
https://github.com/openoms/bitcoin-tutorials/blob/master/lnd.from.source.patched.sh

To be safe as possible make a backup of the lnd directory (https://github.com/rootzoll/raspiblitz/blob/master/FAQ.md#how-to-backup-my-lightning-node)
run:
/home/admin/config.scripts/lnd.rescue.sh backup

To update lnd with the patch run this line on the RaspiBlitz:
wget https://raw.githubusercontent.com/openoms/bitcoin-tutorials/master/lnd.from.source.patched.sh && sudo bash lnd.from.source.patched.sh

If successful should get this message on the end:

Fingers crossed it gets your channels back!

I have modified the patch, please test again. @fluidvoice @Z-and @bjadel
Renamed the file to avoid conflict, new command to run on your RaspiBlitz:

$ wget https://raw.githubusercontent.com/openoms/bitcoin-tutorials/master/lnd.from.source.patched.corr.sh && sudo bash lnd.from.source.patched.corr.sh

Output, which is looking better now:
https://pastebin.com/raw/xzqU7jYE

[openoms]

TL;DR: the patch does not help.

Got the patched lnd running on a node with the error:

Still fails due to [ERR] SRVR: unable to start server: edge not found

2019-06-02 19:15:18.284 [INF] BRAR: Starting contract observer, watching for breaches.
2019-06-02 19:15:18.284 [INF] DISC: Authenticated Gossiper is starting
2019-06-02 19:15:18.284 [INF] NTFN: New block epoch subscription
2019-06-02 19:15:18.616 [INF] CRTR: FilteredChainView starting

2019-06-02 19:15:20.489 [ERR] SRVR: unable to start server: edge not found

2019-06-02 19:15:20.489 [INF] RPCS: Stopping RPC Server
2019-06-02 19:15:20.490 [INF] RPCS: Stopping SignRPC Sub-RPC Server
2019-06-02 19:15:20.490 [INF] RPCS: Stopping ChainRPC Sub-RPC Server
2019-06-02 19:15:20.490 [INF] RPCS: Stopping InvoicesRPC Sub-RPC Server
2019-06-02 19:15:20.490 [INF] RPCS: Stopping WalletKitRPC Sub-RPC Server
2019-06-02 19:15:20.500 [INF] LTND: Shutdown complete

Ignoring the error did not help, this might suggest that the database is corrupted irreversibly.

Will proceed with restoring from the seed and channel.backup file on that node (https://github.com/rootzoll/raspiblitz/blob/master/FAQ.md#2-recover-from-wallet-seed)

There is a fix to prevent this happening already in the lnd source, coming in the next release: https://github.com/rootzoll/raspiblitz/issues/638

[fluidvoice]

pretty much what I thought. Dev thought it was a long shot too.
You have this problem too, or that output was from another user?

[openoms]

@fluidvoice No, it is from one of the already known nodes. Not adding to this number luckily.

[jodobear]

here's another node struck by edge not found
logs: https://termbin.6bn36/com
lng.log: https://termbin.com/qlx7
syslog: https://termbin.com/7tu8

TL;DR: the patch does not help.

Got the patched lnd running on a node with the error:

Still fails due to [ERR] SRVR: unable to start server: edge not found

2019-06-02 19:15:18.284 [INF] BRAR: Starting contract observer, watching for breaches.
2019-06-02 19:15:18.284 [INF] DISC: Authenticated Gossiper is starting
2019-06-02 19:15:18.284 [INF] NTFN: New block epoch subscription
2019-06-02 19:15:18.616 [INF] CRTR: FilteredChainView starting

2019-06-02 19:15:20.489 [ERR] SRVR: unable to start server: edge not found

2019-06-02 19:15:20.489 [INF] RPCS: Stopping RPC Server
2019-06-02 19:15:20.490 [INF] RPCS: Stopping SignRPC Sub-RPC Server
2019-06-02 19:15:20.490 [INF] RPCS: Stopping ChainRPC Sub-RPC Server
2019-06-02 19:15:20.490 [INF] RPCS: Stopping InvoicesRPC Sub-RPC Server
2019-06-02 19:15:20.490 [INF] RPCS: Stopping WalletKitRPC Sub-RPC Server
2019-06-02 19:15:20.500 [INF] LTND: Shutdown complete

Ignoring the error did not help, this might suggest that the database is corrupted irreversibly.

Will proceed with restoring from the seed and channel.backup file on that node (https://github.com/rootzoll/raspiblitz/blob/master/FAQ.md#2-recover-from-wallet-seed)

There is a fix to prevent this happening already in the lnd source, coming in the next release: #638

So, we don't have a solution for this yet. And even if rebuilt from source it will persist. So, we can only wait until it's resolved? Am i missing something here?

And thanks @openoms @fluidvoice

[openoms]

So, we don't have a solution for this yet. And even if rebuilt from source it will persist. So, we can only wait until it's resolved? Am i missing something here?

Unfortunately from my experience this error means that there is database corruption, which we were unable to fix by ignoring the error.
It seems to happen when LND is not shut down correctly. Could be caused by a power outage or hard reset without stopping services (I am speculating here).

The new version of LND (built up from source to a recent commit: https://github.com/rootzoll/raspiblitz/issues/638#issuecomment-498135968) is fixing the problem by preventing it to occur, but cannot fix the faulty nodes.

The way to go to my best knowledge is to restore the lightning wallet from seed+SCB: https://github.com/rootzoll/raspiblitz/blob/master/FAQ.md#2-recover-from-wallet-seed
The onchain funds tend not to reappear on the RaspiBlitz, in this case you can use the Desktop Lightning App.
Also there is a problem if there was a passphrase used on top of the seed as the Lightning App is not compatible with that yet. See the issue: https://github.com/rootzoll/raspiblitz/issues/632

[jodobear]

Unfortunately from my experience this error means that there is database corruption, which we were unable to fix by ignoring the error.
It seems to happen when LND is not shut down correctly. Could be caused by a power outage or hard reset without stopping services (I am speculating here).

I doubt that was the case with mine as, my blitz stays in an undisturbed location with consistent power and suddenly in the evening it showed this error.

The way to go to my best knowledge is to restore the lightning wallet from seed+SCB: https://github.com/rootzoll/raspiblitz/blob/master/FAQ.md#2-recover-from-wallet-seed
The onchain funds tend not to reappear on the RaspiBlitz, in this case you can use the Desktop Lightning App.

Anyone tried this and recovered on-chain funds and channels intact? I have the seed, passphrase for seed and channel.backup file.

Also there is a problem if there was a passphrase used on top of the seed as the Lightning App is not compatible with that yet. See the issue: #632

What about Zap desktop? Anyone tried it here?

I think i'm gonna try to recover via seed+SCB and see what happens.

[openoms]

@jodobear
I don't know about Zap desktop, you can try and please report the outcome if you do.

To restore the from seed+SCB follow the steps described in the FAQ.

If your on-chain funds would be missing you can try this newly found method:
https://github.com/rootzoll/raspiblitz/issues/632#issuecomment-502386886

[jodobear]

@jodobear
I don't know about Zap desktop, you can try and please report the outcome if you do.

To restore the from seed+SCB follow the steps described in the FAQ.

@openoms
tl;dr: no channel or onchain funds recovered.

So, i finally recovered my seed from the excellent location i'd kept in and managed to recover my node. Steps followed:

1. Flashed SD Card with raspiblitz 1.3.
2. Cleaned the HDD.
3. Loaded the SD and fired up the pi with clean HDD.
4. Followed the seed+SCB backup procedure with copying the chain data from another computer.
5. Selected Oldto load the channel.backup file.
6. After lnd sync (took 2 days) i ssh in and it tells me LND accepted channel.backup. It will take 1 hour till you see your channels - that was last night.

I haven't recovered anything. Did i do anything wrong? Should i not have done the recovery process with 1.3 and instead with raspiblitz 1.2

If your on-chain funds would be missing you can try this newly found method:
#632 (comment)

It is not clear what do i do after i get the HD node root key. can someone please help explain..

By the way, I tried Zap and that doesn't work. It has been Importing wallet since about 20 hrs now.

[bjadel]

I proceeded exactly the same as @jodobear and also with me my bitcoin wallet and my channels were not restored. Is there a solution to start the recovery process again separately for the bitcoin wallet (seed) and the lightning channels (channel.backup)?

[openoms]

This document gives some options to try:
https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md#recovering-using-scbs

to copy the channel.backup to /home/admin can use SCP from the directory of the backup file:
scp channel.backup admin@RASPIBLITZ_IP:/home/admin/

then try to restore with:
lncli create -multi_file=channel.backup

As this issue is going since a while, it would worth to ask in the LND Developer Slack too:
https://join.slack.com/t/lightningcommunity/shared_invite/enQtMzQ0OTQyNjE5NjU1LWRiMGNmOTZiNzU0MTVmYzc1ZGFkZTUyNzUwOGJjMjYwNWRkNWQzZWE3MTkwZjdjZGE5ZGNiNGVkMzI2MDU4ZTE

[bjadel]

Thanks @openoms . I will try it. How can I restore the bitcoin wallet with my seed again?

[openoms]

@bjadel both the on-chain and off-chain chain balance is stored with LND on the RaspiBlitz.
Restoring the on-chain balance only from the command line is described here:
https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md#starting-on-chain-recovery

Basicly if you don't use the channel.backup you will only recover the on-chain funds.

Make sure you put the cypher seed (PASSWORD_D) in as well if you used one on the earlier RaspiBlitz version.

[bjadel]

@jodobear Were you successful? I could not restore it. Also the attempt by Seed and another Lighnting App (https://github.com/lightninglabs/lightning-app) failed.

[jodobear]

@jodobear Were you successful? I could not restore it. Also the attempt by Seed and another Lighnting App (https://github.com/lightninglabs/lightning-app) failed.

Nope. Also, haven't been able to test another method as mentioned by @openoms which is:

1. Create a random LND wallet on your node using lncli.
2. Then attempt recovery from seed of your old wallet again using lncli and see if that works.

This since you use lncli directly instead of relying on blitz's python scripts.

Will be able to report on this in a few days.

[jodobear]

@bjadel @openoms @rootzoll

So, my node recovered the on-chain funds! I don't know how, but here's what i did:

1. After being offline for about a week, i set it up but it wouldn't power up. I tried 3 diff chargers/adapters, no luck.
2. In the meantime i removed the SD card (while disassembling it trying to figure what's wrong) and tried again with a different adapter, it powered on - checked with all others and they all worked. Looked online and found similar reports where the SD card image somehow created this problem. I don't understand how my image was corrupted (maybe all the flights and moving).
3. So, i flashed a new image and went through the setup. This time it didn't ask me to recover from Seed/SCB - I just put passwrod A then password C and it caught up to the chain. When i woke up in the morning I had my on-chain funds (though not the ones returned from channel closure).

Here's what i believe can reproduce the results:

1. Follow the complete recovery from Seed/SCB and if it doesn't recover the on-chain funds then..
2. Reflash the SD card and setup the wallet again by inputting the passwords A & C.

OR as per @openoms suggested: create a random new wallet after step 1 then restore previous wallet with funds on the same node. I did not explicitly try this exact method but, i guess what i ended up doing is effectively this.

I still do not have my channels restored even though my pi accepted the channel.backup. Any ideas how i can go about that?

Hope this helps people recover their precious sats!
Cheers!

[jodobear]

@bjadel @openoms @rootzoll

So, my node recovered the on-chain funds! I don't know how, but here's what i did:

1. After being offline for about a week, i set it up but it wouldn't power up. I tried 3 diff chargers/adapters, no luck.

2. In the meantime i removed the SD card (while disassembling it trying to figure what's wrong) and tried again with a different adapter, it powered on - checked with all others and they all worked. Looked online and found similar reports where the SD card image somehow created this problem. I don't understand how my image was corrupted (maybe all the flights and moving).

3. So, i flashed a new image and went through the setup. This time it didn't ask me to recover from Seed/SCB - I just put passwrod A then password C and it caught up to the chain. When i woke up in the morning I had my on-chain funds (though not the ones returned from channel closure).

Here's what i believe can reproduce the results:

1. Follow the complete recovery from Seed/SCB and if it doesn't recover the on-chain funds then..

2. Reflash the SD card and setup the wallet again by inputting the passwords A & C.

OR as per @openoms suggested: create a random new wallet after step 1 then restore previous wallet with funds on the same node. I did not explicitly try this exact method but, i guess what i ended up doing is effectively this.

I still do not have my channels restored even though my pi accepted the channel.backup. Any ideas how i can go about that?

Hope this helps people recover their precious sats!
Cheers!

It appears that the funds i recovered above were my off-chain funds, because while following the recovery.md to recover off-chain funds using following command:

lncli restorechanbackup --multi_file ~/files/channel.backup

I got the following output:

[lncli] unable to restore chan backups: rpc error: code = Unknown desc = unable to unpack chan backup: channel already exists

while my node does not have any channels while 1ml shows 2 channels are open, (edit:start) but last week 1ml showed 6 channels open and my node's Last update shows 4 months(edit:end) and lncli getinfo currently shows:

...
"num_pending_channels": 0,
"num_active_channels": 0,
"num_inactive_channels": 0,
...

While digging further using lncli closedchannels, I find 9 out of 13 channels i had open when my pi crashed as REMOTE_FORCE_CLOSE. So, i checked the closing_tx_hash of the channels with settled_balance > 0 and find all of them with 138 confirmations and the sum of the settled_balance off by 252 sats (i have calculated the miner fee after moving them out of the pi).

This seems like i recovered my off-chain funds. Can anyone please confirm? If so, the SCB worked and sent me the funds after ~10 days? If true, i still need to recover my on-chain funds which i believe i can as per @openoms suggested: create a random new wallet after step 1 then restore previous wallet with funds on the same node. This i tried, and got:

[lncli] unable to generate seed: rpc error: code = Unimplemented desc = unknown service lnrpc.WalletUnlocker

[openoms]

@jodobear
Wait for the channel funds to return onchain in this wallet now. When restoring with and SCB only the online peers can initiate the force-close, so the remaining channel funds are stuck until the peer comes online. In LND the default time-lock (CSV) is 144 blocks and longer in c-lightning.

The fact that the SCB worked implies that this is the correct on-chain wallet too. Did you not cash out the onchain funds before?

Creating a random a new wallet was only suggested when you want to try restoring the same lnd wallet testing with or without the SCB.
lncli create would not let you to restore the same seed as what is already active, so you can get around this by creating a temporary wallet in the meantime. There is no point for you doing this now as the restore process worked as it is proven by the channels force closing.

[jodobear]

The fact that the SCB worked implies that this is the correct on-chain wallet too. Did you not cash out the onchain funds before?

Nope. Only now, which i thought were my on-chain funds but apparently it seems they are my off-chain funds (edit:start) that returned after REMOTE_FORCE_CLOSE (edit:end). And if that's the case then i still need to recover my on-chain funds. And to do that i tried to create a temporary wallet but, that gave the the following error:

[lncli] unable to generate seed: rpc error: code = Unimplemented desc = unknown service lnrpc.WalletUnlocker

[openoms]

@jodobear
You can try the LND Desktop app: https://github.com/lightninglabs/lightning-app/releases too to attempt recovering the on-chain funds. Just use the Seed and passphrase (PASSWORD_D) to recover

[jodobear]

UPDATE on the onchain funds that weren't found on blitz recovery from error code -28:

I have managed to recover the onchain funds by setting up a fresh blitz on rpi4 with v1.5.1 raspiblitz with just the SEED backup. Didn't do anything special, just waited for the blitz to sync and restored my old LND wallet.

If anyone still has the LND wallet seed, try setting up a fresh blitz.