Lnd: walletbalance shows wrong confirmed balance, channels stuck as pending

Created on 2 Jul 2018 · 45Comments · Source: lightningnetwork/lnd

Background

lncli walletbalance is showing confirmed balance which I don't have. lnd lets me open a channel with those funds, but obviously failing to send out the opening transactions because those outputs are already spent. Channels remain pending for over 1 month now, lncli walletbalance is still showing a wrong balance.

Your environment

version of lnd
0.4.2
which operating system (uname -a on *Nix)
4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
version of btcd, bitcoind, or other backend
bitcoind (0.16.1)
any other relevant environment details

Steps to reproduce

In my case my walletbalance shows
{ "total_balance": "139197", "confirmed_balance": "139197", "unconfirmed_balance": "0" }
Although I only really have about 12000 sat. lnd allows me to open a channel with those non existing funds, those channels show up as pending

{
    "total_limbo_balance": "0",
    "pending_open_channels": [
        {
            "channel": {
                "remote_node_pub": "0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961",
                "channel_point": "69a75bfae6642276efc1bbbf3d8860e3710cc6894c0cfaa438b4987418837427:1",
                "capacity": "110000",
                "local_balance": "109819",
                "remote_balance": "0"
            },
            "confirmation_height": 0,
            "commit_fee": "181",
            "commit_weight": "600",
            "fee_per_kw": "250"
        },
        {
            "channel": {
                "remote_node_pub": "0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961",
                "channel_point": "835672640218607009922e594baba1d3fdad67fb181aab44e84852051d5361bf:0",
                "capacity": "116252",
                "local_balance": "115528",
                "remote_balance": "0"
            },
            "confirmation_height": 0,
            "commit_fee": "724",
            "commit_weight": "600",
            "fee_per_kw": "1000"
        }
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
    ],
    "waiting_close_channels": [
    ]
}

but the opening transactions obviously never goes out

log says:

2018-05-31 18:51:13.139 [ERR] FNDG: Unable to complete reservation sign complete: Transaction rejected: output already spent
2018-05-31 18:51:13.139 [ERR] FNDG: Failing funding flow: Transaction rejected: output already spent ((*lnwire.Error)(0xc42e7021c0)({
2018-05-31 18:51:13.139 [ERR] RPCS: unable to open channel to NodeKey(0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961): Transaction rejected: output already spent

2018-06-03 23:22:22.708 [ERR] FNDG: Unable to complete reservation sign complete: Transaction rejected: output already spent
2018-06-03 23:22:22.708 [ERR] FNDG: Failing funding flow: Transaction rejected: output already spent ((*lnwire.Error)(0xc4220b07c0)({
2018-06-03 23:22:22.708 [ERR] RPCS: unable to open channel to NodeKey(0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961): Transaction rejected: output already spent

The weird thing about it is that the balance disappears as confirmed balance from lncli walletbalance, after trying to open a channel with those funds, but after some time (I guess hours) it reappears as confirmed balance while the channel is still pending.

Expected behaviour

lncli walletbalance not showing spent outputs as confirmed

Actual behaviour

lncli walletbalance showing funds which were already spent as confirmed, reappearing after trying to open a channel with those "non existent" funds while the channel stays pending.

When trying to send an onchain transaction

lncli sendcoins --addr 34bFuLRhXgD8En6k3pgokDaqHck6yzpUmL --amt 130000 --sat_per_byte 1
[lncli] rpc error: code = Unknown desc = insufficient funds available to construct transaction

P1 funding wallet

Source

TrabbipG

All 45 comments

Do you have an idea of how these outputs were spent in the first place? If they were part of a funding transaction that never confirmed within our timeout (288 blocks), then we'll mark the channel as funding canceled, but we currently don't handle being able to reuse those outputs again. I'm currently working on a way to fix this. If you still have your seed around, you can start up a new node and restore. You should make sure you close all your channels beforehand though as the seed won't recover the funds in your channels.

wpaulino on 3 Jul 2018

Can you try to update to the current master? Since 0.4.2 a few bugs have been fixed to ensure that we detect "output already spent"-like errors from the chain backend.

Are you using the bitcoind backend?

Roasbeef on 3 Jul 2018

@wpaulino I must have used them for funding other channels. It's a bit hard for me to keep track of my overall balance because the channelbalance seems to shift up or down, depending on the mempool I assume. I would say that there was no transaction which didn't confirm in time, but I can't be 100% sure. Is there a way to find that out?

I do have my seed, so starting from scratch is an option if there's no other way currently.

@Roasbeef I'm using bitcoind. I also just updated my node, but it didn't help. Walletbalance now shows

 lncli walletbalance
{
    "total_balance": "216074",
    "confirmed_balance": "216074",
    "unconfirmed_balance": "0"
}

without deposit or channel closing. Pending channels are also still here.

Also it says this at bootup, maybe it helps. It's been over a month now and it still waits for confirmations?

2018-07-03 16:59:09.436 [INF] FNDG: Waiting for funding tx (69a75bfae6642276efc1bbbf3d8860e3710cc6894c0cfaa438b4987418837427) to reach 3 confirmations
2018-07-03 16:59:09.436 [INF] NTFN: New confirmation subscription: txid=69a75bfae6642276efc1bbbf3d8860e3710cc6894c0cfaa438b4987418837427, numconfs=3
2018-07-03 16:59:42.898 [INF] RPCS: [listchannels] fetched 14 channels from DB
2018-07-03 17:03:15.814 [INF] NTFN: New confirmation subscription: txid=835672640218607009922e594baba1d3fdad67fb181aab44e84852051d5361bf, numconfs=3
2018-07-03 17:03:15.814 [INF] FNDG: Waiting for funding tx (835672640218607009922e594baba1d3fdad67fb181aab44e84852051d5361bf) to reach 3 confirmations

TrabbipG on 3 Jul 2018

So it's the case that those two funding transactions were just never broadcast. As a result, they should be removed from the set of pending channels. Your balance is now correct, but there needs to be logic to double spend those inputs (or just delete those pending channels), @wpaulino is working on a patch to resolve this.

Roasbeef on 6 Jul 2018

@Roasbeef Yes, those funding transactions never got broadcasted because those outputs were already spent, but the walletbalance before trying to open those two channels was already wrong.

Weirdly the balance changed now after trying to send an onchain transaction.

lncli walletbalance
{
    "total_balance": "216074",
    "confirmed_balance": "216074",
    "unconfirmed_balance": "0"
}

lncli sendcoins --addr 37DKFw4pXXEoBgvTbv5r9119UzdNVqUvxa --amt 216074 --sat_per_byte 5
[lncli] rpc error: code = Unknown desc = insufficient funds available to construct transaction
lncli sendcoins --addr 37DKFw4pXXEoBgvTbv5r9119UzdNVqUvxa --amt 200000 --sat_per_byte 5
[lncli] rpc error: code = Unknown desc = -25: Missing inputs
lncli sendcoins --addr 37DKFw4pXXEoBgvTbv5r9119UzdNVqUvxa --amt 150000 --sat_per_byte 5
[lncli] rpc error: code = Unknown desc = insufficient funds available to construct transaction



md5-c3bfd7b74d6103af384a60b67a1a4d4c



lncli walletbalance
{
    "total_balance": "13884",
    "confirmed_balance": "2699",
    "unconfirmed_balance": "11185"
}

total_balance could actually be the correct one now, I'm just confused why it shows an unconfirmed_balance , because obviously no transaction was broadcasted, listchaintxns also doesn't show anything. Assuming the unconfirmed_balance will be confirmed, it seems like updating to the current master branch and trying to send an onchain transaction might have fixed this.

Is there a way for me to delete those pending channels or do I have to wait for the patch you mentioned?

TrabbipG on 6 Jul 2018

I'm seeing the same issue - the possibly related factor I noticed is that by tracking down a transaction referenced as being rebroadcast on startup it has a fee that is 1 satoshi off from a 1 sat/vbyte fee and therefore doesn't get confirmed

alexbosworth on 11 Jul 2018

I tracked down the unconfirmed balance in my case to a spend of a received HTLC output or HTLC success that uses a fee 1 sat below the min relay fee

alexbosworth on 12 Jul 2018

Debugging by looking at the transaction it forms and broadcasts I can see that one of the inputs it's selecting has already been spent into a channel. So I guess the issue is that it thinks one of its coins is not spent but it has been spent.

alexbosworth on 12 Jul 2018

I have the same issue on commit: a43d02f8f049a82304a3c994bb1e326a3a8e1572

$ ncli pendingchannels
{
"total_limbo_balance": "164076",
"pending_open_channels": [
],
"pending_closing_channels": [
],
"pending_force_closing_channels": [
{
"channel": {
"remote_node_pub": "03b439ff61f39f557fdea45129230f4f0c22b6625fcbbdba0dee52cfc56eed5e4a",
"channel_point": "a710f5a009d9721adf876b92a54b602ed066dd342c13dd3fba4be14f14870bee:1",
"capacity": "166325",
"local_balance": "164076",
"remote_balance": "0"
},
"closing_txid": "b8c76bce5f40a371921c01e073529b9b408cde9b71655b28ddefeb7e4c1896e1",
"limbo_balance": "164076",
"maturity_height": 0,
"blocks_til_maturity": 0,
"recovered_balance": "0",
"pending_htlcs": [
]
}
],
"waiting_close_channels": [
]
}

This channel was force closed by the remote host (not sure why). Checking a block explorer shows that the closing_txid was never confirmed, and in fact a conflicting transaction was confirmed instead:

edit: I also updated to the lastest commit (b0288d46773ac6d45e5dc4d5e6a80dd3034d0b9f) but the channel didn't change.

Also I am using bitcoind as the backend.

jpentland on 16 Jul 2018

@jpentland, your issue sounds a lot like #1341. #1554 attempts to resolve that. Note that you won't be able to clear the channel from your set of pending force closed channels until the forgetchannel PR is merged.

wpaulino on 16 Jul 2018

I'm having this same issue using btcd as the backend.

2018-07-20 14:47:38.784 [INF] LNWL: Inserting unconfirmed transaction 4199c61b057b404f28d19a0366dbfbb52eb5c6766459bff1d2a5caefc489bd42
2018-07-20 14:47:38.942 [INF] DISC: Broadcasting batch of 8 new announcements
2018-07-20 14:47:39.131 [ERR] FNDG: Unable to complete reservation sign complete: Transaction rejected: output already spent
2018-07-20 14:47:39.131 [INF] FNDG: Cancelling funding reservation for node_key=03cb7983dc247f9f81a0fa2dfa3ce1c255365f7279c8dd143e086ca333df10e278, chan_id=47f54fda08198f60a786ad45b1a5692ddef5f214305c8484aa3bd068f8e4966a

When opening a channel.

juscamarena on 20 Jul 2018

@juscamarena on startup, if you start with LNWL=trace, what does lnd try to rebroadcast?

Roasbeef on 20 Jul 2018

I have a similar issue I think, with lnd version 0.4.2-beta commit=10b0df61a71d6931c3e5da577f094f24d8d87a43, using bitcoind v0.16.1.

I had autopilot enabled. At 19:23 it broadcast a funding transaction spending utxo abdbb3c2:1

2018-07-21 19:22:59.665 [INF] FNDG: Generated ChannelPoint(6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701:0) for pendingID(5642bc2b575301889c0ae2865325b109706e449076a2e58260ea35ed426f7538)
2018-07-21 19:22:59.841 [INF] LNWL: Broadcasting funding tx for ChannelPoint(6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701:0): (*wire.MsgTx)(0xc422400180)({
 Version: (int32) 1,
 TxIn: ([]*wire.TxIn) (len=1 cap=15) {
  (*wire.TxIn)(0xc4225b1b00)({
   PreviousOutPoint: (wire.OutPoint) abdbb3c23ba74f883517d0b3c399a2eae7c0329454803e00dcbcd0c955a1f791:1,

Then 17 minutes later at 19:40 it tried to spend the same utxo again to fund a different channel:

2018-07-21 19:40:34.003 [INF] FNDG: Generated ChannelPoint(2facfe9147121b64761bef31c3808f43f92311349dbba01217bb666f0bd9b83c:1) for pendingID(6f1b8d4b7b40120c109ad779e31ca4715d5b1394f64125029cebbe96a6b35456)
2018-07-21 19:40:34.089 [INF] LNWL: Broadcasting funding tx for ChannelPoint(3759970b1885cc79033e4d4f0380272a28a432a175b4a39b43a5d073cff69769:0): (*wire.MsgTx)(0xc423ca4500)({
 Version: (int32) 1,
 TxIn: ([]*wire.TxIn) (len=1 cap=15) {
  (*wire.TxIn)(0xc4283a2a20)({
   PreviousOutPoint: (wire.OutPoint) abdbb3c23ba74f883517d0b3c399a2eae7c0329454803e00dcbcd0c955a1f791:1,

The attempted double-spend fails, and so does the attempt to cancel the reservation:

2018-07-21 19:40:34.089 [INF] LNWL: Inserting unconfirmed transaction 3759970b1885cc79033e4d4f0380272a28a432a175b4a39b43a5d073cff69769
2018-07-21 19:40:34.219 [ERR] FNDG: Unable to complete reservation sign complete: Transaction rejected: output already spent
2018-07-21 19:40:34.219 [INF] FNDG: Cancelling funding reservation for node_key=033dd79123e7c386655f72d3eed186c071ef3b4a625d3fb9f613635e6483ff6823, chan_id=cdab39af680406a4c9614c73b4ae328cfc68e9d29618499cb44264f9ab230c4e
2018-07-21 19:40:34.219 [ERR] FNDG: unable to cancel reservation: unable to cancel reservation: attempted to cancel non-existent funding state

The first transaction confirmed at 19:50, so was unconfirmed when lnd attempted to double-spend it.

bob-333 on 22 Jul 2018

I also noticed the following log entries:

2018-07-21 19:24:26.602 [INF] LTND: Received shutdown request.
2018-07-21 19:24:26.602 [INF] LTND: Shutting down...
2018-07-21 19:24:26.602 [INF] LTND: Gracefully shutting down.
2018-07-21 19:24:26.602 [INF] ATPL: Autopilot Agent stopping
[...]
2018-07-21 19:33:50.898 [INF] CRTR: FilteredChainView stopping
2018-07-21 19:33:50.916 [INF] FNDG: Funding manager shutting down

2018-07-21 19:33:57.391 [INF] LTND: Version 0.4.2-beta commit=10b0df61a71d6931c3e5da577f094f24d8d87a43
2018-07-21 19:33:57.391 [INF] LTND: Active chain: Bitcoin (network=mainnet)

So I shut down and restarted lnd between the two transactions. It took 9 minutes to shut down.

bob-333 on 22 Jul 2018

If he has pending channels it also triggers a rescan from when the channel was opened causing me long startup times, I'll try in a bit, have to apply a patch by Wilmer Paulino first.

juscamarena on 23 Jul 2018

@bob-333 in that first broadcast, was the transaction included in the mempool of the node? As in, was there an error?

Roasbeef on 23 Jul 2018

If you grep y'all's logs for: reservation timed out waiting for peer, does anything come up?

Roasbeef on 23 Jul 2018

it also triggers a rescan from when the channel was opened causing me long startup times

Rescans in master for historical spend dispatches are now async.

Roasbeef on 23 Jul 2018

🎉1

@Roasbeef http://paste.ubuntu.com/p/YwbkfxjsFt/

juscamarena on 24 Jul 2018

Here's some more logs, set Trace just for LNWL but didn't show, setting system wide did unless I did the per subsystem setting wrong: https://paste.ubuntu.com/p/q7YpvndqjR/

juscamarena on 24 Jul 2018

@Roasbeef It looks like that first transaction was pretty normal.

Here are the logs:

 LockTime: (uint32) 0
})

2018-07-21 19:22:59.842 [INF] LNWL: Inserting unconfirmed transaction 6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701                                 
2018-07-21 19:22:59.953 [INF] CNCT: Creating new ChannelArbitrator for ChannelPoint(6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701:0)                
2018-07-21 19:22:59.953 [INF] CNCT: ChannelArbitrator(6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701:0): starting state=StateDefault                 
2018-07-21 19:22:59.953 [INF] NTFN: New block epoch subscription                                                                                                       
2018-07-21 19:22:59.955 [INF] NTFN: New spend subscription: utxo=6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701:0                                    
2018-07-21 19:23:00.054 [INF] FNDG: Finalizing pendingID(5642bc2b575301889c0ae2865325b109706e449076a2e58260ea35ed426f7538) over ChannelPoint(6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701:0), waiting for channel open on-chain                                                                                          
2018-07-21 19:23:00.054 [INF] CNCT: Close observer for ChannelPoint(6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701:0) active                         
2018-07-21 19:23:00.054 [INF] FNDG: Waiting for funding tx (6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701) to reach 6 confirmations                 
2018-07-21 19:23:00.054 [INF] NTFN: New confirmation subscription: txid=6bcca1c3bf04c2dcca722785823eb120ee97a90c7938d4ce21f3663a00fdd701, numconfs=6
2018-07-21 19:23:00.979 [INF] HSWC: Sent 0 satoshis and received 0 satoshis in the last 10 seconds (0.100000 tx/sec)

bob-333 on 24 Jul 2018

I see a ton of:

lnd.log:2018-07-20 15:38:18.345 [WRN] FNDG: reservation timed out waiting for peer (peerID:&{0x174c520 92034206821044701267266611382922809398774087712877799381646209490804561470072 28108798742920263598834841103944403701817574075537330892433740656000889207719}, chanID:47f54fda08198f60a786ad45b1a5692ddef5f214305c8484aa3bd068f8e4966a)

In that trace (over 500 instances), so that points towards an issue in the zombie reservation sweeper. If it unlocks outputs that are actually still in use, then this may lead to a double spend. However, since the transaction was inserted into the txstore, then these inputs should still be marked as unspent...

Roasbeef on 25 Jul 2018

https://github.com/lightningnetwork/lnd/pull/1635 should get rid of the annoying [ERR] FNDG: unable to cancel reservation: unable to cancel reservation: attempted to cancel non-existent funding state message. TLDR; we kept the channel among pending reservations even after adding it to the DB.

halseth on 26 Jul 2018

Can try that now here.

juscamarena on 26 Jul 2018

So looks like we go on to publish the transaction, but delete it from the TxStore if we encounter an error: https://github.com/btcsuite/btcwallet/blob/64b5b448f5e6853a2d870f388504a7807bc951f1/wallet/wallet.go#L3251

Since a channel pending at this point is not currently deleted from the database (and I don't think we safely can, see https://github.com/lightningnetwork/lnd/issues/1623#issuecomment-408077477), it will retry broadcasting the funding tx at startup. But in the meantime I see no reason that the outputs it uses cannot be double spent.

I think this could get us into this situation.

halseth on 26 Jul 2018

Does this explicitly lead to the wallet being out of state and having already spent utxos?

Here's a view of my listunspent by manually querying and logging ListUnspentWitness utxos.

https://pastebin.com/GyRgVPb2 (Privacy leak but oh well)

Many of them are already spent near the end, and they contribute to walletbalance as well as causing me issues whenever I want to sweep funds off the wallet or open new channels.

Examples:
be5a59086987aaa124ef5021074f7b0cde9f64be3b5cd79a92cc55b2c0d2bf4e 1
25c84da62dc66167c717a3039df3744d8a817f148b47a8a7e4373d1ee17198b5 1
aa0ea668497db575f05e2839e5672e412ca20d15060a06704c87f8123debfc15 0

I've also previously rescanned the wallet and am back to having the issue.

juscamarena on 22 Aug 2018

I got same issue after re-enabling autopilot with 21841c9f6b065a625e5dcb68c18dfbf3ee25326b:

[ERR] FNDG: unable to broadcast funding txn: Transaction rejected: output already spent

I have three pending channels, all of which has no tx in mempool/chain.

Talkless on 22 Aug 2018

If I would update lnd now, would these pending channels that used spent utxo's magically disappear, or something specific would need to be done?

Talkless on 4 Sep 2018

👍1

@Talkless they won't disappear now, but they'll no longer occur. The fix to ensure that the wallet utxo state is clean is to run the dropwtxmgr tool which will force a full normal rescan.

Roasbeef on 5 Sep 2018

@Roasbeef and this dropwtxmgr tool is available in master, or will be available later in 5.x?

Talkless on 5 Sep 2018

@Talkless you can find it in https://github.com/btcsuite/btcwallet

halseth on 7 Sep 2018

@halseth Oh, so it's not gonna be a lncli command?

Talkless on 7 Sep 2018

@Roasbeef I updated to 0.5.0 today, but that issue still occurs.

lncli openchannel --node_key 0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961 --local_amt 400000 { "funding_txid": "71990451cdd735ee4d9065eaf9dff174b24e0eee8d8b72db0f7b5f3bc30859e6" }
log still says

2018-09-15 18:15:43.745 [ERR] FNDG: unable to broadcast funding txn: Transaction rejected: output already spent

but channel is pending again

"channel": { "remote_node_pub": "0374ecf61ed6c1208c42339f47decde2bc0c4393ac95f07827b3471e939d7eb961", "channel_point": "71990451cdd735ee4d9065eaf9dff174b24e0eee8d8b72db0f7b5f3bc30859e6:1", "capacity": "400000", "local_balance": "398179", "remote_balance": "0" }, "confirmation_height": 0, "commit_fee": "1821", "commit_weight": "600", "fee_per_kw": "2516" }

I'm also still confused about my actual wallet/channelbalance. I have a few pending channels with that problem and 400000 sat as walletbalance although those 400000 sat disappear as soon as I try to send an onchain transaction -> insufficient funds available to construct transaction, afterwards it goes down to about 2000 sat, but as soon as I restart lnd the walletbalance goes to 400000 again. Does dropwtxmgr clean that up?

TrabbipG on 15 Sep 2018

Could someone give a hint how to actually use that tool? This is what I get (based on https://github.com/btcsuite/btcwallet/blob/74a7124666b49cd7e67da87c61169f4fb83fbd73/docs/force_rescans.md):

root@odroid-hc1:/home/lnd/lnd.go/src/github.com/btcsuite/btcwallet/cmd/dropwtxmgr# export GOROOT=/usr/lib/go-1.10
root@odroid-hc1:/home/lnd/lnd.go/src/github.com/btcsuite/btcwallet/cmd/dropwtxmgr# export GOPATH=/home/lnd/lnd.go
root@odroid-hc1:/home/lnd/lnd.go/src/github.com/btcsuite/btcwallet/cmd/dropwtxmgr# export PATH=${GOROOT}/bin:${GOPATH}/bin:$PATH
root@odroid-hc1:/home/lnd/lnd.go/src/github.com/btcsuite/btcwallet/cmd/dropwtxmgr# go get
# github.com/btcsuite/btcwallet/walletdb/bdb
../../walletdb/bdb/db.go:12:2: imported and not used: "github.com/coreos/bbolt"
../../walletdb/bdb/db.go:55:10: undefined: bolt
root@odroid-hc1:/home/lnd/lnd.go/src/github.com/btcsuite/btcwallet/cmd/dropwtxmgr# echo $?
2

Talkless on 16 Sep 2018

@Talkless GOPATH must be set to _directory_ that contains all third-party go code. For instance, mine is set to ~/go. The tool can then be installed using

go get -u github.com/btcsuite/btcwallet/cmd/dropwtxmgr

Finally, run this on the wallet database

dropwtxmgr --db=/path/to/data/chain/bitcoin/mainnet/wallet.db

cfromknecht on 17 Sep 2018

@cfromknecht thanks, I managed to insall and run dropwtxmgr, I've checked README.md of the btcwallet.

Currently my lnd is rescanning blocks, will see if these redundant pending channels will disappear (whey are still visible so far).

Talkless on 17 Sep 2018

So I did run dropwtxmgr and after resyncinc my walletbalance is 0 now, so no more spent outputs in there apparently. I still have some funds missing now. I assume they are somehow stuck in my pending channels. Is it possible that when I tried to open a channel my real balance got mixed together with unspent outputs and now they are stuck as a pending channel without having broadcasted a valid tx?

update
So actually lncli walletbalance did change over time again. When I initially wrote this comment it showed "total_balance": "0", but after some time it changed to about "total_balance": "75500". I thought it showed some spent output again, but this time it actually was real balance I was able to send on chain to my bitcoin node. After I send that transaction lncli walletbalance again showed "total_balance": "0", and then after a few minutes the balance changed to "total_balance": "32000" and afterwards to "total_balance": "190000", all spendable. Afterwards I had "total_balance": "355" left. Out of curiosity I ran dropwtxmgr again and it shows "total_balance": "0", so the 355 sat are missing again.

I was wondering if it would make sense if we had a possibility to check your utxos like bitcoin-cli listunspent

TrabbipG on 17 Sep 2018

I still see 6 pending channels with non-existing channel_point's after dropwtxmgr and rescanning (took 3 days, ugh...). I believe it was 5 before, I guess 6'th was created after I've launched v0.5, maybe eve during rescanning..?

Talkless on 20 Sep 2018

I've managed to get rid of these forever-pending channels, with Veggen @ #lnd help! You just have to force-close that channel! All 6 pendings are gone now.

Talkless on 11 Nov 2018

🎉1

Hello!

I have same problem with fresh version of LND ("version": "0.5.2-beta commit=v0.5.2-beta"):
Before i had the pending channel (in "pending_open_channels") with unexisted channel_point transaction
I did:

l closechannel --force 357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8 0

After the channel moved to:

"waiting_close_channels": [
    {
        "channel": {
            "remote_node_pub": "027388b2266aaf4d98dd54026e1f3d01dab3c5734debf57188442e84fe9bc9f7b5",
            "channel_point": "357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8:0",
            "capacity": "2000000",
            "local_balance": "1997276",
            "remote_balance": "0"
        },
        "limbo_balance": "1997276"
    }
]

Now i have in logs of LND:

2019-02-12 18:48:24.548 [INF] HSWC: Removing channel link with ChannelID(d84d301fc7d6a17f4f958fc45ff15cc113bc98473866a5e460c7d75c94557335)
2019-02-12 18:48:24.548 [INF] CNCT: Attempting to force close ChannelPoint(357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8:0)
2019-02-12 18:48:24.569 [INF] CNCT: ChannelArbitrator(357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8:0): force closing chan
2019-02-12 18:48:25.189 [INF] HSWC: Removing channel link with ChannelID(d84d301fc7d6a17f4f958fc45ff15cc113bc98473866a5e460c7d75c94557335)
2019-02-12 18:48:25.192 [INF] CNCT: Broadcasting force close transaction, ChannelPoint(357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8:0): (*wire.MsgTx)(0xc0254e3900)({
 Version: (int32) 2,
 TxIn: ([]*wire.TxIn) (len=1 cap=1) {
  (*wire.TxIn)(0xc02141e240)({
   PreviousOutPoint: (wire.OutPoint) 357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8:0,
   SignatureScript: ([]uint8) {
   },
   Witness: (wire.TxWitness) (len=4 cap=4) {
    ([]uint8) <nil>,
    ([]uint8) (len=71 cap=144) {
     00000000  30 44 02 20 66 de 64 44  62 09 55 51 54 68 0f 22  |0D. f.dDb.UQTh."|
     00000010  de e5 93 48 06 fd ea d9  fe f2 45 86 62 86 5e ed  |...H......E.b.^.|
     00000020  11 5f 3c c6 02 20 07 64  5f 7c 93 c1 a3 fd 9f 4e  |._<.. .d_|.....N|
     00000030  59 41 bd d2 84 e8 62 51  0e da b8 00 a7 1b d8 48  |YA....bQ.......H|
     00000040  3f de 81 c3 06 56 01                              |?....V.|
    },
    ([]uint8) (len=71 cap=144) {
     00000000  30 44 02 20 7d 07 37 7c  0c b7 71 13 dc 07 30 a1  |0D. }.7|..q...0.|
     00000010  1a 92 93 d4 c8 0d 38 81  1b 20 dc 1c 7c 30 d6 ce  |......8.. ..|0..|
     00000020  6a 2e 5d 32 02 20 64 73  d6 1e 6e fe 08 72 b5 94  |j.]2. ds..n..r..|
     00000030  14 03 51 3f 5d af 22 ef  a5 47 23 53 49 c0 5c f8  |..Q?]."..G#SI.\.|
     00000040  78 73 20 0b c5 60 01                              |xs ..`.|
    },
    ([]uint8) (len=71 cap=500) {
     00000000  52 21 02 df c2 2a 2b b0  fb 64 c0 ca c9 1d 70 e5  |R!...*+..d....p.|
     00000010  5b 4c 9c 15 9a 19 23 41  09 f9 9c 46 3b 3e 47 4e  |[L....#A...F;>GN|
     00000020  f9 9a 64 21 03 a2 b2 2f  52 c8 c2 c9 f5 cf d8 14  |..d!.../R.......|
     00000030  6f 06 e1 07 d5 1a 67 2c  df 77 81 5e f3 7e bd dd  |o.....g,.w.^.~..|
     00000040  58 fc e6 10 db 52 ae                              |X....R.|
    }
   },
   Sequence: (uint32) 2162509994
  })
 },
 TxOut: ([]*wire.TxOut) (len=1 cap=1) {
  (*wire.TxOut)(0xc00c3854a0)({
   Value: (int64) 1997276,
   PkScript: ([]uint8) (len=34 cap=34) {
    00000000  00 20 08 86 ca 09 f8 48  6c 3f 0d f9 6c e1 ea f1  |. .....Hl?..l...|
    00000010  45 60 2d 50 cd 60 c7 91  dd 43 1b e2 11 b4 21 bb  |E`-P.`...C....!.|
    00000020  a2 82                                             |..|
   }
  })
 },
 LockTime: (uint32) 547523292
})

2019-02-12 18:48:25.192 [INF] LNWL: Inserting unconfirmed transaction 1a374274a01dbdc96fffa2c97e02c5430d7f27a3beed418107351e9d91d02480
2019-02-12 18:48:25.194 [ERR] CNCT: ChannelArbitrator(357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8:0): unable to broadcast close tx: Transaction rejected: output already spent
2019-02-12 18:48:25.207 [INF] CNCT: ChannelArbitrator(357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8:0): noop trigger userTrigger
2019-02-12 18:48:25.207 [ERR] CNCT: Unable to disable channel 357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8:0 on close: edge not found
2019-02-12 18:48:25.207 [INF] PEER: Waiting for confirmation of cooperative close of ChannelPoint(357355945cd7c760e4a566384798bc13c15cf15fc48f954f7fa1d6c71f304dd8:0) with txid: 1a374274a01dbdc96fffa2c97e02c5430d7f27a3beed418107351e9d91d02480
2019-02-12 18:48:25.207 [INF] NTFN: New confirmation subscription: txid=1a374274a01dbdc96fffa2c97e02c5430d7f27a3beed418107351e9d91d02480, numconfs=1

Before i had non-existing channel_point and after force closing i have same too. @wpaulino @Roasbeef How can i prune this dead channel?

Thanks for helping!

LNBIG-COM on 12 Feb 2019

And right now there (in pending_open_channels) i have second stuck channel:

    {
        "channel": {
            "remote_node_pub": "029cf833ceef83b31b2e202d46c99c2dc9b5fa56259d8b84d128ee37e65e4e96d3",
            "channel_point": "8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1",
            "capacity": "9000000",
            "local_balance": "8998240",
            "remote_balance": "0"
        },
        "confirmation_height": 0,
        "commit_fee": "1760",
        "commit_weight": "600",
        "fee_per_kw": "2432"
    },

I found info about channel in logs. It's old channel and here how it was created:
Please pay attention there at line Unable to broadcast funding tx for ChannelPoint(8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1): Transaction rejected: output already spent
In that time it could not be created because output already spent. But why this channel exists already 3 weeks?

2019-01-24 13:24:17.918 [INF] FNDG: Generated ChannelPoint(8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1) for pendingID(aa38ac947faf6a608443717554df0ec1dadfc4c2449276e7dc811db6e6633eae)
2019-01-24 13:24:18.624 [INF] DISC: gossipSyncer(029cf833ceef83b31b2e202d46c99c2dc9b5fa56259d8b84d128ee37e65e4e96d3): applying new update horizon: start=1970-01-01 01:00:00 +0100 CET, end=2106-02-07 07:28:15 +0100 CET, backlog_size=67663
2019-01-24 13:24:18.624 [INF] DISC: gossipSyncer(029cf833ceef83b31b2e202d46c99c2dc9b5fa56259d8b84d128ee37e65e4e96d3): buffering chan range reply of size=402
2019-01-24 13:24:18.624 [INF] DISC: gossipSyncer(029cf833ceef83b31b2e202d46c99c2dc9b5fa56259d8b84d128ee37e65e4e96d3): filtering through 402 chans
2019-01-24 13:24:18.625 [INF] DISC: gossipSyncer(029cf833ceef83b31b2e202d46c99c2dc9b5fa56259d8b84d128ee37e65e4e96d3): remote peer has no new chans
2019-01-24 13:24:18.625 [INF] DISC: gossipSyncer(029cf833ceef83b31b2e202d46c99c2dc9b5fa56259d8b84d128ee37e65e4e96d3): applying gossipFilter(start=2019-01-24 12:24:18.62516464 +0100 CET m=+771432.529599609)
2019-01-24 13:24:18.730 [INF] FNDG: Broadcasting funding tx for ChannelPoint(8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1): (*wire.MsgTx)(0xc042a57ec0)({
 Version: (int32) 1,
 TxIn: ([]*wire.TxIn) (len=1 cap=15) {
  (*wire.TxIn)(0xc01f7c7e60)({
   PreviousOutPoint: (wire.OutPoint) de606ec2e984ed8bb1c6ed9c43898bce90a456126ae8a32f260bcd8fb93ae3fe:1,
   SignatureScript: ([]uint8) <nil>,
   Witness: (wire.TxWitness) (len=2 cap=2) {
    ([]uint8) (len=71 cap=144) {
     00000000  30 44 02 20 53 dc 6b 31  ea 06 27 d3 cb cd 19 3a  |0D. S.k1..'....:|
     00000010  41 37 fc bd 4f b7 60 6d  a7 71 6a 9b 50 da ce 68  |A7..O.`m.qj.P..h|
     00000020  cf 6b 5c d6 02 20 58 86  b1 47 75 b5 f6 02 d6 06  |.k\.. X..Gu.....|
     00000030  0a 87 18 69 4d a0 3e 4a  52 fa ad 82 00 e5 66 7a  |...iM.>JR.....fz|
     00000040  7f 04 39 be 5d 7b 01                              |..9.]{.|
    },
    ([]uint8) (len=33 cap=33) {
     00000000  03 21 e6 fb 1f e0 2b d7  22 c7 1e 16 ee 02 50 93  |.!....+.".....P.|
     00000010  64 4c f8 a3 43 cf ea 2d  0e bf 41 50 23 b4 d4 48  |dL..C..-..AP#..H|
     00000020  1e                                                |.|
    }
   },
   Sequence: (uint32) 4294967295
  })
 },
 TxOut: ([]*wire.TxOut) (len=2 cap=15) {
  (*wire.TxOut)(0xc009af3100)({
   Value: (int64) 7277579,
   PkScript: ([]uint8) (len=22 cap=500) {
    00000000  00 14 d5 39 0f 9a 6c ac  1b ff e9 c9 9e 7e 57 46  |...9..l......~WF|
    00000010  dd 5c 33 5a 11 d8                                 |.\3Z..|
   }
  }),
  (*wire.TxOut)(0xc011cc01c0)({
   Value: (int64) 9000000,
   PkScript: ([]uint8) (len=34 cap=500) {
    00000000  00 20 86 dd e9 cd bb b9  4a b7 92 66 14 ab c8 68  |. ......J..f...h|
    00000010  34 bb 75 47 91 d4 9f f4  64 86 d6 9b 9f 71 ae 25  |4.uG....d....q.%|
    00000020  6e bc                                             |n.|
   }
  })
 },
 LockTime: (uint32) 0
})

2019-01-24 13:24:18.730 [INF] LNWL: Inserting unconfirmed transaction 8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425
2019-01-24 13:24:18.732 [ERR] FNDG: Unable to broadcast funding tx for ChannelPoint(8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1): Transaction rejected: output already spent
2019-01-24 13:24:18.732 [INF] CNCT: Creating new ChannelArbitrator for ChannelPoint(8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1)
2019-01-24 13:24:18.733 [INF] CNCT: ChannelArbitrator(8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1): starting state=StateDefault
2019-01-24 13:24:18.733 [INF] NTFN: New block epoch subscription
2019-01-24 13:24:18.733 [INF] NTFN: New spend subscription: spend_id=169, outpoint=8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1, height_hint=559898
2019-01-24 13:24:18.737 [INF] FNDG: Finalizing pendingID(aa38ac947faf6a608443717554df0ec1dadfc4c2449276e7dc811db6e6633eae) over ChannelPoint(8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1), waiting for channel open on-chain
2019-01-24 13:24:18.737 [INF] NTFN: New confirmation subscription: txid=8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425, numconfs=3
2019-01-24 13:24:18.737 [INF] CNCT: Close observer for ChannelPoint(8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1) active
2019-01-24 13:24:18.737 [INF] FNDG: Waiting for funding tx (8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425) to reach 3 confirmations
2019-01-24 13:24:18.737 [INF] ATPL: Triggering attachment directive dispatch, total_funds=0.08334517 BTC
2019-01-24 13:24:18.812 [ERR] NTFN: Rescan to determine the spend details of 8a25db9e9eb2e340b436e4237e0e5bd9022477b883badc2995857d1b34092425:1 failed: transaction not found within range

LNBIG-COM on 12 Feb 2019

@LNBIG-COM force closing the channel actually just makes the situation worse, as the AbandonChannel RPC won't completely remove all state for force closed channels. It should only be used for channels that are pending open due to the funding transaction being invalid. In your case, the outputs your funding transactions are attempting to spend are not even confirmed, so there's most likely a conflict. I'd be careful when abandoning these channels as they can still confirm if they actually propagate to other nodes' mempools.

wpaulino on 13 Feb 2019

Has time has come to add that safety check to abandon channel to only do so for pending open channels?

cfromknecht on 13 Feb 2019

I have the same issue with more than 3 BTC missing after force-closing all channels, walletbalance is showing zero. Using LND 7.1, tried with the newest 8.0 - same results... while trying to make it works from some backup it's returning wide range of errors.

As I know, backups are used to recover something and if somebody told me "_Don't use LND backups!_" I should ask what's their purpose if it's not good to use them?

MasterNeuron on 6 Oct 2019

@MasterNeuron after force-closing your funds need to wait for a timelock before they show up in your wallet. This can be up to 2000 blocks (~2 weeks) for large channels. Only after the timelock has passed can LND sweep the funds into your wallet.

What people probably meant when saying "Don't use LND backups!" is: Don't use a wallet.db or channel.db from a file system backup you made, it could lead to your node having an old channel state and losing all channel funds.
What is safe to use are Static Channel Backups (SCBs), they are in a file called channels.backup. This file should be copied to a secure location and updated after every new channel that is opened.

But restoring that SCB file means disaster recovery: All channels are force-closed and no normal operation will be possible on those channels afterwards.