Lightning: Can't broadcast opening transactions since my node ran out of disk space

[chrisguida]

Issue and Steps to Reproduce

I've been running a node for a while now, based on Stadicus's c-lightning gist. Spruned has a memory leak which I haven't tracked down yet, but it no longer eats up all my memory because I have a cron job that resets it 4 times a day. Not ideal, but functional.

Anyway, my node ran out of disk space a couple weeks back and it seems like since then I can't open new channels, which is a problem because I have virtually no outbound capacity. The problem appears to be that the channel opening transactions aren't being broadcast. But the channel still shows up in my listpeers, and then when I try to close them, they just hang around forever in various closing states.

Does anyone have experience with this to help me debug? At the very least I probably need to reincarnate my node, but I'm uncertain if I can do this without losing funds.

Thanks for your assistance! :)

getinfo output

{
   "id" : "0370caf6a4aa0d1aacd133e2ceb9e61a6a26bae28dcf799bb43036e75329c0866e",
   "alias" : "MX_awesome",
   "color" : "0370ca",
   "num_peers" : 5,
   "num_pending_channels" : 0,
   "num_active_channels" : 4,
   "num_inactive_channels" : 1,
   "address" : [
      {
         "type" : "ipv4",
         "address" : "198.199.117.213",
         "port" : 9735
      }
   ],
   "binding" : [
      {
         "type" : "ipv6",
         "address" : "::",
         "port" : 9735
      },
      {
         "type" : "ipv4",
         "address" : "0.0.0.0",
         "port" : 9735
      }
   ],
   "version" : "v0.7.0-397-g22600fa",
   "blockheight" : 586098,
   "network" : "bitcoin",
   "msatoshi_fees_collected" : 0,
   "fees_collected_msat" : "0msat"
}

[cdecker]

@chrisguida not really a case we can protect against, but I can help you recover your funds.

First of all, make sure you have some disk space again, so that we can actually broadcast transactions again :wink:

Then you can run lightning-cli dev-rescan-outputs to reset the state of the outputs that were used to "create" the channels. This will make sure that you can then withdraw your funds to some other place (possibly an external wallet, while you tear down the node): lightning-cli withdraw [address] all. If you plan to recreate the node from scratch you should first close all channels that are not in state CHANNELD_AWAITING_LOCKIN, first with a normal lightning-cli close [peer_id] and then with a lightning-cli -k id=[node_id] force=true if the peer doesn't collaborate.

I suggest against attempting to salvage the node, but if you want to, you can manually mark the channels that will now never confirm because you double-spent their funding as CLOSED using the following command:

lightning-cli dev-forget-channel [peer_id]

or by manually modifying the database :warning: :

sqlite3 ~/.lightning/lightningd.sqlite3 "UPDATE channels SET state=10 WHERE state==2;"

[chrisguida]

Thanks for the response! So you think there's probably no way to debug it and get it running again? I was thinking I could use this as an opportunity to really dig into how c-lightning works. But I guess that might be slightly masochistic? :p

[chrisguida]

How long is dev-rescan-outputs supposed to take? It's been hanging for a few minutes. I guess I should have restarted my node with logging=debug first so I could follow along.

[chrisguida]

Hm, I got lightning-cli: reading response: socket closed. I guess it worked?

Anyway, one of my channels says AWAITING_UNILATERAL:Attempting to reconnect and the other says: Loaded from database. So now I need to decide whether to "salvage" my node... what was the reason for recommending against that? Is it pretty broken at this point?

[cdecker]

Getting the node to work again shouldn't be too hard, but manually editing the DB and using dev- commands could end up having undesired consequences, hence the suggestion to migrate to a new node.

dev-rescan-outputs will iterate through all of your funds and ask bitcoind whether it considers the output spent or not. The runtime depends on the number of outputs and how quick your bitcoind is, but minutes sounds a bit long. Any logs you can share?

Whatever you do I'd definitely withdraw to a new address, just to make sure that any funding transaction that didn't confirm is double-spent and can't confirm later.

[chrisguida]

I turned log-level to debug, restarted my node, and re-ran dev-rescan-outputs, but there's still no log output from the command. It just hangs there.

Okay, I see, I'll withdraw to a new address.

[chrisguida]

dev-rescan-outputs still hanging after 15 minutes. Is it safe to proceed without this command completing properly?

[chrisguida]

Getting this error while trying to withdraw:

$ lightning-cli withdraw [bech32 address] all
{
   "code" : 301,
   "message" : "Cannot afford transaction"
}

Guess my node is pretty screwed up :/

[chrisguida]

dev-rescan-outputs still hanging.

[ZmnSCPxj]

Looks like some trouble with the bitcoind? We might not have a decent fee estimate, and dev-rescan-outputs asks the bitcoind about the status of every output.

[chrisguida]

I suppose I should switch to a real bitcoind. Currently I'm using sPRUNED. Seems to be more trouble than it's worth at this point.

[cdecker]

Oh, yeah spruned is pretty much unusable when we have loads of gossip like we do currently, and the electrum servers on which they rely are under constant DDoS

[chrisguida]

Ok cool, I'll switch spruned to a full bitcoind instance and then run dev-rescan-outputs and see what I get.

[chrisguida]

Okay, I installed bitcoind, waited for it to sync to the chain, and reconfigured lightningd to use it instead of spruned. I'm running dev-rescan-outputs but it's hanging like it did before. What should I try now?

Thanks!

[chrisguida]

Oh wait never mind. Took several minutes but it finished! Huzzah!

[chrisguida]

Okay, I've run the command lightning-cli withdraw [address] all and it worked!

I have two properly functioning channels, and two channels whose opening transactions were never broadcast. These two invalid channels are refusing to close on a normal lightning-cli close [peer_id|channel_id].

@cdecker I also tried your suggestion of lightning-cli -k id=[node_id] force=true but I got the error message:

{
   "code" : -32601,
   "message" : "Unknown command 'id=03c2abfa93eacec04721c019644584424aab2ba4dff3ac9bdab4e9c97007491dda'"
}

I'm guessing you meant lightning-cli close -k id=[node_id] force=true ?

[chrisguida]

Here's the whole sequence:

$ lightning-cli close 03c2abfa93eacec04721c019644584424aab2ba4dff3ac9bdab4e9c97007491dda
{
   "code" : -1,
   "message" : "Peer has no active channel"
}

This is weird since this peer shows up in listpeers with a channel in state CLOSINGD_COMPLETE, status = Loaded from database

$ lightning-cli close a8d17d1d865b8ec23aa407f8d0c03d92a342e0d148911147cd1db7d0f1cd20bd
{
   "code" : -32602,
   "message" : "Channel ID not found: 'a8d17d1d865b8ec23aa407f8d0c03d92a342e0d148911147cd1db7d0f1cd20bd'"
}

This is the channel ID of the only channel I have with the peer above. It's definitely sitting right there in listpeers, how can I make it just go away?

$ lightning-cli close 5b2cce7a619d124d9da7505849ac59d324717af2708a8b29e77158e8e0314d31
{
   "code" : -1,
   "message" : "Channel is in state AWAITING_UNILATERAL"
}

$ lightning-cli -k id=03c2abfa93eacec04721c019644584424aab2ba4dff3ac9bdab4e9c97007491dda force=true
{
   "code" : -32601,
   "message" : "Unknown command 'id=03c2abfa93eacec04721c019644584424aab2ba4dff3ac9bdab4e9c97007491dda'"
}

[chrisguida]

Hey @cdecker I'm stuck, can you help me out?

Thanks :)

[darosior]

Hey @cdecker I'm stuck, can you help me out?

Thanks :)

"message" : "Channel is in state AWAITING_UNILATERAL

Seems like your channel has already been unilaterally closed. You can have a more informative message about the state of the unilateral close (i.e. the answer to "when can I withdraw my funds using a block chain transaction ?") using the listpeers command.

[chrisguida]

@darosior I'm not sure what you're saying. I withdrew all my on-chain funds, but i'm trying to get rid of the ghost channels and get my node back to where it can open new channels. Failing that, I'm trying to shut down all of my channels and spin up a new node.

@cdecker suggested the command lightning-cli -k id=[node_id] force=true to close the ghost channels, but it gave me an error, so I'm seeking clarification on what the correct command is.

I haven't tried yet, but I think I should be able to close my two normal channels normally since they were originally broadcast properly and are functioning normally.

I'd like to see if I can clear out the ghost channels and fix the problem that is preventing my broadcasting of new opening transactions and then continue operating my node. If this is impossible, I'd like to close all the channels, remove my funds, and spin up a new node.

Thanks for your help.

[chrisguida]

Anyone there? :)

[chrisguida]

Is there anywhere I can go to have a conversation? I feel like this problem could be resolved much more quickly over a different medium.

[darosior]

Hello chris,

Sorry I've indeed read your last message too quickly. Are you in sync ? (Is bitcoin-cli getblockcount the same number as shown in lightning-cli getinfo's blockheight field ?) If yes, have you run dev-rescan-outputs since being in sync ? What is the output of listpeers ?

[chrisguida]

Hi @darosior, yes, I'm in sync, yes, I ran dev-rescan-outputs in this comment, which enabled me to run lightning-cli withdraw [address] all in this comment.

[chrisguida]

The output of listpeers looks like this:

{ "peers" : [ { "id" : "03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f", "connected" : true, "netaddr" : [ "34.239.230.56:9735" ], "globalfeatures" : "", "localfeatures" : "8a", "channels" : [ { "state" : "CHANNELD_NORMAL", "scratch_txid" : "cccfb71a36c6fbb6d52ffbd7c272136459494df6c1ae119ab20f7db3fbe27f65", "owner" : "lightning_channeld", "short_channel_id" : "582407x2986x0", "direction" : 0, "channel_id" : "c2d1543d40b516f753335adf633f84b97221c67ce8d60ad918a44d5116db48fb", "funding_txid" : "fb48db16514da418d90ad6e87cc62172b9843f63df5a3353f716b5403d54d1c2", "private" : false, "funding_allocation_msat" : { "03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f" : 0, "0370caf6a4aa0d1aacd133e2ceb9e61a6a26bae28dcf799bb43036e75329c0866e" : 500000000 }, "funding_msat" : { "03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f" : "0msat", "0370caf6a4aa0d1aacd133e2ceb9e61a6a26bae28dcf799bb43036e75329c0866e" : "500000000msat" }, "msatoshi_to_us" : 500000000, "to_us_msat" : "500000000msat", "msatoshi_to_us_min" : 500000000, "min_to_us_msat" : "500000000msat", "msatoshi_to_us_max" : 500000000, "max_to_us_msat" : "500000000msat", "msatoshi_total" : 500000000, "total_msat" : "500000000msat", "dust_limit_satoshis" : 546, "dust_limit_msat" : "546000msat", "max_htlc_value_in_flight_msat" : 18446744073709551615, "max_total_htlc_in_msat" : "18446744073709551615msat", "their_channel_reserve_satoshis" : 5000, "their_reserve_msat" : "5000000msat", "our_channel_reserve_satoshis" : 5000, "our_reserve_msat" : "5000000msat", "spendable_msatoshi" : 495000000, "spendable_msat" : "495000000msat", "htlc_minimum_msat" : 0, "minimum_htlc_in_msat" : "0msat", "their_to_self_delay" : 144, "our_to_self_delay" : 720, "max_accepted_htlcs" : 483, "status" : [ "CHANNELD_NORMAL:Reconnected, and reestablished.", "CHANNELD_NORMAL:Funding transaction locked. Channel announced." ], "in_payments_offered" : 0, "in_msatoshi_offered" : 0, "in_offered_msat" : "0msat", "in_payments_fulfilled" : 0, "in_msatoshi_fulfilled" : 0, "in_fulfilled_msat" : "0msat", "out_payments_offered" : 43, "out_msatoshi_offered" : 284856, "out_offered_msat" : "284856msat", "out_payments_fulfilled" : 0, "out_msatoshi_fulfilled" : 0, "out_fulfilled_msat" : "0msat", "htlcs" : [] } ] }, { "id" : "03da1c27ca77872ac5b3e568af30673e599a47a5e4497f85c7b5da42048807b3ed", "connected" : true, "netaddr" : [ "46.229.165.142:9735" ], "globalfeatures" : "", "localfeatures" : "81", "channels" : [ { "state" : "CHANNELD_NORMAL", "scratch_txid" : "9e84fe37874045c9b307daa878f759cdc06eaa7dfadadd8b01e88ce73db2cf76", "owner" : "lightning_channeld", "short_channel_id" : "582445x713x1", "direction" : 0, "channel_id" : "5582ff60c1a9b75aa36e22a17b3dcee18b8b1f72b2a7c25632731671df1db68a", "funding_txid" : "8bb61ddf7116733256c2a7b2721f8b8be1ce3d7ba1226ea35ab7a9c160ff8255", "private" : false, "funding_allocation_msat" : { "0370caf6a4aa0d1aacd133e2ceb9e61a6a26bae28dcf799bb43036e75329c0866e" : 0, "03da1c27ca77872ac5b3e568af30673e599a47a5e4497f85c7b5da42048807b3ed" : 2020000000 }, "funding_msat" : { "0370caf6a4aa0d1aacd133e2ceb9e61a6a26bae28dcf799bb43036e75329c0866e" : "0msat", "03da1c27ca77872ac5b3e568af30673e599a47a5e4497f85c7b5da42048807b3ed" : "2020000000msat" }, "msatoshi_to_us" : 5010000, "to_us_msat" : "5010000msat", "msatoshi_to_us_min" : 0, "min_to_us_msat" : "0msat", "msatoshi_to_us_max" : 5010000, "max_to_us_msat" : "5010000msat", "msatoshi_total" : 2020000000, "total_msat" : "2020000000msat", "dust_limit_satoshis" : 546, "dust_limit_msat" : "546000msat", "max_htlc_value_in_flight_msat" : 18446744073709551615, "max_total_htlc_in_msat" : "18446744073709551615msat", "their_channel_reserve_satoshis" : 20200, "their_reserve_msat" : "20200000msat", "our_channel_reserve_satoshis" : 20200, "our_reserve_msat" : "20200000msat", "spendable_msatoshi" : 0, "spendable_msat" : "0msat", "htlc_minimum_msat" : 0, "minimum_htlc_in_msat" : "0msat", "their_to_self_delay" : 144, "our_to_self_delay" : 242, "max_accepted_htlcs" : 483, "status" : [ "CHANNELD_NORMAL:Reconnected, and reestablished.", "CHANNELD_NORMAL:Funding transaction locked. Waiting for their announcement signatures." ], "in_payments_offered" : 4, "in_msatoshi_offered" : 5020000, "in_offered_msat" : "5020000msat", "in_payments_fulfilled" : 3, "in_msatoshi_fulfilled" : 5010000, "in_fulfilled_msat" : "5010000msat", "out_payments_offered" : 0, "out_msatoshi_offered" : 0, "out_offered_msat" : "0msat", "out_payments_fulfilled" : 0, "out_msatoshi_fulfilled" : 0, "out_fulfilled_msat" : "0msat", "htlcs" : [] } ] }, { "id" : "03c2abfa93eacec04721c019644584424aab2ba4dff3ac9bdab4e9c97007491dda", "connected" : false, "channels" : [ { "state" : "CLOSINGD_COMPLETE", "scratch_txid" : "4c681956c06c09fbc33b96551ff539cb477ee4aed189e33c167ff04f92db6e25", "channel_id" : "a8d17d1d865b8ec23aa407f8d0c03d92a342e0d148911147cd1db7d0f1cd20bd", "funding_txid" : "bc20cdf1d0b71dcd47119148d1e042a3923dc0d0f807a43ac28e5b861d7dd1a8", "private" : false, "funding_allocation_msat" : { "03c2abfa93eacec04721c019644584424aab2ba4dff3ac9bdab4e9c97007491dda" : 0, "0370caf6a4aa0d1aacd133e2ceb9e61a6a26bae28dcf799bb43036e75329c0866e" : 1000000000 }, "funding_msat" : { "03c2abfa93eacec04721c019644584424aab2ba4dff3ac9bdab4e9c97007491dda" : "0msat", "0370caf6a4aa0d1aacd133e2ceb9e61a6a26bae28dcf799bb43036e75329c0866e" : "1000000000msat" }, "msatoshi_to_us" : 1000000000, "to_us_msat" : "1000000000msat", "msatoshi_to_us_min" : 1000000000, "min_to_us_msat" : "1000000000msat", "msatoshi_to_us_max" : 1000000000, "max_to_us_msat" : "1000000000msat", "msatoshi_total" : 1000000000, "total_msat" : "1000000000msat", "dust_limit_satoshis" : 546, "dust_limit_msat" : "546000msat", "max_htlc_value_in_flight_msat" : 18446744073709551615, "max_total_htlc_in_msat" : "18446744073709551615msat", "their_channel_reserve_satoshis" : 10000, "their_reserve_msat" : "10000000msat", "our_channel_reserve_satoshis" : 10000, "our_reserve_msat" : "10000000msat", "spendable_msatoshi" : 990000000, "spendable_msat" : "990000000msat", "htlc_minimum_msat" : 0, "minimum_htlc_in_msat" : "0msat", "their_to_self_delay" : 144, "our_to_self_delay" : 144, "max_accepted_htlcs" : 483, "status" : [ "Loaded from database" ], "in_payments_offered" : 0, "in_msatoshi_offered" : 0, "in_offered_msat" : "0msat", "in_payments_fulfilled" : 0, "in_msatoshi_fulfilled" : 0, "in_fulfilled_msat" : "0msat", "out_payments_offered" : 0, "out_msatoshi_offered" : 0, "out_offered_msat" : "0msat", "out_payments_fulfilled" : 0, "out_msatoshi_fulfilled" : 0, "out_fulfilled_msat" : "0msat", "htlcs" : [] } ] }, { "id" : "02a04446caa81636d60d63b066f2814cbd3a6b5c258e3172cbdded7a16e2cfff4c", "connected" : false, "channels" : [ { "state" : "AWAITING_UNILATERAL", "scratch_txid" : "9b87220e085b1ff201963d3b09093a8382fcc6263b37fbeee0faaf0d6386f149", "channel_id" : "5b2cce7a619d124d9da7505849ac59d324717af2708a8b29e77158e8e0314d31", "funding_txid" : "314d31e0e85871e7298b8a70f27a7124d359ac495850a79d4d129d617ace2c5b", "private" : false, "funding_allocation_msat" : { "02a04446caa81636d60d63b066f2814cbd3a6b5c258e3172cbdded7a16e2cfff4c" : 0, "0370caf6a4aa0d1aacd133e2ceb9e61a6a26bae28dcf799bb43036e75329c0866e" : 1000000000 }, "funding_msat" : { "02a04446caa81636d60d63b066f2814cbd3a6b5c258e3172cbdded7a16e2cfff4c" : "0msat", "0370caf6a4aa0d1aacd133e2ceb9e61a6a26bae28dcf799bb43036e75329c0866e" : "1000000000msat" }, "msatoshi_to_us" : 1000000000, "to_us_msat" : "1000000000msat", "msatoshi_to_us_min" : 1000000000, "min_to_us_msat" : "1000000000msat", "msatoshi_to_us_max" : 1000000000, "max_to_us_msat" : "1000000000msat", "msatoshi_total" : 1000000000, "total_msat" : "1000000000msat", "dust_limit_satoshis" : 546, "dust_limit_msat" : "546000msat", "max_htlc_value_in_flight_msat" : 18446744073709551615, "max_total_htlc_in_msat" : "18446744073709551615msat", "their_channel_reserve_satoshis" : 10000, "their_reserve_msat" : "10000000msat", "our_channel_reserve_satoshis" : 10000, "our_reserve_msat" : "10000000msat", "spendable_msatoshi" : 990000000, "spendable_msat" : "990000000msat", "htlc_minimum_msat" : 0, "minimum_htlc_in_msat" : "0msat", "their_to_self_delay" : 144, "our_to_self_delay" : 144, "max_accepted_htlcs" : 483, "status" : [ "AWAITING_UNILATERAL:Attempting to reconnect" ], "in_payments_offered" : 0, "in_msatoshi_offered" : 0, "in_offered_msat" : "0msat", "in_payments_fulfilled" : 0, "in_msatoshi_fulfilled" : 0, "in_fulfilled_msat" : "0msat", "out_payments_offered" : 0, "out_msatoshi_offered" : 0, "out_offered_msat" : "0msat", "out_payments_fulfilled" : 0, "out_msatoshi_fulfilled" : 0, "out_fulfilled_msat" : "0msat", "htlcs" : [] } ] }, { "id" : "024a05bfe6b562e01834f9ce3eb2921d3cdbc50cd39c0d03a78dabb027d137c3d6", "connected" : true, "netaddr" : [ "[::ffff:72.50.206.96]:53137" ], "globalfeatures" : "", "localfeatures" : "81", "channels" : [] }, { "id" : "02bd8d4e4355eef64735cf2d0bccb12647270bad0c615fc054e60bb28aa0b9bd84", "connected" : true, "netaddr" : [ "[::ffff:18.188.244.93]:53682" ], "globalfeatures" : "", "localfeatures" : "81", "channels" : [] }, { "id" : "030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f", "connected" : true, "netaddr" : [ "52.50.244.44:9735" ], "globalfeatures" : "", "localfeatures" : "81", "channels" : [] } ] }

[darosior]

Hmm if you are looking to close 5b2cce7a619d124d9da7505849ac59d324717af2708a8b29e77158e8e0314d31, it seems it never actually locked https://blockstream.info/tx/314d31e0e85871e7298b8a70f27a7124d359ac495850a79d4d129d617ace2c5b

[darosior]

If you install v0.7.2 (in rc for now, checkout the releases), the close behavior has been modified and you should be able to close any channel.

[chrisguida]

Hmm if you are looking to close 5b2cce7a619d124d9da7505849ac59d324717af2708a8b29e77158e8e0314d31, it seems it never actually locked https://blockstream.info/tx/314d31e0e85871e7298b8a70f27a7124d359ac495850a79d4d129d617ace2c5b

Yes, the title of this thread is "can't broadcast opening transactions". The bottom two channels appear only on my node, not publicly on the blockchain.

I'll download the new version and let you know how it goes, thanks! :)

[darosior]

Yes, the title of this thread is "can't broadcast opening transactions". The bottom two channels appear only on my node, not publicly on the blockchain.

Ok then check that you have your on chain funds as before (i.e. that the utxo the never broadcasted opening tx was spending) and you should be ok using dev-forget-channel.

[darosior]

How did it go @chrisguida ?

[chrisguida]

Hey @darosior, thanks for checking in, I'm working on this tomorrow, sorry I haven't had a chance to work on it yet :)

[chrisguida]

If you install v0.7.2 (in rc for now, checkout the releases), the close behavior has been modified and you should be able to close any channel.

Okay, I've installed 0.7.2 from the ppa (fresh off the presses) and I tried the close command on the bottom two channels and I get:

$ lightning-cli close a8d17d1d865b8ec23aa407f8d0c03d92a342e0d148911147cd1db7d0f1cd20bd
{
   "code": -32602,
   "message": "Channel ID not found: 'a8d17d1d865b8ec23aa407f8d0c03d92a342e0d148911147cd1db7d0f1cd20bd'"
}

$ lightning-cli close 5b2cce7a619d124d9da7505849ac59d324717af2708a8b29e77158e8e0314d31
{
   "code": -1,
   "message": "Channel is in state AWAITING_UNILATERAL"
}

When you say the close behavior has been modified and I can close any channel, what does that mean? Seems like close still does nothing to these channels. Should I just dev-forget-channel them?

[darosior]

If you install v0.7.2 (in rc for now, checkout the releases), the close behavior has been modified and you should be able to close any channel.

Okay, I've installed 0.7.2 from the ppa (fresh off the presses) and I tried the close command on the bottom two channels and I get:

$ lightning-cli close a8d17d1d865b8ec23aa407f8d0c03d92a342e0d148911147cd1db7d0f1cd20bd
{
   "code": -32602,
   "message": "Channel ID not found: 'a8d17d1d865b8ec23aa407f8d0c03d92a342e0d148911147cd1db7d0f1cd20bd'"
}

$ lightning-cli close 5b2cce7a619d124d9da7505849ac59d324717af2708a8b29e77158e8e0314d31
{
   "code": -1,
   "message": "Channel is in state AWAITING_UNILATERAL"
}

Ok so the first one seems to have already been forgotten ? The other one can, as mentioned above, be forgotten __if you are sure you are in possession of the funding utxo__ using dev-forget-channel.

When you say the close behavior has been modified and I can close any channel, what does that mean?

It means that as of 0.7.2(.1) the close command will force close after 48h by default. It doesn't affect you though since it seems your funding tx has never made it to the block chain.

Seems like close still does nothing to these channels. Should I just dev-forget-channel them?

After a double check (see above) I think so.

[chrisguida]

Ok so the first one seems to have already been forgotten ? The other one can, as mentioned above, be forgotten if you are sure you are in possession of the funding utxo using dev-forget-channel.

No, they're both still there as in the listpeers output I posted earlier. I'll dev-forget-channel them and see what happens.

When you say the close behavior has been modified and I can close any channel, what does that mean?

It means that as of 0.7.2(.1) the close command will force close after 48h by default. It doesn't affect you though since it seems your funding tx has never made it to the block chain.

Seems like close still does nothing to these channels. Should I just dev-forget-channel them?

After a double check (see above) I think so.

Okay cool, I'll dev-forget-channel them both, I'm positive that both never made it to the blockchain.

[chrisguida]

Hm, I guess I'm missing something. I'm getting this:

~$ lightning-cli dev-forget-channel 5b2cce7a619d124d9da7505849ac59d324717af2708a8b29e77158e8e0314d31
{
   "code": -32601,
   "message": "Unknown command 'dev-forget-channel'"
}

[darosior]

make clean && ./configure --enable-developer && make from 0.7.2.1 git tag (more or less current master) :-)

[chrisguida]

Ah okay. I suppose there wasn't much point in me installing from the ppa then haha

[chrisguida]

I'm getting a compile error:

ModuleNotFoundError: No module named 'mako'

What might be the problem?

[darosior]

I'm getting a compile error:

ModuleNotFoundError: No module named 'mako'

What might be the problem?

pip3 install mako or apt install python3-mako(on a Debian-like) : as you prefer

[chrisguida]

Boom! Zombie channels gone! Thanks @darosior!

Do you think I should try re-funding my node and trying to open new outgoing channels now, or do you think it's a better idea to spin up a new node? I did replace the bitcoin node, before it was sPRUNED and now it's a full bitcoind. Thoughts?

[ZmnSCPxj]

Probably better to spin up a new node, who knows how good the db state is now....

[chrisguida]

So the db is probably toast at this point? Any point in trying to debug it?

[ZmnSCPxj]

Maybe if we decide to support trying to hold on after running out of HDD? @cdecker may be more interested in DB than I am. In the interim, I suggest backing up the current lightning-dir and then make a new one, assuming you now have unlimited disk space that is.

[chrisguida]

Okay, so I guess the process is:

1) Close my 2 remaining open channels
2) Withdraw on-chain funds
3) stop lightningd
4) Back up and then delete the .lightning dir (guessing that's what you mean by "lightning-dir" @ZmnSCPxj)
5) Make a new .lightning where the old one was
6) restart lightningd

Sound good?

Thanks! Sorry for being so cautious, just trying to make sure I don't screw anything up :)

[darosior]

Once you have withdrawn all your funds you are safe deleting the .lightning and doing a fresh install (use the last release, not the master I mentioned above).

[chrisguida]

Cool, looks like tag v0.7.2.1 is actually the latest release, as mentioned here.

Rebuilding now...

[chrisguida]

Closing remaining channels...

What do I do if lightning-cli close hangs for a long time?

[chrisguida]

Oh wow, took 12 minutes but now I have

{
   "tx": "0200000001c2d1543d40b516f753335adf633f84b97221c67ce8d60ad918a44d5116db48fb0000000000ffffffff01ec92070000000000160014f2147f6afa895764480a1868fb7c80bc14d9eba300000000",
   "txid": "1c96b490a468ccfa95c56f5c1b04a8d2c686164aadadecafb869f032844a3e98",
   "type": "unilateral"
}

Guess they didn't want to cooperate :/

On to the next channel!

[chrisguida]

Hm, weird, still not seeing the transaction 1c96b490a468ccfa95c56f5c1b04a8d2c686164aadadecafb869f032844a3e98 on block explorers or on my bitcoin node. Maybe this is the same problem with broadcasting I was having with opening transactions. Shall I rebroadcast it?

[chrisguida]

Gonna wait to close the other active channel until I hear back. Thanks guys!

[chrisguida]

@darosior ?

[darosior]

Are we still talking about the ghost channel which funding tx never made it to the block chain ? If yes I've already answered above:

If you install v0.7.2 (in rc for now, checkout the releases), the close behavior has been modified and you should be able to close any channel.

Okay, I've installed 0.7.2 from the ppa (fresh off the presses) and I tried the close command on the bottom two channels and I get:

$ lightning-cli close a8d17d1d865b8ec23aa407f8d0c03d92a342e0d148911147cd1db7d0f1cd20bd
{
   "code": -32602,
   "message": "Channel ID not found: 'a8d17d1d865b8ec23aa407f8d0c03d92a342e0d148911147cd1db7d0f1cd20bd'"
}

$ lightning-cli close 5b2cce7a619d124d9da7505849ac59d324717af2708a8b29e77158e8e0314d31
{
   "code": -1,
   "message": "Channel is in state AWAITING_UNILATERAL"
}

Ok so the first one seems to have already been forgotten ? The other one can, as mentioned above, be forgotten if you are sure you are in possession of the funding utxo using dev-forget-channel.

When you say the close behavior has been modified and I can close any channel, what does that mean?

It means that as of 0.7.2(.1) the close command will force close after 48h by default. It doesn't affect you though since it seems your funding tx has never made it to the block chain.

Seems like close still does nothing to these channels. Should I just dev-forget-channel them?

After a double check (see above) I think so.

[chrisguida]

No, I nuked the ghost channels in this comment https://github.com/ElementsProject/lightning/issues/2819#issuecomment-523649163

I have two remaining open channels, which were broadcasted normally and on the public blockchain. I tried to close one but it seems to not want to close. What should I do?

[chrisguida]

Oh interesting, just logged back in and the channel is gone. Closing the final channel...

[chrisguida]

Cool, this one closed right away, status CLOSINGD_COMPLETE. Still in listpeers, I guess I'll wait for it to disappear then withdraw all.

[chrisguida]

Okay, I withdrew all, but it looks like I still have an output in listfunds:

{
   "outputs": [
      {
         "txid": "fb48db16514da418d90ad6e87cc62172b9843f63df5a3353f716b5403d54d1c2",
         "output": 1,
         "value": 585384,
         "amount_msat": "585384000msat",
         "address": "bc1qr2rlllpz0d35rdqlqwz55ugslzkw0ccqhs4wtj",
         "status": "unconfirmed"
      }
   ],
   "channels": []
}

The output appears not confirmed on my node, but on block explorers it appears to have been confirmed 2 months ago. How do I unstick this output?

[darosior]

Have you tried dev-rescan-outputs ?

[chrisguida]

Thanks, just tried it. No effect:

{
   "outputs": [
      {
         "txid": "fb48db16514da418d90ad6e87cc62172b9843f63df5a3353f716b5403d54d1c2",
         "output": 1,
         "value": 585384,
         "amount_msat": "585384000msat",
         "address": "bc1qr2rlllpz0d35rdqlqwz55ugslzkw0ccqhs4wtj",
         "status": "unconfirmed"
      }
   ],
   "channels": []
}

What else can I try?

[cdecker]

According to this there is a change output that is still unspent: https://blockstream.info/tx/fb48db16514da418d90ad6e87cc62172b9843f63df5a3353f716b5403d54d1c2

You may need to specify minconf=0 so it also picks up "unconfirmed" outputs. Notice that the unconfirmed status is a bug in this case and is fixed in #3024

[chrisguida]

You may need to specify minconf=0

Thanks, where do I specify this?

~$ lightning-cli dev-rescan-outputs minconf=0
{
   "code": -32602,
   "message": "unknown parameter: minconf"
}

~$ lightning-cli withdraw [address] all minconf=0
{
   "code": -32602,
   "message": "'feerate' prefix should be an integer, not 'minconf=0'"
}

Is this a configuration option?

Also wow, cool block explorer. Guess I'll be using that one moving forward :)

[chrisguida]

Tried putting minconf=0 in the config file and it crashed the program. Guess that's not it.

[darosior]

man lightning-withdraw ?

[chrisguida]

Oh wow, lightning-withdraw seems like a useful little utility. Unfortunately it doesn't seem to be included in my installation. How do I install it?

[chrisguida]

Oh wait, looks like man lightning-withdraw is just the man page for lightning-cli withdraw. Okay, I made it work with lightning-cli withdraw [address] all normal 0. Just needed that extra normal in there.

Woohoo! listfunds output:

{
   "outputs": [],
   "channels": []
}

Time to tear down this node...

[chrisguida]

Okay, looks like all is well!

New node at 0203a17c5b14ffdfb63c6ddff919fa20ef62a55b0f49954072a20deeb866d3af88@198.199.117.213:9735.

Thanks so much for your assistance!

[chrisguida]

I'd still like to learn how to debug and fix a database that has been corrupted like mine was, but I'll close this issue for now since my problem is fixed.

[niftynei]

as a note, if you want to specify parameters for a cli call by key rather than as an ordered array, you can use the -k flag.

lightning-cli dev-rescan-outputs -k minconf=0

[chrisguida]

Oh, thanks so much, @niftynei! That'll be super useful for later :)