Lightning: Tried to pay but getting WIRE_TEMPORARY_CHANNEL_FAILURE

[satoshidoriannakamoto]

Tried:

cli/lightning-cli pay lnbcxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Getting this error...

lightningd(8817): Sending 201231786 over 2 hops to deliver 201230585
lightningd(8817): peer 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8: htlc 0 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
lightningd(8817): Sending 201231786 over 2 hops to deliver 201230585
lightningd(8817): peer 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8: htlc 1 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
lightningd(8817): Sending 201231786 over 2 hops to deliver 201230585
lightningd(8817): peer 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8: htlc 2 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)

[ZmnSCPxj]

This will eventually be fixed by #638. Temporary channel failures should cause pay command to retry, not error, and it will try another route (or fail to find a route and definitely fail).

An issue not fixed by #638 is if one of your local channels is not in a useable state (not yet locked in, or already closing, or depleted) and routing decides to use it; currently the routing algorithm does not pay attention to local channel state (it should!).

[satoshidoriannakamoto]

I think I lost $50. Reckless! But worth it.

[ZmnSCPxj]

Hmm it should not happen money loss for #638. Can you give more detail why you think money loss?

[satoshidoriannakamoto]

I had a channel open and made a lightning BOLT payment to Blockstream. The payment failed, hence, "WIRE_TEMPORARY_CHANNEL_FAILURE". I close the channel. Now the state of that channel says, "ONCHAIND_MUTUAL". I try to reopen the channel:

cli/lightning-cli connect < ip xxx >

But I get this msg:

channel ALL: lightning_closingd: bad reestablish msg: WIRE_PING 001200100000"

So I have money stuck and I'm not sure what do do. Can you help? Thanks.

Additional info:

lightning_gossipd(12732): TRACE: hand_back_peer 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8: now local again
lightning_gossipd(12732): TRACE: 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8: we sent them a fatal error, closing

[ZmnSCPxj]

Channel close is an onchain tx. You will have to wait for it onchain. Unfortunately this can take some time depending on congestion onchain. You got a mutual close, which is slightly cheaper and has no timelock, meaning you will be able to get the funds as soon as the closing tx confirms. We cannot show the txid of the close tx yet, because showing it on close is not yet implemented, see #714 and #622. We cannot accelerate it either yet, as we need some kind of CPFP+RBF, see #475.

[satoshidoriannakamoto]

Ok. I will wait then. I managed to open another channel with a different node and bought a sticker and a t-shirt successfully. I should have not connected to a node called "RECKLESS" in the first place probably.

[cdecker]

I think I lost $50. Reckless! But worth it.

Since the close was mutual you should get your funds back. Could you try lightning-cli listfunds to check?

[satoshidoriannakamoto]

I did a lightning-cli listfunds but it is not there yet. I then did a cli/lightning-cli listpeers and the peer with the funds has a state of ONCHAIND_MUTUAL. It has been 6 hours so I'll continue to wait it out.

[ianthius]

jimgolian try to comb your logs for a txid of the closing tx. It's also maybe in the sql db?

[satoshidoriannakamoto]

Where is the log file? Not sure how to query sql db.

I just did a cli/lightning-cli listpeers and the the peer with the state of ONCHAIND_MUTUAL has disappeared. I thought my funds were recovered, but when I did lightning-cli listfunds, the funds from that failed transaction was not there. Hmmm...

[ZmnSCPxj]

The data should be in ~/.lightning/ directory, including a lighting.sqlite3 file (the database). Logs are usually placed there (usually crash,.log in case of crash).

This is strange, it should be in listfunds if the peer has disappeared.

[cdecker]

Notice that the actual transaction ID will not be the same as the funding transaction ID, instead the funds will be listed under a the settlement TX ID or one of its children.

[satoshidoriannakamoto]

Checked crash.log but nothing much in there. I am querying the sqlite database and see tables. Which table and columns should I be looking at for the transaction ID? I went through all of the tables and a lot of them have BLOB data I can't read easily.

[cdecker]

SELECT hex(prev_out_tx), prev_out_index, status FROM outputs; should show you every output we know of and its state (the hashes are reversed though).

[satoshidoriannakamoto]

Here is the output of the SQL query:

F478BD7753DABB2BA9CA961AD28EACD9BCF7B5D22839105E10927D041DCBBD05|1|2
06EAE3CAC6335F535530809D9196D55F6CCB218FDB5506A6320D0282AFA19933|0|2
D63B2D720E70CB7F3C08CEC674C5161E0C270EB3501A13F680595BBDD4DC6F14|0|2
902B5D45167C17D8FD3A2C6160BB3752B02EBC12513EEA0C77D1C529AC8AC114|0|2
866AFA7DA47422E02A75CAAAB1D73D391F464A0B967BD8503F38B393689AF0D6|0|2
22A6227CB240E735B8E996ED2B2188FEB6D45F182DAB921F1DDACE9406A0FA7B|1|0

[satoshidoriannakamoto]

Well, this is strage. I ran the command "listpeers" and the transaction with the missing funds showed up again as a peer. Just earlier today it was gone from the list, and now it is back. It still has a state of "ONCHAIND_MUTUAL".

A command of "listfunds" reveals the bitcoin in the failed transaction has not been recovered yet.

[ZmnSCPxj]

@jimgolian Has this gotten fixed?

[satoshidoriannakamoto]

Been a while since I ran c-lightning, but I booted up my Linux box and checked. I did a cli/lightning-cli listpeers and still see the ONCHAIND_MUTUAL.

[cdecker]

Please update to the latest master and restart c-lightning. The onchaind state should disappear since the channel is now closed for good and listfunds should show you the returned funds.

[satoshidoriannakamoto]

I updated by doing a 'git pull' and 'make'. I ran 'cli/lightning-cli listpeers'. I get:

1 ONCHAIN
4 CHANNELD_NORMAL
1 CHANNELD_AWAITING_LOCKIN

I no longer see ONCHAIND_MUTUAL.

I did a 'cli/lightning-cli listfunds' and I see six channels with various amounts of 'channel_total_sat'. I see one output with status "unconfirmed".

[cdecker]

Great, thanks for checking @jimgolian. From the above output from the SQlite query I think the channel was the one created by the funding transaction 3399a1af82020d32a60655db8f21cb6c5fd596919d803055535f33c6cae3ea06 and which was collaboratively closed 12 blocks later with settlement transaction 146fdcd4bd5b5980f6131a50b30e270c1e16c574c6ce083c7fcb700e722d3bd6, the funds were in your wallet (third entry in the list, but in reversed byte order), and where then used to fund a new channel in transaction 14c18aac29c5d1770cea3e5112bc2eb05237bb60612c3afdd8177c16455d2b90 (fourth entry above, and that channel is still active).