Lightning: AWAITING_UNILATERAL bug

[Eskyee]

AWAITING_UNILATERAL channels stuck and seems. to crash my node. at different times,
my node is over a year and 3 months. old, I guess it seems related to old nodes ,

how c-lightning can knock out my node due to a CPU spike is strange to me;
would love some overstanding on this issue

info:
id: 0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c
alias: btcpayjungle-lightning-node
color: 0265e5
num_peers: 2
num_pending_channels: 1
num_active_channels: 2
num_inactive_channels: 0
address:
- type: ipv4
address: 51.144.112.185
port: 9735
binding:
- type: ipv4
address: 0.0.0.0
port: 9735
version: basedon-v0.7.0-3
blockheight: 578161
network: bitcoin
msatoshi_fees_collected: 11293
fees_collected_msat: 11293msat
peers:
- id: 022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7
connected: false
channels:
- state: AWAITING_UNILATERAL
scratch_txid: 8aefc94e1eae524625a1cab2c275c1ce3678d56a89de2e27a4bc22e9ecf0098e
short_channel_id: 522794x2541x0
direction: 1
channel_id: ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a57
funding_txid: 573a28ca041c1e6423e4e846f382d5d8c33b7fa1fbf0953ac27efd34f6731fae
private: false
funding_allocation_msat:
0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c: 0
022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7: 5000000
funding_msat:
0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c: 0msat
022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7: 5000000msat
msatoshi_to_us: 0
to_us_msat: 0msat
msatoshi_to_us_min: 0
min_to_us_msat: 0msat
msatoshi_to_us_max: 0
max_to_us_msat: 0msat
msatoshi_total: 5000000
total_msat: 5000000msat
dust_limit_satoshis: 546
dust_limit_msat: 546000msat
max_htlc_value_in_flight_msat: '18446744073709551615'
max_total_htlc_in_msat: 18446744073709551615msat
their_channel_reserve_satoshis: 0
their_reserve_msat: 0msat
our_channel_reserve_satoshis: 546
our_reserve_msat: 546000msat
spendable_msatoshi: 0
spendable_msat: 0msat
htlc_minimum_msat: 0
minimum_htlc_in_msat: 0msat
their_to_self_delay: 144
our_to_self_delay: 144
max_accepted_htlcs: 483
status:
- 'AWAITING_UNILATERAL:Attempting to reconnect'
in_payments_offered: 0
in_msatoshi_offered: 0
in_offered_msat: 0msat
in_payments_fulfilled: 0
in_msatoshi_fulfilled: 0
in_fulfilled_msat: 0msat
out_payments_offered: 0
out_msatoshi_offered: 0
out_offered_msat: 0msat
out_payments_fulfilled: 0
out_msatoshi_fulfilled: 0
out_fulfilled_msat: 0msat
htlcs: []
- id: 02809e936f0e82dfce13bcc47c77112db068f569e1db29e7bf98bcdd68b838ee84
connected: true
netaddr:
- '40.69.71.114:54450'
global_features: ''
local_features: '81'
globalfeatures: ''
localfeatures: '81'
channels:
- state: AWAITING_UNILATERAL
scratch_txid: d2cb2266ac4b3487d42f57b6e70142222a4c0b283a6de161acc98832868c4e36
short_channel_id: 567080x257x0
direction: 0
channel_id: e53623375ea4a467e5476ddd132d9fd80a95e213aa51b493319e8bd8da6c6564
funding_txid: 64656cdad88b9e3193b451aa13e2950ad89f2d13dd6d47e567a4a45e372336e5
private: false
funding_allocation_msat:
0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c: 0
02809e936f0e82dfce13bcc47c77112db068f569e1db29e7bf98bcdd68b838ee84: 100000000
funding_msat:
0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c: 0msat
02809e936f0e82dfce13bcc47c77112db068f569e1db29e7bf98bcdd68b838ee84: 100000000msat
msatoshi_to_us: 0
to_us_msat: 0msat
msatoshi_to_us_min: 0
min_to_us_msat: 0msat
msatoshi_to_us_max: 0
max_to_us_msat: 0msat
msatoshi_total: 100000000
total_msat: 100000000msat
dust_limit_satoshis: 546
dust_limit_msat: 546000msat
max_htlc_value_in_flight_msat: '18446744073709551615'
max_total_htlc_in_msat: 18446744073709551615msat
their_channel_reserve_satoshis: 1000
their_reserve_msat: 1000000msat
our_channel_reserve_satoshis: 1000
our_reserve_msat: 1000000msat
spendable_msatoshi: 0
spendable_msat: 0msat
htlc_minimum_msat: 0
minimum_htlc_in_msat: 0msat
their_to_self_delay: 144
our_to_self_delay: 144
max_accepted_htlcs: 483
status:
- 'AWAITING_UNILATERAL:Attempting to reconnect'
in_payments_offered: 0
in_msatoshi_offered: 0
in_offered_msat: 0msat
in_payments_fulfilled: 0
in_msatoshi_fulfilled: 0
in_fulfilled_msat: 0msat
out_payments_offered: 0
out_msatoshi_offered: 0
out_offered_msat: 0msat
out_payments_fulfilled: 0
out_msatoshi_fulfilled: 0
out_fulfilled_msat: 0msat
htlcs: []

My Node!!

getinfo output

{
"id": "0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c",
"alias": "btcpayjungle-lightning-node",
"color": "0265e5",
"num_peers": 3,
"num_pending_channels": 2,
"num_active_channels": 2,
"num_inactive_channels": 0,
"address": [
{
"type": "ipv4",
"address": "51.144.112.185",
"port": 9735
}
],
"binding": [
{
"type": "ipv4",
"address": "0.0.0.0",
"port": 9735
}
],
"version": "basedon-v0.7.0-3",
"blockheight": 578161,
"network": "bitcoin",
"msatoshi_fees_collected": 11293,
"fees_collected_msat": "11293msat"
}

Channel "AWAITING_UNILATERAL" since days. Attempted force closing, CPU 100% this happen about two weeks ago to me,

tried closing out all channels to clear the problem, and force close,
but this has come back again, seems I can't get rid of the last channels

posted this issue in btcpay server GitHub incase others discover this problem,

2019-05-27T14:55:43.696Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13924: Peer connection lost
2019-05-27T14:55:43.697Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13924: Owning subdaemon lightning_openingd died (9)
2019-05-27T15:15:43.527Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13925: Peer connection lost
2019-05-27T15:15:43.528Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13925: Owning subdaemon lightning_openingd died (9)
2019-05-27T15:28:12.532Z lightningd(5): bitcoin-cli: finished bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind estimatesmartfee 100 ECONOMICAL (31251 ms)
2019-05-27T15:28:12.633Z lightningd(5): bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind estimatesmartfee 100 ECONOMICAL exited with status 1
2019-05-27T15:29:20.734Z lightningd(5): bitcoin-cli: finished bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind getblockhash 578048 (80002 ms)
2019-05-27T15:29:28.336Z lightningd(5): bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind getblockhash 578048 exited 1 (after 1 other errors) 'error: Could not connect to the server bitcoind:43782 (error code 1 - "EOF reached")??Make sure the bitcoind server is running and that you are connecting to the correct RPC port.?'
2019-05-27T15:29:53.836Z lightningd(5): FATAL SIGNAL 6 (version basedon-v0.7.0-3)
2019-05-27T15:29:57.136Z lightningd(5): backtrace: common/daemon.c:45 (crashdump) 0x55bd0884276c
2019-05-27T15:30:00.287Z lightningd(5): backtrace: (null):0 ((null)) 0x7f525ed0c0e9
bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind getblockhash 578048 exited 1 (after 1 other errors) 'error: Could not connect to the server bitcoind:43782 (error code 1 - "EOF reached")
Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
'
lightningd: Fatal signal 6 (version basedon-v0.7.0-3)
0x55bd08842718 crashdump
common/daemon.c:40
0x7f525ed0c0e9 ???
???:0
Log dumped in crash.log.20190527153000
network=bitcoin added in /root/.lightning/config
alias=btcpayjungle-lightning-node added to /root/.lightning/config
Waiting /root/.nbxplorer/btc_fully_synched to be created...
The chain is fully synched
C-Lightning starting, listening on port 9735
2019-05-27T20:32:59.215Z lightningd(5): --------------------------------------------------
2019-05-27T20:32:59.215Z lightningd(5): Server started with public key 0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c, alias btcpayjungle-lightning-node (color #0265e5) and lightningd basedon-v0.7.0-3
2019-05-27T20:34:20.238Z lightningd(5): lightning_openingd-02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #13519: Peer connection lost
2019-05-27T20:34:20.239Z lightningd(5): 02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #13519: Owning subdaemon lightning_openingd died (9)
2019-05-27T20:35:43.955Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13518: Peer connection lost
2019-05-27T20:35:43.955Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13518: Owning subdaemon lightning_openingd died (62208)
2019-05-27T20:45:43.419Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13521: Peer connection lost
2019-05-27T20:45:43.420Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13521: Owning subdaemon lightning_openingd died (62208)

https://github.com/btcpayserver/btcpayserver/issues/839

[Eskyee]

https://github.com/ElementsProject/lightning/issues/2171
https://github.com/ElementsProject/lightning/issues/1865
https://github.com/ElementsProject/lightning/issues/796
https://github.com/ElementsProject/lightning/issues/1865

seems related to a few issues here,

[ZmnSCPxj]

AWAITING_UNILATERAL is the force-closing state, it means "we are already doing a force close".

High CPU load is probably not related to the AWAITING_UNILATERAL but if it is, top should report lightning_onchaind as having high CPU load; if it is lightningd or lightning_gossipd, it is (probably) not caused by AWAITING_UNILATERAL.

[Eskyee]

I managed to close out around 20 channels two weeks ago, force-closed a couple, recovered funds from most channels ok,

but then noticed a stuck peer and two active channels they are not force-closing, as this seems stuck
to me as weeks have gone by,

a few more logs, if that helps

C-Lightning starting, listening on port 9735
2019-05-14T23:52:11.056Z lightningd(5): Restarting onchaind for channel 6091
2019-05-14T23:52:11.056Z lightningd(5): 028303182c9885da93b3b25c9621d22cf34475e63c123942e402ab530c0556e675 chan #6091: Peer permanent failure in ONCHAIN: Funding transaction spent
2019-05-14T23:52:11.056Z lightningd(5): (tx e15db667fb706165f7e7529af151ea294a26f7d3324a95d70a45a90d82f1b327)
2019-05-14T23:52:11.064Z lightningd(5): 028303182c9885da93b3b25c9621d22cf34475e63c123942e402ab530c0556e675 chan #6091: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2019-05-14T23:52:11.632Z lightningd(5): --------------------------------------------------
2019-05-14T23:52:11.632Z lightningd(5): Server started with public key 0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c, alias btcpayjungle-lightning-node (color #0265e5) and lightningd basedon-v0.7.0-3
2019-05-14T23:52:13.108Z lightningd(5): 028303182c9885da93b3b25c9621d22cf34475e63c123942e402ab530c0556e675 chan #6091: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-05-14T23:54:44.073Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13932: Peer connection lost
2019-05-14T23:54:44.073Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13932: Owning subdaemon lightning_openingd died (62208)
2019-05-14T23:55:29.520Z lightningd(5): 0363f6132fa34f0cffcf18b94218cacff7fefebbd085ffdd0cb1bd37d12461f685 chan #9724: State changed from CHANNELD_NORMAL to CHANNELD_SHUTTING_DOWN
2019-05-14T23:57:44.230Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13934: Peer connection lost
2019-05-14T23:57:44.230Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13934: Owning subdaemon lightning_openingd died (62208)
2019-05-15T00:03:50.229Z lightningd(5): 038bcbbbcef1b32a8f566f2ada3c2b15cad089a3e23b259bbcab42fd464fe4770d chan #5814: Peer transient failure in CHANNELD_NORMAL: Reconnected
2019-05-15T00:04:09.713Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from CHANNELD_AWAITING_LOCKIN to CHANNELD_SHUTTING_DOWN
2019-05-15T00:18:58.434Z lightningd(5): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13935: Peer connection lost
2019-05-15T00:18:58.435Z lightningd(5): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13935: Owning subdaemon lightning_openingd died (9)
2019-05-15T00:22:46.443Z lightningd(5): 03557fd11b58cb93d2ad4fab4dd4cff7462a97e21e8f6b7a712cb8bd27a96b9eef chan #37: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:22:46.443Z lightningd(5): (tx dd62411faa7318127dc3f278513010db82c54d578b6630eaec7a4343f23d7be9)
2019-05-15T00:22:46.449Z lightningd(5): 03557fd11b58cb93d2ad4fab4dd4cff7462a97e21e8f6b7a712cb8bd27a96b9eef chan #37: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:24:53.131Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:24:53.131Z lightningd(5): (tx 4aceda0d6a00c73f2b00c18c7bb13a9bb8047ca58fd03a1ee6a3bdfdf83c6073)
2019-05-15T00:24:53.140Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:26:49.742Z lightningd(5): 0363f6132fa34f0cffcf18b94218cacff7fefebbd085ffdd0cb1bd37d12461f685 chan #9724: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:26:49.742Z lightningd(5): (tx 5afc33b600eb045ed9fa14e6a3256fbd6492036561e1e43242fca9f69c0fbf3f)
2019-05-15T00:26:49.748Z lightningd(5): 0363f6132fa34f0cffcf18b94218cacff7fefebbd085ffdd0cb1bd37d12461f685 chan #9724: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:29:58.753Z lightningd(5): 023668a30d0a27304695df3fb1af55a4fb75153eac34840817cae0e6a57894fd51 chan #9: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:29:58.753Z lightningd(5): (tx 74b1b9ee3a4938f3d08bc2cae5b0ea77e11920444d75a58caa3375dffcb0944e)
2019-05-15T00:29:58.757Z lightningd(5): 023668a30d0a27304695df3fb1af55a4fb75153eac34840817cae0e6a57894fd51 chan #9: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:30:50.694Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-05-15T00:30:50.694Z lightningd(5): (tx 4aceda0d6a00c73f2b00c18c7bb13a9bb8047ca58fd03a1ee6a3bdfdf83c6073)
2019-05-15T00:30:50.696Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-05-15T00:30:50.696Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-05-15T00:30:51.588Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-05-15T00:45:20.217Z lightningd(5): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13936: Peer connection lost
2019-05-15T00:45:20.220Z lightningd(5): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13936: Owning subdaemon lightning_openingd died (9)
2019-05-15T00:49:20.747Z lightningd(5): 022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7 chan #36: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:49:20.747Z lightningd(5): (tx 8aefc94e1eae524625a1cab2c275c1ce3678d56a89de2e27a4bc22e9ecf0098e)
2019-05-15T00:49:20.752Z lightningd(5): 022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7 chan #36: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:50:23.719Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T00:50:24.429Z lightningd(5): 038b95886d1e9a93b66c7bf782fe279ef41b3b25d65cc9822d836309bfa641f144 chan #25: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:50:24.429Z lightningd(5): (tx d21f87a726e095a4b1727b5e85b3d4831f82b41585f83d3246e78762af6691df)
2019-05-15T00:50:24.435Z lightningd(5): 038b95886d1e9a93b66c7bf782fe279ef41b3b25d65cc9822d836309bfa641f144 chan #25: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:54:27.610Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:04:28.481Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:09:59.473Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:12:34.745Z lightningd(5): lightning_openingd-03ea061d3b061e2a5174ae7ccc09bf47ebeb1d3696f89e24e8d971ba1bed7462d0 chan #13937: Peer connection lost
2019-05-15T01:12:34.746Z lightningd(5): 03ea061d3b061e2a5174ae7ccc09bf47ebeb1d3696f89e24e8d971ba1bed7462d0 chan #13937: Owning subdaemon lightning_openingd died (62208)
2019-05-15T01:15:33.355Z lightningd(5): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13938: Peer connection lost
2019-05-15T01:15:33.355Z lightningd(5): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13938: Owning subdaemon lightning_openingd died (62208)
2019-05-15T01:19:00.850Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:22:31.596Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:34:02.978Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:47:17.178Z lightningd(5): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13939: Peer connection lost
2019-05-15T01:47:17.179Z lightningd(5): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13939: Owning subdaemon lightning_openingd died (9)
2019-05-15T02:06:35.285Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:08:06.289Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:10:37.427Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:14:18.965Z lightningd(5): lightning_openingd-02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #13940: Peer connection lost
2019-05-15T02:14:18.966Z lightningd(5): 02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #13940: Owning subdaemon lightning_openingd died (9)
2019-05-15T02:22:39.466Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:25:43.403Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13941: Peer connection lost
2019-05-15T02:25:43.403Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13941: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:35:43.942Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13942: Peer connection lost
2019-05-15T02:35:43.943Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13942: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:44:41.862Z lightningd(5): lightning_openingd-0313edbe53a41171af2bd1baf76d9095aa58806a05793a91b750d7f02bffe1f38d chan #13943: Peer connection lost
2019-05-15T02:44:41.863Z lightningd(5): 0313edbe53a41171af2bd1baf76d9095aa58806a05793a91b750d7f02bffe1f38d chan #13943: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:45:11.487Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:45:43.703Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13945: Peer connection lost
2019-05-15T02:45:43.703Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13945: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:45:46.567Z lightningd(5): lightning_openingd-021067b3b56fdad3985168f4bd5ba31f1c0e3dbcd6c908675f4be5afb99edb7e39 chan #13944: Peer connection lost
2019-05-15T02:45:46.567Z lightningd(5): 021067b3b56fdad3985168f4bd5ba31f1c0e3dbcd6c908675f4be5afb99edb7e39 chan #13944: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:49:42.275Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?

btcpayserver_bitcoind btcpayserver/lightning:v0.7.0-3
I gets high cpu spikes when my btcpay server node crash! from c-lightning when Im monitoring with docker stats, strange thing is only happens every now and then!!

[Eskyee]

I can also check with my spark wallet

[darosior]

nits: You can use the <details> markup for logs, it makes the conversation follow-up easier ;)

<details>
<summary>Click me to see logs!</summary>
<p>logs</p>
</details>

About your problem, what was the capacity of your channel ? It seems that the unilateral close transaction is created with no output ( == all capacity is consumed by fees)

[Eskyee]

https://1ml.com/node/0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c
thanks I will check details
also you can check my node history here, 1ml

[Eskyee]

https://1ml.com/channel/623511053896056832 been trying to close this channel for the last couple of weeks, also this seem to be the node I can't disconnect AWAITING_UNILATERAL ..
Still a noob with c-lightning but this seem to be my problem node here https://1ml.com/node/022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7

[ZmnSCPxj]

This is the relevant part:

   error code: -26?error message:?bad-txns-vout-empty (code 16)?

This can happen if the HTLC and channel values are so low that mining fees dominate over it. Looking at your channels:

   msatoshi_total: 5000000
   msatoshi_total: 100000000

The first is 5000 satoshi, or 0.05 mBTC, which is rather small.

The other one is 1mBTC though. And the invalid Bitcoin transaction seems to be only a single one, so it is likely to be the smaller channel.

This bug is already supposedly fixed #632 and #1171 ; this was a spec-level bug I believe.

@cdecker @rustyrussell do you have idea why this is apparently happening again?

Even so, it probably is not related to the CPU load problem, because if we look at timestamps:

   2019-05-15T02:06:35.285Z lightningd(5): Broadcasting tx 
   2019-05-15T02:08:06.289Z lightningd(5): Broadcasting tx 
   2019-05-15T02:10:37.427Z lightningd(5): Broadcasting tx

It's one and a half minutes between attempts to broadcast, so it should not be the cause of the CPU load.

[Eskyee]

Thanks for the information
Makes more sense to me now,

I will check my CPU spikes timeline to find the cause of this with my BTCPAY Server

Cheers