At it again trying to make a donation:
(lc is my alias for lightning-cli)
lc pay lnbc1m1pdxcqsdpp5g4aanza2a205herm09tzkw4sul6y0zc0w9drmxrt3g9l2kpe070qdq6f38zq3tcwpkx7un9wgs9g6tswvcqzyst8wscs3k05d996quytvezsf2860hdlhw87p234anjhjht8q8zwgywtpn24zep6tcxx68duqky59glp5vu60t3685h7jcdkjmkwklxkcqxf0dd0
"failed: INVALID 0 (WIRE_TEMPORARY_CHANNEL_FAILURE: Maximum value exceeded)"
Why is that happening when all the intervening channels have enough funds?
lc decodepay lnbc1m1pdxcqsdpp5g4aanza2a205herm09tzkw4sul6y0zc0w9drmxrt3g9l2kpe070qdq6f38zq3tcwpkx7un9wgs9g6tswvcqzyst8wscs3k05d996quytvezsf2860hdlhw87p234anjhjht8q8zwgywtpn24zep6tcxx68duqky59glp5vu60t3685h7jcdkjmkwklxkcqxf0dd0
{ "currency" : "bc", "timestamp" : 1517027853, "created_at" : 1517027853, "expiry" : 3600, "payee" : "035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8", "msatoshi" : 100000000, "description" : "LN Explorer Tips", "min_final_cltv_expiry" : 144, "payment_hash" : "457bd98baaea9f4be47b79562b3ab0e7f4478b0f715a3d986b8a0bf558397f9e", "signature" : "3044022059dd0c42367d1a52e81c22d991412a3e9f76feee3f82a8d7b395e5759c0713900220472c33554590e97831b476f016250a8f868ce69eb8e8f4bfa586da5bb3adf35b" }
lc getroute 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8 100000000 1{ "route" :
[
{ "id" : "03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1", "channel" : "506216:1409:0", "msatoshi" : 100000100, "delay" : 153 },
{ "id" : "035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8", "channel" : "504481:631:1", "msatoshi" : 100000000, "delay" : 9 } ] }
Here's that channel:
lc listpeers:
{ "state" : "OPENINGD", "owner" : "lightning_openingd", "dust_limit_satoshis" :
546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat"
: 0, "to_self_delay" : 144, "max_accepted_htlcs" : 483 } ] },
{ "id" : "03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1", "connected" : true, "netaddr" :
[ "[::ffff:45.55.47.17]:51494" ], "channels" :
[
lc listchannels|egrep $me'.*03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1'
{ "source" : "03a3c1fc0a489e2872a2afe8fa16c2714b5fe80d52d8c1ce8ecc4361a77391da82", "destination" : "03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1", "short_channel_id" : "506216:1409:0", "flags" : 0, "active" : true, "public" : true, "last_update" : 1516991366, "base_fee_millisatoshi" : 1000, "fee_per_millionth" : 10, "delay" : 14 },
Here's the funding information:
Having trouble finding it. I reconnected to the peer. I found this:
PubKey 03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1
Alias lnd.rompert.com
URI 03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1@45.55.47.17:9735
Last Update January 26th 2018, 10:28:11 am
Color #ff0088
lc listpeers:
{ "state" : "OPENINGD", "owner" : "lightning_openingd", "dust_limit_satoshis" :
546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat"
: 0, "to_self_delay" : 144, "max_accepted_htlcs" : 483 } ] },
{ "id" : "03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1", "connected" : true, "netaddr" :
[ "[::ffff:45.55.47.17]:51494" ], "channels" :
Well, I'm confused. How do I look more into this?
lc listchannels:
{ "source" : "03a3c1fc0a489e2872a2afe8fa16c2714b5fe80d52d8c1ce8ecc4361a77391da82", "destination" : "03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1", "short_channel_id" : "506216:1409:0", "flags" : 0, "active" : true, "public" : true, "last_update" : 1516991366, "base_fee_millisatoshi" : 1000, "fee_per_millionth" : 10, "delay" : 14 },
Here's an undocumented command argument: I can put the channel in the listchannels command:
lc listchannels "506216:1409:0"
{ "channels" :
[
{ "source" : "03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1", "destination" : "03a3c1fc0a489e2872a2afe8fa16c2714b5fe80d52d8c1ce8ecc4361a77391da82", "short_channel_id" : "506216:1409:0", "flags" : 1, "active" : true, "public" : true, "last_update" : 1516982896, "base_fee_millisatoshi" : 1000, "fee_per_millionth" : 1, "delay" : 144 },
{ "source" : "03a3c1fc0a489e2872a2afe8fa16c2714b5fe80d52d8c1ce8ecc4361a77391da82", "destination" : "03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1", "short_channel_id" : "506216:1409:0", "flags" : 0, "active" : true, "public" : true, "last_update" : 1516991366, "base_fee_millisatoshi" : 1000, "fee_per_millionth" : 10, "delay" : 14 } ] }
I still don't know the funding transaction, the current balance, or the history of transactions.
There are literally no funding log entries ever for this channel:
$ egrep "506216:1409:0" ~/log/log*|sort -u
/home/ulmo/log/log:lightning_gossipd(13312): TRACE: Channel 506216:1409:0(0) was updated.
/home/ulmo/log/log:lightning_gossipd(13312): TRACE: Channel 506216:1409:0(0) was updated (LOCAL)
/home/ulmo/log/log:lightning_gossipd(13312): TRACE: Channel 506216:1409:0(1) was updated.
/home/ulmo/log/log:lightning_gossipd(13312): TRACE: Deferring update for pending channel 506216:1409:0(0)
/home/ulmo/log/log:lightning_gossipd(13312): TRACE: Deferring update for pending channel 506216:1409:0(1)
/home/ulmo/log/log:lightning_gossipd(13312): TRACE: Ignoring update for unknown channel 506216:1409:0
/home/ulmo/log/log:lightning_gossipd(13312): TRACE: Received channel_announcement for channel 506216:1409:0
/home/ulmo/log/log:lightning_gossipd(13312): TRACE: Received channel_update for channel 506216:1409:0(0)
/home/ulmo/log/log:lightning_gossipd(13312): TRACE: Received channel_update for channel 506216:1409:0(1)
So, I can't even find the funding transaction, or the balance, or the current channel capacity.
But, the payment should work, right? I tried it earlier today, and it failed, and I found the channel tp be used and its funding transaction, and it looked sufficient. Ok, trying again, reading log:
Here it is in the log:
lightningd(13304): Connected json input
lightning_gossipd(13312): TRACE: req: type WIRE_GOSSIP_GETROUTE_REQUEST len 78
lightning_gossipd(13312): TRACE: Trying to find a route from 03a3c1fc0a489e2872a2afe8fa16c2714b5fe80d52d8c1ce8ecc4361a77391da82 to 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8 for 100000000 msatoshi
lightning_gossipd(13312): TRACE: find_route: via 03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1
lightning_gossipd(13312): TRACE: 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8 (0+1=100)
lightning_gossipd(13312): TRACE: =100000000(+100)
lightning_gossipd(13312): REPLY WIRE_GOSSIP_GETROUTE_REPLY with 0 fds
lightningd(13304): Sending 100000100 over 2 hops to deliver 100000000
lightning_channeld(3074): TRACE: Adding HTLC 0 msat=100000100 cltv=506593 gave 4
lightning_channeld(3074): REPLY WIRE_CHANNEL_OFFER_HTLC_REPLY with 0 fds
lightningd(13304):jcon fd 28: Failing: failed: INVALID 0 (WIRE_TEMPORARY_CHANNEL_FAILURE: Maximum value exceeded)
lightningd(13304): peer 03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1: Failing HTLC 0 due to peer death
lightningd(13304):jcon fd 28: Closing (No such file or directory)
lightning_gossipd(13312): TRACE: Failed connected out for 02e7ac300471ac96db1ecc52f3f4e2bf26a3cd3cfb1e8ba68f2b6fadc4ab37440e, will try again
lightning_gossipd(13312): TRACE: Failed connected out for 03a99d5dc70b67b49d0d10f84cf641fd4eb30ff08e25fc11cd057bb9b953cb2595, will try again
I have trouble following the thread of that transaction in the log due to the noise around it. Can we get some way to mark each payment somehow so grep could catch it? I'm looking at "due to peer death" and thinking is process 13304, 13312 and 3074 all related to that one transaction? No timestamps, either, but that's secondary.
So, does it attempt to find another route? The payment shouldn't fail just because one route failed.
Furthermore, what's the fastest way to find out what the maximum value is, and what the current channel balance and funding is?
I found it in the sqlite3 database:
First, sqlite3 doesn't find it using select command:
sqlite> select state,funder,hex(funding_tx_id),funding_tx_outnum,funding_satoshi,short_channel_id from channels where short_channel_id='506216:1409:0';
sqlite>
But, if I take out the where clause and list all of them and grep the output, I find it:
egrep "506216:1409:0" /tmp/chans
3|1|0152FA7982FFB1A7F70934FDB379AEFF8BC9204A78EACD851C04EA04E16A6BE9|0|51000|506216:1409:0
That transaction is not known:
bitcoin-cli getrawtransaction '0152fa7982ffb1a7f70934fdb379aeff8bc9204a78eacd851c04ea04e16a6be9'
error code: -5
error message:
No such mempool or blockchain transaction. Use gettransaction for wallet transactions.
According to documentation, getrawtransaction requires -txindex, which I have turned on and working. blockchain . info: transaction not found.
It's in the log inside some other stuff:
egrep -i 0152fa7982ffb1a7f70934fdb379aeff8bc9204a78eacd851c04ea04e16a6be9 ~/log/*
/home/ulmo/log/log:lightning_openingd(11122): TRACE: Read decrypt 0022657e22351f2af198b0d641cd73cb55c9340b80c15673b9709e17e0a88cecde2d**0152fa7982ffb1a7f70934fdb379aeff8bc9204a78eacd851c04ea04e16a6be9**0000d88f936bced9570a1b82b13b89240f86741a2f4d9c79de419f161bdfba20d8981c6273d262fa57833415048e9470413b5d19f9b8476aa684230f3ebc27139738
/home/ulmo/log/log:lightning_channeld(11128): TRACE: Read decrypt 00240152fa7982ffb1a7f70934fdb379aeff8bc9204a78eacd851c04ea04e16a6be902f802b6304e82af6e0f3a42e21bbf61ee385c1970475e7e21915e4954a947762b
/home/ulmo/log/log:lightning_channeld(11128): TRACE: Read decrypt 01030152fa7982ffb1a7f70934fdb379aeff8bc9204a78eacd851c04ea04e16a6be907b9680005810000230ed7ecf9393952281959ec8030f1db679876c45676df8a6ce608ba8468f2bf2a3768e2bda8156fd4beb32325fd7f36e15c892fe59631f1d19a28b12caf8d275c4830388a1f4b1358feb2b7c83fcd577adb84e1cb5baa857dd1032f41d01a5448961b9b4be63bb2da6414dd1690180ea05e306277079a0f7d6497d58a268fd6
/home/ulmo/log/log:lightning_channeld(12702): TRACE: Read decrypt 00880152fa7982ffb1a7f70934fdb379aeff8bc9204a78eacd851c04ea04e16a6be90000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000319b5a3b8c8d7e2a2039bee682b4efb105dfbe1ec0e0d83964c256132bd0dc3fe
/home/ulmo/log/log:lightning_channeld(12702): TRACE: Read decrypt 00240152fa7982ffb1a7f70934fdb379aeff8bc9204a78eacd851c04ea04e16a6be902f802b6304e82af6e0f3a42e21bbf61ee385c1970475e7e21915e4954a947762b
/home/ulmo/log/log:lightning_channeld(3074): TRACE: Read decrypt 00880152fa7982ffb1a7f70934fdb379aeff8bc9204a78eacd851c04ea04e16a6be90000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000319b5a3b8c8d7e2a2039bee682b4efb105dfbe1ec0e0d83964c256132bd0dc3fe
/home/ulmo/log/log:lightning_channeld(3074): TRACE: Read decrypt 00240152fa7982ffb1a7f70934fdb379aeff8bc9204a78eacd851c04ea04e16a6be902f802b6304e82af6e0f3a42e21bbf61ee385c1970475e7e21915e4954a947762b
I figured out how to fool sqlite3 into working:
sqlite> select state,funder,hex(funding_tx_id),funding_tx_outnum,funding_satoshi,short_channel_id from channels where hex(short_channel_id)=hex('506216:1409:0');
3|1|0152FA7982FFB1A7F70934FDB379AEFF8BC9204A78EACD851C04EA04E16A6BE9|0|51000|506216:1409:0
So, that means the funding_tx_id is right there in front of me in Lightning's database, and yet, it doesn't exist. Let's look up state and funder: What's funder? Is that peer #?
sqlite> select id,hex(node_id),address from peers where id=1;
1|031C527494F295220D26A1814C18E7DB3F4F03203CE1A53A0B74B52ED570A3972A|24.4.228.213:9735
sqlite>
No, that doesn't make sense; 972a is my other node, and that's not who funded this channel. "Funder" is something else. In htlc_state.h there is "RCVD_ADD_ACK_COMMIT" in position 3 (0 starting point). I wonder where the sqlite database entries are done in order to know which variable "state" refers to. I found CREATE TABLE channels in wallet/db.c. The foo to handle that data must not be greppable since I couldn't find it.
Ok, I have to go to bed soon.
I don't know if there's a better way to do this, but reading the API on the messages sent here, you can get the funding tx id from the short name of the channel name, because the 3 numbers separated by the colon are blockHeight:transactionIndex:outputIndex.
So you can manually inspect that with bitcoin-cli e.g for the channel you're looking at 506216:1409:0, you have:
$ bitcoin-cli getblockhash 506216
000000000000000000581a267de7f871a7816aa22b8b045477ab0948865b3f25
$ bitcoin-cli getblock 000000000000000000581a267de7f871a7816aa22b8b045477ab0948865b3f25 # This will give you a json string describing the block
If you read the output of the last command into, say, a json object in python called block, you can get the transactionId doing
>>> print(block["tx"][1409])
e96b6ae104ea041c85cdea784a20c98bffae79b3fd3409f7a7b1ff8279fa5201
And then check the 0th output index of the transaction (or look it up on a browser):
$ bitcoin-cli gettxout e96b6ae104ea041c85cdea784a20c98bffae79b3fd3409f7a7b1ff8279fa5201 0
{
"bestblock": "0000000000000000000e18f64662a8f94f69033bcac8e0dc95282eaf2d701eb4",
"confirmations": 98,
"value": 0.00051000,
"scriptPubKey": {
"asm": "0 c404ff79399e043ec13f7242cd80054820efa690e1983b89f1a55e751044c51c",
"hex": "0020c404ff79399e043ec13f7242cd80054820efa690e1983b89f1a55e751044c51c",
"type": "witness_v0_scripthash"
},
"coinbase": false
}
There's a probably a simpler way to do all this, but that's what I figured so far. So you can definitely find the capacity of each channel, but I don't think you can know (and you probably shouldn't know as it's not broadcasted) how the funding tx funds are split between the two endpoints.
Your listpeers shows a channel still opening? So it'll have to use a different channel, and looks like the middleman there is telling you it doesn't have capacity.
pay does not retry yet, that is #638, which is still in progress, sorry...
Note that pay does not even report routing failures back to the routing algorithm (again, that is part of #638) so manually retrying will not work either... sorry.
Thanks. I haven't had time to just sit down and stare at things, so I didn't analyze the channel naming system.
Looks like it got funded with just over 50k, so I retried a payment from last night, then 50,000, then 100 or so:
) lightning-cli pay lnbc1m1pdxcqsdpp5g4aanza2a205herm09tzkw4sul6y0zc0w9drmxrt3g9l2kpe070qdq6f38zq3tcwpkx7un9wgs9g6tswvcqzyst8wscs3k05d996quytvezsf2860hdlhw87p234anjhjht8q8zwgywtpn24zep6tcxx68duqky59glp5vu60t3685h7jcdkjmkwklxkcqxf0dd0
"failed: INVALID 0 (WIRE_TEMPORARY_CHANNEL_FAILURE: Maximum value exceeded)"
) lightning-cli pay lnbc500u1pdxcasrpp5ucgchkre94jpkn4vetxrjxxsgfpnamvyrfhca45zkyhnn0ajd3nsdq6f38zq3tcwpkx7un9wgs9g6tswvcqzysye5ysmjv37avjpv2ekwwku2gpzwwtvxe9gtr6ly7nr6cnu626hkq42sjak6nf8shczv9hy2s8mcknctjkjwfuhn7yfly5jgam40mwjcqfk44y8
"failed: INVALID 0 (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded)"
) lightning-cli pay lnbc1u1pdxca38pp57srfjwm309xs42ka9nr0gq68q3mptah9em4n92v6pj8t280rdjasdq6f38zq3tcwpkx7un9wgs9g6tswvcqzysnehufg32ajut2nqk8acgu0jpa3kjkrxyr4fztj7au96p2ztsyzg9wax0exyc4m60usz6wk0ezkzut9ap032rjz3pqt5lcd0g4unz89sqqkeqle
"failed: INVALID 0 (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded)"
The last one in the log is starting to look familiar:
lightningd(13304): Connected json input
lightning_gossipd(13312): TRACE: req: type WIRE_GOSSIP_GETROUTE_REQUEST len 78
lightning_gossipd(13312): TRACE: Trying to find a route from 03a3c1fc0a489e2872a2afe8fa16c2714b5fe80d52d8c1ce8ec
c4361a77391da82 to 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8 for 100000 msatoshi
lightning_gossipd(13312): TRACE: find_route: via 03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596
fe1
lightning_gossipd(13312): TRACE: 035f1498c929d4cefba4701ae36a554691f526ff60b1766badd5a49b3c8b68e1d8 (0+1=0)
lightning_gossipd(13312): TRACE: =100000(+0)
lightning_gossipd(13312): REPLY WIRE_GOSSIP_GETROUTE_REPLY with 0 fds
lightningd(13304): Sending 100000 over 2 hops to deliver 100000
lightning_channeld(20734): TRACE: balance = 18446744073709502616, fee is 0, reserve is 510000
lightning_channeld(20734): TRACE: Adding HTLC 0 msat=100000 cltv=506649 gave 5
lightning_channeld(20734): REPLY WIRE_CHANNEL_OFFER_HTLC_REPLY with 0 fds
lightningd(13304):jcon fd 25: Failing: failed: INVALID 0 (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded)
lightningd(13304): peer 03cbf298b068300be33f06c947b9d3f00a0f0e8089da3233f5db37e81d3a596fe1: Failing HTLC 0 due to peer death
lightningd(13304):jcon fd 25: Closing (No such file or directory)
I see what I was looking for there. "reserve is 51000" matches what you found.
I don't have time to retrace this today; I'm about to go back to work.
Thanks to ZmnSCPxj (sounds like a dev alias), I'm wondering how they'll build the database of failed routes. Now is when I realize some portions of the software are pre-alpha. The fundamentals are building.
Thanks to ZmnSCPxj (sounds like a dev alias), I'm wondering how they'll build the database of failed routes. Now is when I realize some portions of the software are pre-alpha. The fundamentals are building.
You are welcome. As it happens, I am merely a randomly-generated Internet entity, and am most definitely not an AI who is incapable of passing the Turing test. The current plan is simply to deactivate the channels (i.e. tell the routing system to not consider the channel during routing) until the channel is activated again by a channel update gossip message, but arguably for temporary failures that is a bit of an overkill, as if the failure is temporary we should consider to retry it again even without a channel update (indeed, we can consider virtually increasing channel traversal cost (currently only the product of cltv-delta and fees) by a large amount in case of temporary channel failure, then decay this over time so that we become more likely to route via the channel again in the hope that the temporary failure was truely temporary). But that will be another update, at least we want to be able to let routing actually consider alternate routes rather than the current situation where it very quixotically keeps returning the same route over and over.
The current plan is simply to deactivate the channels (i.e. tell the routing system to not consider the channel during routing) until the channel is activated again by a channel update gossip message, but arguably for temporary failures that is a bit of an overkill,
That was my first reaction. Also, binary state can similarly work against you if it’s activated again and still not working great (in addition to your overkill concern).
as if the failure is temporary we should consider to retry it again even without a channel update (indeed, we can consider virtually increasing channel traversal cost (currently only the product of cltv-delta and fees) by a large amount in case of temporary channel failure, then decay this over time so that we become more likely to route via the channel again in the hope that the temporary failure was truely temporary).
Understanding and knowing how those equations are implemented are important to making these types of systems work well. But that sounds like an improvement.
I think also trying out enough different routes just to know how they are doing to keep a minimum amount of “best quality level we can get” top contenders would help keep options available whenever a current best option becomes unbest. So, a little random spice to know more.
I almost started to suggest NOOP transactions to test that, but I decided to stop there because I don’t want to introduce the horrible cascading NOOP load failures without knowing more, and LN might not even support it.
But that will be another update, at least we want to be able to let routing actually consider alternate routes rather than the current situation where it very quixotically keeps returning the same route over and over.
Sounds fine!
Shall we close this one as a duplicate of #550?
Duplicate of #638 & #550.
Most helpful comment
I don't know if there's a better way to do this, but reading the API on the messages sent here, you can get the funding tx id from the short name of the channel name, because the 3 numbers separated by the colon are blockHeight:transactionIndex:outputIndex.
So you can manually inspect that with
bitcoin-clie.g for the channel you're looking at 506216:1409:0, you have:If you read the output of the last command into, say, a json object in python called
block, you can get the transactionId doingAnd then check the 0th output index of the transaction (or look it up on a browser):
There's a probably a simpler way to do all this, but that's what I figured so far. So you can definitely find the capacity of each channel, but I don't think you can know (and you probably shouldn't know as it's not broadcasted) how the funding tx funds are split between the two endpoints.