Lightning: How to debug "getroute -> Could not find a route"
Issue and Steps to Reproduce
I have three nodes A, B, C with two channels A->B and B->C with both the same capacity of 1000 sat.
a single hop works as expected:
on NodeA getroute B 500000 1 and on NodeB getroute C 500000 1 -> shows a result
but A->C does not work:
on NodeA getroute C 500000 1 returns { "code" : -1, "message" : "Could not find a route" }
We also noticed that asking for a too high amount does not fail es expected, eg:
on NodeA getroute B 2500000 1 returns the same result as with "500000", even the channel would not have the needed capacity -> is this a known issue?
(numbers are simplified)
getinfo output
e.g. nodeA
lightning-cli getinfo 1
{ "id" : "03d40a4ade94703907f83e0b74fc07633db6d0842fad6aae6c4acef3bfb09d5fa4", "port" : 9735, "address" :
[
{ "type" : "ipv4", "address" : "178.63.97.197", "port" : 9735 },
{ "type" : "ipv6", "address" : "2a01:4f8:121:4406::2", "port" : 9735 } ], "version" : "v0.5.2-2016-11-21-1886-gb3d4e16", "blockheight" : 508091, "network" : "bitcoin" }
lightning-cli listpeers | jq "."
{
"peers": [
{
"id": "028314f021602092779aedd4ef39f3b5809f9b6046f8bc28359b16e659ab1dcd7c",
"connected": true,
"netaddr": [
"[::ffff:80.108.14.34]:55378"
],
"channels": [
{
"state": "CHANNELD_NORMAL",
"owner": "lightning_channeld",
"short_channel_id": "508082:728:1",
"funding_txid": "edfc4bb5437e6acf97eea8d7036068368b8bb61957c2ebc84adcde8e6f2359a6",
"msatoshi_to_us": 319121000,
"msatoshi_total": 319121000,
"dust_limit_satoshis": 546,
"max_htlc_value_in_flight_msat": 18446744073709552000,
"channel_reserve_satoshis": 0,
"htlc_minimum_msat": 0,
"to_self_delay": 144,
"max_accepted_htlcs": 483
}
]
},
NodeA->NodeB (even amount exhaust channel)
lightning-cli getroute 028314f021602092779aedd4ef39f3b5809f9b6046f8bc28359b16e659ab1dcd7c 50000000000 1
{ "route" :
[
{ "id" : "028314f021602092779aedd4ef39f3b5809f9b6046f8bc28359b16e659ab1dcd7c", "channel" : "508082:728:1", "msatoshi" : 2755359744, "delay" : 9 } ] }
NodeA->NodeC (amount should be fine here, different amounts or risk-factors did not change anything)
lightning-cli getroute 0318378b5eb8d6d6f2d9bf1b527aea72567feb010899e9dcd45b7cb2129b9b8e0d 1000 .9 1
{ "code" : -1, "message" : "Could not find a route" }
DanielWeigl
All 13 comments
While looking in the map to find nodes and channels to route over, the routing algorithm in gossipd completely ignores the given amount. Once it has selected a route, only then will the routing algorithm look at the given amount, and only to fill in the amounts being passed at each hop. So currently, it is expected that getroute will ignore the given msatoshi completely when selecting channels to hop over.
Can you show the result of listchannels command at NodeA? If it lists the B->C channel it should be able to find a route from A to C, unless that route is marked inactive for some reason. If not, can you get listchannels at NodeB also?
ZmnSCPxj
on 7 Feb 2018
On NodeA
lightning-cli listchannels | grep $B | jq "."
{
"source": "03d40a4ade94703907f83e0b74fc07633db6d0842fad6aae6c4acef3bfb09d5fa4",
"destination": "028314f021602092779aedd4ef39f3b5809f9b6046f8bc28359b16e659ab1dcd7c", // A->B
"short_channel_id": "508082:728:1",
"flags": 1,
"active": true,
"public": true,
"last_update": 1518009373,
"base_fee_millisatoshi": 1000,
"fee_per_millionth": 10,
"delay": 14
}
cli/lightning-cli listchannels | grep $C | jq "."
(empty)
on NodeB
{
"source": "03d40a4ade94703907f83e0b74fc07633db6d0842fad6aae6c4acef3bfb09d5fa4",
"destination": "028314f021602092779aedd4ef39f3b5809f9b6046f8bc28359b16e659ab1dcd7c", //A->B
"short_channel_id": "508082:728:1",
"flags": 1,
"active": true,
"public": true,
"last_update": 1518000112,
"base_fee_millisatoshi": 1000,
"fee_per_millionth": 10,
"delay": 14
}
...
{
"source": "028314f021602092779aedd4ef39f3b5809f9b6046f8bc28359b16e659ab1dcd7c",
"destination": "0318378b5eb8d6d6f2d9bf1b527aea72567feb010899e9dcd45b7cb2129b9b8e0d", // B->C
"short_channel_id": "508085:918:0",
"flags": 0,
"active": true,
"public": false,
"last_update": 0,
"base_fee_millisatoshi": 1000,
"fee_per_millionth": 10,
"delay": 14
}
and on NodeC
cli/lightning-cli listchannels | grep $B | jq "."
{
"source": "028314f021602092779aedd4ef39f3b5809f9b6046f8bc28359b16e659ab1dcd7c",
"destination": "03d40a4ade94703907f83e0b74fc07633db6d0842fad6aae6c4acef3bfb09d5fa4", // A->B
"short_channel_id": "508082:728:1",
"flags": 0,
"active": true,
"public": true,
"last_update": 1518009373,
"base_fee_millisatoshi": 1000,
"fee_per_millionth": 10,
"delay": 14
}
lightning-cli listchannels | grep $C | jq "."
(empty)
So,
Node AandNode Cknow aboutA->BNode Bknows aboutA->BandB->C
Just for testing, i asked NodeC:
cli/lightning-cli getroute $A 10000 1
{ "code" : -1, "message" : "Could not find a route" }
and in the debug logs of NodeC I see:
2018-02-07T13:34:18.495Z lightning_gossipd(31278): Trying to find a route from 0318378b5eb8d6d6f2d9bf1b527aea72567feb010899e9dcd45b7cb2129b9b8e0d to 03d40a4ade94703907f83e0b74fc07633db6d0842fad6aae6c4acef3bfb09d5fa4 for 10000 msatoshi
2018-02-07T13:34:18.495Z lightning_gossipd(31278): find_route: cannot find myself (03d40a4ade94703907f83e0b74fc07633db6d0842fad6aae6c4acef3bfb09d5fa4)
The "public" : false on the B<->C channel means the channel is private between them and will not be propagated to the routemaps of any nodes other than B and C.
Thus, A cannot know this channel B<->C because it is not (yet?) public.
My understanding, default is that channels become public, after the channel becomes deep enough (6 confirms). So if you did not specifically make this channel private, then we need to find out why it is not being publicized.
However it is strange point: C does not know the B<->C channel. Can you run listpeers on C?
(the "cannot find myself" is normal if C is unaware of any channels to itself. C claims to know the A<->B channel but not the B<->C channel, so it is not knowledgeable of any channels to itself and so cannot find itself (the BOLT protocol has this strangeness, that channels are gossiped before nodes are). So we need to find listpeers to know what C thinks its state with B is.)
ZmnSCPxj
on 7 Feb 2018
Also, I want to confirm:
- A = 03d40a4ade94703907f83e0b74fc07633db6d0842fad6aae6c4acef3bfb09d5fa4
- B = 028314f021602092779aedd4ef39f3b5809f9b6046f8bc28359b16e659ab1dcd7c
- C = 0318378b5eb8d6d6f2d9bf1b527aea72567feb010899e9dcd45b7cb2129b9b8e0d
Is it correct?
ZmnSCPxj
on 7 Feb 2018
Yes, A/B/C is correct, sorry - i should have posted this mapping to make it more clear.
On nodeC
./cliC listpeers | jq "."
{
"peers": [
{
"id": "028314f021602092779aedd4ef39f3b5809f9b6046f8bc28359b16e659ab1dcd7c", //B
"connected": true,
"netaddr": [
"[::ffff:xx.xxx.xx.xxx]:43348"
],
"channels": [
{
"state": "CHANNELD_NORMAL",
"owner": "lightning_channeld",
"short_channel_id": "508085:918:0",
"funding_txid": "35507921c8ae1cfbcccc2c4f9865623704a8f0716200a8dd5df2906536396ba2",
"msatoshi_to_us": 0,
"msatoshi_total": 319121000,
"dust_limit_satoshis": 546,
"max_htlc_value_in_flight_msat": 18446744073709552000,
"channel_reserve_satoshis": 0,
"htlc_minimum_msat": 0,
"to_self_delay": 144,
"max_accepted_htlcs": 483
}
]
}
]
}
On NodeB
./cliB listpeers
{
"id": "0318378b5eb8d6d6f2d9bf1b527aea72567feb010899e9dcd45b7cb2129b9b8e0d",
"connected": true,
"netaddr": [
"xx.xxx.xx.xx:9736"
],
"channels": [
{
"state": "CHANNELD_NORMAL",
"owner": "lightning_channeld",
"short_channel_id": "508085:918:0",
"funding_txid": "35507921c8ae1cfbcccc2c4f9865623704a8f0716200a8dd5df2906536396ba2",
"msatoshi_to_us": 319121000,
"msatoshi_total": 319121000,
"dust_limit_satoshis": 546,
"max_htlc_value_in_flight_msat": 18446744073709552000,
"channel_reserve_satoshis": 0,
"htlc_minimum_msat": 0,
"to_self_delay": 144,
"max_accepted_htlcs": 483
}
]
}
./cliB listchannels 508085:918:0
--> empty
The funding tx has currently >20 confirms, but A is still unable to route from A->C
(if you feel this behavior is a one-off situation and not really a bug, feel free to close this issue - i was just curious on how to debug this problem in dept. So far i learned a lot. My NodeA and NodeC is running for >1week now and made several git pull && make updates, maybe the a database migration went wrong or something. NodeB is more or less started from an fresh state)
DanielWeigl
on 7 Feb 2018
The error you reported is generated by node A so we only need to look at listchannels for node A. If that doesn't contain the channel from B -> C then it can't find a route. If you restart node B is should also sync gossip from and to A so maybe that'll help get all the nodes to learn about the channels.
cdecker
on 7 Feb 2018
@cdecker I am curious why B<->C is claimed by B to be non-public, and why C claims not to know the channel. If B thinks the channel is private then B is right not to gossip it to A, so A will not know the channel.
ZmnSCPxj
on 7 Feb 2018
After restarting all nodes and waiting
worldview of NodeA
./cliA listchannels | grep -E "$A|$B|$C" | sed s/$A/A/ | sed s/$B/B/ | sed s/$C/C/
{ "source" : "A", "destination" : "B", "short_channel_id" : "508082:728:1", "flags" : 1, "active" : true, "public" : true, "last_update" : 1518017218, "base_fee_millisatoshi" : 1000, "fee_per_millionth" : 10, "delay" : 14 },
{ "source" : "B", "destination" : "A", "short_channel_id" : "508082:728:1", "flags" : 0, "active" : true, "public" : true, "last_update" : 1518017218, "base_fee_millisatoshi" : 1000, "fee_per_millionth" : 10, "delay" : 14 },
worldview of NodeB
./cliB listchannels | grep -E "$A|$B|$C" | sed s/$A/A/ | sed s/$B/B/ | sed s/$C/C/
(empty)
(it lists a few channels related to A/B with others, but none related to A->B or B->C)
worldview of NodeC
./cliC listchannels | grep -E "$A|$B|$C" | sed s/$A/A/ | sed s/$B/B/ | sed s/$C/C/
{ "source" : "A", "destination" : "B", "short_channel_id" : "508082:728:1", "flags" : 1, "active" : true, "public" : true, "last_update" : 1518017218, "base_fee_millisatoshi" : 1000, "fee_per_millionth" : 10, "delay" : 14 },
{ "source" : "B", "destination" : "A", "short_channel_id" : "508082:728:1", "flags" : 0, "active" : true, "public" : true, "last_update" : 1518017218, "base_fee_millisatoshi" : 1000, "fee_per_millionth" : 10, "delay" : 14 },
(filtered to lines only containing src and dst = {A/B/C})
Whats the meaning of source and destination in this context? Is it normal to see two entries for A->B and B->A for one channel?
So NodeB has a channel with NodeC in listpeers, but not in listchannels
DanielWeigl
on 7 Feb 2018
Whats the meaning of source and destination in this context? Is it normal to see two entries for A->B and B->A for one channel?
For a channel A<->B, there is a "connection" A->B, where the fees paid are decided by A (so that if you route via this channel, you pay A to do so). There is also another connection A<-B, where the fees paid are decided by B. The fee rates in either direction can be different as per the node decision. Thus, listchannels lists channels twice, once for each direction, in order to show the differing fees demanded by the source node. It is not obvious in your output as the fees are the same in both directions, but if you configured A and B differently as for fees, you would see the difference.
ZmnSCPxj
on 7 Feb 2018
Restarting the "entire network" can cause channel announcements to get lost, I just remembered. We need to implement gossip persistence and then this should no longer be an issue.
cdecker
on 7 Feb 2018
The public flag means that we haven't announced it yet to the rest of the world, not whether we plan to do so. Once funding has 6 confirmations we should announce it
cdecker
on 7 Feb 2018
closing for now, not a real bug to solve, but this issue provided me a lot of information on how to generally debug the state of lighting-c
DanielWeigl
on 18 Mar 2018
The announcements getting forgotten should be addressed once #1240 gets merged :-)
cdecker
on 18 Mar 2018
Related issues
igreshev
路
4Comments
brunoaduarte
路
5Comments
Xian001
路
3Comments
SPIRY-RO
路
4Comments
gallizoltan
路
3Comments
Most helpful comment
For a channel A<->B, there is a "connection" A->B, where the fees paid are decided by A (so that if you route via this channel, you pay A to do so). There is also another connection A<-B, where the fees paid are decided by B. The fee rates in either direction can be different as per the node decision. Thus,
listchannelslists channels twice, once for each direction, in order to show the differing fees demanded by the source node. It is not obvious in your output as the fees are the same in both directions, but if you configured A and B differently as for fees, you would see the difference.