Lnd: FeeInsufficient error paying invoice to routed (not directly connected) c-lightning node

[aantonop]

I'm trying to pay an invoice generated by the Blockstream c-lightning store (buying stickers!)
My lnd node is configured, funded, well connected and has 3 channels open to other well funded and connected nodes.

If I use queryroutes, I see 5-6 routes with 0-2 SAT total fees and plenty of capacity. The invoice is for 81009 SAT payment. When I try to "payinvoice", I get this error:

"payment_error": "FeeInsufficient(fee=81009343 mSAT...

That fee is actually the entire invoice amount, not the fee. The payment amount is 81009, and lnd seems to add 0.343 SAT to the end of it.

[aantonop]

I have now replicated this error with the destination node being lnd too. Invoice amount is 10 SAT.

"payment_error": "FeeInsufficient(fee=10800 mSAT...

So lnd->c-lightning and lnd->lnd. In both cases not directly connected. In both cases queryroutes shows between 5 and 12 routes, all with low fees, plenty of capacity. In both cases, the "fee" shown is actually the invoice amount plus some extra sub-satoshi amount.

[bretton]

I'm having a similar problem, routed payments to other nodes (c-lightning and not) gives the error when there is no direct channel, but succeeds once a direct channel is established.

When trying to pay an invoice generated at http://minimax.bight.nl/ I get

FeeInsufficient(fee=101001 mSAT..

lncli queryroutes reveals many paths of 3 or 4 hops, and at least two of those paths have zero fees set.

[aantonop]

Additional information:

$ lnd --version
lnd version 0.4.0-alpha

running on mainnet

[Roasbeef]

by the Blockstream c-lightning store

I think the store is currently in the process of being updated, as it's running a pretty old version of clightning and their current master has had many bug fixes and improvements since then. Users have reported issues even establishing connections, so I'd wait a few days before interacting with it to give them time to gracefully update their service.

That fee is actually the entire invoice amount, not the fee. The payment amount is 81009, and lnd seems to add 0.343 SAT to the end of it.

The amount in the returned error is actually the amount of the HTLC, not the fee itself.

I have now replicated this error with the destination node being lnd too. Invoice amount is 10 SAT.

The issue would point to some intermediate node that either isn't upholding its advertised fees, or keeps changing them mid payment retry.

This PR was recently merged which fixes some issues that users have encountered due to old un-upgraded nodes on the network. Before if we encountered an insufficient fee error, we would halt routing attempts rather than continuing with the fee fee update applied. If you update to the latest master (there'll be a patch release soon fixing this along with other bugs reported, yay testers!), are you able to reproduce this?

[aantonop]

Now at commit @807b84e

v0.4.0-beta

Now I'm getting a different error "ExpiryTooSoon", but I'm not getting the FeeInsufficient error any longer...

[Roasbeef]

ExpiryTooSoon would point to either the destination node, or an intermediate node not being synced to the chain. I'll dig into this as it may be another gap in our handling or the set of possible routing errors that can be returned.

[aantonop]

Payment to the lnd node (fresh invoice) is giving error:
IncorrectCltvExpiry

[Roasbeef]

Is your node, and the node you're trying to pay synced to the chain? This error can occur if a node is either setting the wrong time locks, or from the PoV of another node, the timelocks are off since it doesn't know what the current blockheight is.

These two time lock related errors are a case where, atm, we'll stop payment attempts as we assume the issue is with our local node not setting the proper timelock values. Instead, we should examine the error returned, and our current height in order to decide whether we should continue (to route around the faulty node) or halt as our local state is faulty.

[aantonop]

Both nodes (source, destination) are fully synched. Error detail:

Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]: 2018-03-22 20:12:37.037 [ERR] CRTR: Attempt to send payment 39f6a9fb190c97b2c4b38fc9ac75270ec5cda06655c87430758b9f8e768630a2 failed: IncorrectCltvExpiry(expiry=514879, update=(lnwire.ChannelUpdate) {
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  Signature: (lnwire.Sig) (len=64 cap=64) {
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:   00000000  01 02 bd d9 f7 4d 3e 85  02 6f c6 1f f3 88 a0 02  |.....M>..o......|
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:   00000010  fa 94 77 c1 22 33 af 93  9d 76 d1 ca 4c e4 61 c5  |..w."3...v..L.a.|
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:   00000020  89 3e 40 58 eb 0c 24 90  48 52 e4 94 ad 55 b8 92  |.>@X..$.HR...U..|
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:   00000030  28 7d 62 88 d1 6c f0 0b  7a 5c 39 d5 13 9d f1 66  |(}b..l..z\9....f|
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  },
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  ChainHash: (chainhash.Hash) (len=32 cap=32) 00000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f33fa,
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  ShortChannelID: (lnwire.ShortChannelID) 7:14125056:701,
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  Timestamp: (uint32) 23219,
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  Flags: (lnwire.ChanUpdateFlag) 56473,
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  TimeLockDelta: (uint16) 0,
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  HtlcMinimumMsat: (lnwire.MilliSatoshi) 3940649673949184 mSAT,
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  BaseFee: (uint32) 65536000,
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]:  FeeRate: (uint32) 65536000
Mar 22 20:12:37 ip-172-30-0-66 lnd[4942]: }

[Roasbeef]

Are you trying to send a "raw" payment, so manually specifying details on the command line, or is this an invoice you've received?

[Roasbeef]

TimeLockDelta: (uint16) 0,

That's an invalid value even. The timelock delta can't be zero, otherwise a race condition if a party goes on-chain to claim an HTLC is introduced. We need a value > 0 in order to give node C time to settle is incoming HTLC after it settles its outgoing HTLC.

Seems the path finding logic should be modified to skip nodes that have a timelock delta of zero.

[Roasbeef]

In general, that returned channel policy looks off:

ShortChannelID: (lnwire.ShortChannelID) 7:14125056:701,

That would mean the channel is in block 7, tx index 14125056, which is impossible.

Flags: (lnwire.ChanUpdateFlag) 56473,

There are no currently defined flags that go that high. Max value is 3 atm.

My hunch is that it's trying to route through a node that has memory corruption due to a use after free, or pointer over run, or something of the like.

Our logic should handle this though, and just route around the faulty node, should have a PR up later today.

[aantonop]

Here's the other error (paying for Blockstream stickers):

Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]: 2018-03-22 20:42:25.633 [ERR] CRTR: Attempt to send payment 3b667f267215b8a6fe6f6529bfc60a8209182bb8be250f59cde3d355daf450e8 failed: ExpiryTooSoon(update=(lnwire.ChannelUpdate) {
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  Signature: (lnwire.Sig) (len=64 cap=64) {
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:   00000000  e6 49 f4 e6 04 f3 eb 79  b5 f7 cf 92 ed b5 91 03  |.I.....y........|
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:   00000010  9c c9 8a 5a 31 60 ec 7c  a0 ff da 98 4b 6b 51 e1  |...Z1`.|....KkQ.|
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:   00000020  2d 5e 25 34 df 1d 95 08  48 7e 0a 97 2d 55 7f f1  |-^%4....H~..-U..|
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:   00000030  1f c8 de d5 43 2c 09 57  58 8b e9 da 1e 73 eb 3b  |....C,.WX....s.;|
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  },
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  ChainHash: (chainhash.Hash) (len=32 cap=32) 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f,
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  ShortChannelID: (lnwire.ShortChannelID) 514583:647:0,
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  Timestamp: (uint32) 1521741941,
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  Flags: (lnwire.ChanUpdateFlag) 0,
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  TimeLockDelta: (uint16) 4,
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  HtlcMinimumMsat: (lnwire.MilliSatoshi) 1000 mSAT,
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  BaseFee: (uint32) 0,
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]:  FeeRate: (uint32) 1
Mar 22 20:42:25 ip-172-30-0-66 lnd[4942]: }

[Roasbeef]

I think the issue with that one, is a misconfigured node. The operator looks like they've set their timelock delta value _too low_. Most nodes have a grace period, and the timelock delta needs to be greater than this grace period. For example: if my grace period is 10, then my timelock delta can't be 5, as otherwise the time gap is insufficient.

[aantonop]

My time_lock_delta is 4. If it is an intermediary node, we should route
around it, I guess.

I am ready to test any PRs you suggest.

[Roasbeef]

Ah, well that returned value would be from the _source_ of the error. I'll see if I can reproduce myself. Agreed we should route around it, but I'm going to investigate to see if there's a deeper issue. Either way will have a PR up to test soon.

[halfik]

Was looking if FinalIncorrectCltvExpiry error was reported and have different question regarding errors.
Why there is always "payment_route": null when an error occurs?
Shoudnt this provide information about route used for payment?

PS. I got FinalIncorrectCltvExpiry with payment to node im connected with, but my node was restarded a moment before and i think it was still syncing. Payment went through with next try.

[bretton]

PR seems to have solved the LND to LND payment errors I was having (where no direct channel existed) - now routes properly.

[Roasbeef]

Re-investigating this in light of some new evidence...

[Roasbeef]

Aight, I've located the _actual_ root cause. Have some newly added failing tests locally. Should have a PR with a proper remedy tomorrow-ish. Thanks for testing!

[engenegr]

Have the same issue today with 2 different nodes in mainnet:

-- CommunityJar
-- Blockstream store

I have the same output in both situations:

2018-06-06 16:02:55.852 [ERR] CRTR: Attempt to send payment 24f2ffd1c8819e9c0e1f1b64e84d0d45570547890d8ba7a0512df58586bb9bfc failed: FeeInsufficient(fee=1002012 mSAT, update=(lnwire.ChannelUpdate) {
Signature: (lnwire.Sig) (len=64 cap=64) {
00000000 5e 1b 19 ad 2a 66 51 2f 4d c9 2b 83 e7 07 46 99 |^...*fQ/M.+...F.|
00000010 dc 49 21 e1 c5 f1 c6 0a c6 3e e9 77 64 e2 7e 23 |.I!......>.wd.~#|
00000020 07 c1 5f 36 79 d2 73 ab 49 24 4a 65 cc 40 38 f2 |.._6y.s.I$Je.@8.|
00000030 3f 4b 1e 36 75 0b 67 6d c8 eb 01 f1 2d b7 d3 c0 |?K.6u.gm....-...|
},
ChainHash: (chainhash.Hash) (len=32 cap=32) 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f,
ShortChannelID: (lnwire.ShortChannelID) 526107:658:1,
Timestamp: (uint32) 1528253917,
Flags: (lnwire.ChanUpdateFlag) 1,
TimeLockDelta: (uint16) 14,
HtlcMinimumMsat: (lnwire.MilliSatoshi) 1000 mSAT,
BaseFee: (uint32) 1000,
FeeRate: (uint32) 10
}

Queryroutes executes successfully (i see routes to both end nodes for different amounts)

[Roasbeef]

Is that a terminal failure? We'll retry in light of errors like that. Also queryroutes just looks at if the channels exist or not, if all the nodes for example are offline, it'll still display results as it looks at static graph information.

[engenegr]

@Roasbeef Sorry, what do you mean "Is that a terminal failure"?

Thank you very much for clarifying about queryroutes. I still learn a lot from my own experience but should learn some posts and BOLTs. Of course, now I understand that there could be no route because some nodes go offline... especially if there are a lot of intermediate nodes on the route.

At least I got lucky with Community Jar, I've sent 1000 satoshis successfully.

[halseth]

Terminal failure means the last failure that happens before path finding stops.

[engenegr]

Ok. The definition is clear, however, I do not fully understand this.

Luckily, I could repeat error again and seems that caught it in the log successfully.
here http://paste.debian.net/1028422/ , I suppose it starts from the line 200.

Hope this will help.

[Roasbeef]

If it's just the remote node rapidly changing their fee rate, then we'll ignore them after they send two duplicate fee updates. We recently fixed an issue w.r.t the way lnd handles fee forwarding (if it's an lnd node), it's possible that they haven't yet updated to the newest version with this fixed.

[joostjager]

It seems that sometimes the proper fee to send together with the payment amount into the network is not calculated correctly. This may happen with 3+ hop routes and relatively high fee rates. It may lead to a FeeInsufficient error. Work in progress in https://github.com/lightningnetwork/lnd/pull/1382.

[nharmon9]

I am creating a send to route transaction with the route circling back to myself on simnet. When I vary the total_time_lock by 1sec the error goes from ExpiryTooSoon to ExpiryTooFar.

[nharmon9]

I still have the same error when I just create a route to another node.

[halseth]

@nharmon9 It depends on your block timestamps.