Lnd: 0.10 LND force closing channels with older nodes

[juscamarena]

Background

Describe your issue here.
0.10 node force closing channels with: 020-05-07 18:18:11.517 [ERR] HSWC: ChannelLink(612081:1755:1): failing link: ChannelPoint(b99fc7d8823c324e940ff8997c7689110c63a475798fe7a1cb338ba1e1961552:1): unable to accept new commitment: not enough HTLC signatures. with error: invalid commitment

Your environment

• version of lnd
  I have two nodes, one on 0.09 (with wumbo enabled) and one on 0.10 (no patches).

• which operating system (uname -a on *Nix)

• version of btcd, bitcoind, or other backend
• any other relevant environment details

Steps to reproduce

Tell us how to reproduce this issue. Please provide stacktraces and links to code in question.

Not sure but I created a wumbo invoice on the 0.10 node and attempted paying it with a 0.09 wumbo enabled node. There were errors paying even with a direct channel and plenty of inbound capacity.

I cleanly restarted and the 0.10 node force closed.

The logs for 0.10 node force closing the channel are at the end of this snippet:

https://paste.ubuntu.com/p/Snww6ZdGpZ/

[Roasbeef]

attempted paying it with a 0.09 wumbo enabled node

That was also patched to allow larger payments in addition to the larger channels?

[Roasbeef]

2020-05-07 18:18:11.517 [ERR] HSWC: ChannelLink(612081:1755:1): failing link: ChannelPoint(b99fc7d8823c324e940ff8997c7689110c63a475798fe7a1cb338ba1e1961552:1): unable to accept new commitment: not enough HTLC signatures. with error: invalid commitment

[Roasbeef]

2020-05-07 18:18:11.926 [WRN] CRTR: Failed to find route for payment a3f898cb3d20f16f434223cf4ece7e944c3509c8f50d0b54384e7f36617c690a: insufficient local balance
2020-05-07 18:18:11.938 [WRN] RPCS: Unable to send payment: insufficient_balance

[juscamarena]

Yep, payments and channels wumbo enabled for the 0.09 node.

Here's the 0.09 node logs grepping b99fc7d8823c324e940ff8997c7689110c63a475798fe7a1cb338ba1e1961552

https://paste.ubuntu.com/p/xxggKT3qNt/

[joostjager]

Same error message repro'ed in https://github.com/lightningnetwork/lnd/pull/4193

[MrManPew]

Could it be that this "bug" is responsible for the latest reduction in network capacity - https://1ml.com/statistics

[yaslama]

@joostjager FYI, we saw recently that channels were closed without good reason by our node. And from the logs, it seems that this bug is the culprit:

2020-05-26 00:06:49.074 [ERR] HSWC: ChannelLink(609632:1101:0): failing link: ChannelPoint(9c0e40ff15f8d794fea8f3f801421c72a7cc976cd21d47742dc2e45b7bd3458e:0): unable to accept new commitment: not enough HTLC signatures. with error: invalid commitment

2020-06-01 10:01:31.569 [ERR] HSWC: ChannelLink(626422:2535:0): failing link: ChannelPoint(b55bf3f670449c426a2cac45ee8fabf58ca46579cdba4bcb0fe33bf6d5dee5db:0): unable to accept new commitment: not enough HTLC signatures. with error: invalid commitment

2020-06-02 17:31:20.777 [ERR] HSWC: ChannelLink(629805:756:0): failing link: ChannelPoint(09ad3b53125b71a1d04eca481f17f0822548763ea40da2f0a33c403ea7ab2592:0): unable to accept new commitment: not enough HTLC signatures. with error: invalid commitment

[Roasbeef]

So we've identified the root cause for this issue. We're testing a fix atm, which is slated to land as part of 0.11.