Lightning: I accidentally killed my favourite channel when sending multiple payments across custom routes

[billygarrison]

Issue and Steps to Reproduce

I've been working on scripts for circular payments and rebalancing channels, and have been having most payments fail with WIRE_TEMPORARY_CHANNEL_FAILURE. Details on that failure can be found in my other issue https://github.com/ElementsProject/lightning/issues/2362#issuecomment-466409818 (I still need help figuring out why I keep getting that error, or how to handle/avoid it).

Anyway, I wrote a script that pays msatoshi out through each of my positively unbalanced channels and back in to myself through my single most negatively unbalanced channel (this channel is the one I killed).

When I'd run this script, 0 out of 10 payments would succeed, and the error would be WIRE_TEMPORARY_CHANNEL_FAILURE. Some times, one or two of the payments would succeed, and they tended to be the higher capacity channels.

Anyway, after running this script maybe 5-10 times, the channel is now being unilaterally closed by my node. I'm wondering if anyone can tell why my node might have forced the channel closed based on this info? The log looks like this:

    {
      "type": "INFO",
      "time": "223814.202942210",
      "source": "lightningd(5589):",
      "log": "Resolving invoice '20190223 1027 x rebalance' with HTLC 66"
    },
    {
      "type": "SKIPPED",
      "num_skipped": 31
    },
    {
      "type": "UNUSUAL",
      "time": "223814.497259817",
      "source": "lightningd(5589):",
      "log": "02c12b5459cf107ee0440cae41902f1189db50fa003a077f3a6fbe6b5760218695 chan #7: Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: sent ERROR commit_sig with no changes (tx ae3529dea72bc03f565bfaafce1e8e14d2af5913f5cb167c3699b2f4a067cf70)"
    },
    {
      "type": "SKIPPED",
      "num_skipped": 1
    },
    {
      "type": "INFO",
      "time": "223814.514138712",
      "source": "lightningd(5589):",
      "log": "02c12b5459cf107ee0440cae41902f1189db50fa003a077f3a6fbe6b5760218695 chan #7: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL"
    },

...

    {
      "type": "INFO",
      "time": "223963.858298392",
      "source": "lightningd(5589):",
      "log": "0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4 chan #23: Peer transient failure in CHANNELD_NORMAL: lightning_channeld: Owning subdaemon lightning_channeld died (62208)"
    },
    {
      "type": "SKIPPED",
      "num_skipped": 3757
    },
    {
      "type": "INFO",
      "time": "223975.530351646",
      "source": "lightningd(5589):",
      "log": "03bb88ccc444534da7b5b64b4f7b15e1eccb18e102db0e400d4b9cfe93763aa26d chan #32: htlc 11 failed from 2th node with code 0x4008 (WIRE_PERMANENT_CHANNEL_FAILURE)"
    },
    {
      "type": "SKIPPED",
      "num_skipped": 73
    },
    {
      "type": "UNUSUAL",
      "time": "223984.504398311",
      "source": "lightningd(5589):",
      "log": "02c12b5459cf107ee0440cae41902f1189db50fa003a077f3a6fbe6b5760218695 chan #7: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent (tx ae3529dea72bc03f565bfaafce1e8e14d2af5913f5cb167c3699b2f4a067cf70)"
    },

getinfo output

{
  "id": "02843833374b6c42daa4b780df2066a9985cb8cf97bc4566dd63cf693d32cbf7b5",
  "alias": "BillyGarrison",
  "color": "009900",
  "num_peers": 16,
  "num_pending_channels": 0,
  "num_active_channels": 15,
  "num_inactive_channels": 1,
  "address": [
    {
      "type": "ipv4",
      "address": "47.55.123.202",
      "port": 9735
    }
  ],
  "binding": [
  ],
  "version": "v0.6.3",
  "blockheight": 564333,
  "network": "bitcoin",
  "msatoshi_fees_collected": 3825
}

[cdecker]

This appears to be an issue with lnd sending signatures without any changes in them:

Peer permanent failure in CHANNELD_NORMAL: lightning_channeld: sent ERROR commit_sig with no changes

This is not allowed by the spec, which is why we get upset with them and close the channel.

[billygarrison]

Ah okay thanks, good to know! I'll post an issue on their GH then.

[billygarrison]

On second thought I'm going to re-open this.

Do you know of a reason that this might have caused a unilateral close instead of a cooperative one?

[rustyrussell]

Yes, because they are not allowed to do that! If we reconnect they'll just do the same thing again. It's a known lnd issue that they do this under stress however, so we may have to just ignore it.

Please leave issue open, so we can tag workaround....

[rustyrussell]

Fixed in #2580. Will be in v0.7.1. Thanks!