Lightning: peer connected but channel list is empty and a getroute can't find a route to the peer

[kilimanjarolows]

Issue and Steps to Reproduce

I connected, made a channel and created a lightning transaction with a peer a few days ago.

Today I can't find neither this channel or a route to the peer.

It seems that the peer is connected. 'listpeer' results in:

{ "peers" : 
    [ 
        { "id" : "<peerid>", "connected" : true, "netaddr" : 
            [ "xxx.xxx.xxx.xxx:9735" ], "channels" : 
            [ 
                { "state" : "CHANNELD_NORMAL", "owner" : "lightning_channeld", "short_channel_id" : "<shortchannelid>", "channel_id" : "<channelid>", "funding_txid" : "<fundingtxid>", "msatoshi_to_us" : 19900000, "msatoshi_total" : 20000000, "dust_limit_satoshis" : 546, "max_htlc_value_in_flight_msat" : 18446744073709551615, "channel_reserve_satoshis" : 0, "htlc_minimum_msat" : 0, "their_to_self_delay" : 6, "our_to_self_delay" : 6, "to_self_delay" : 6, "max_accepted_htlcs" : 483, "status" : 
                    [ "CHANNELD_NORMAL:Reconnected, and reestablished.", "CHANNELD_NORMAL:Funding transaction locked. Waiting for their announcement signatures." ] } ] } ] }

But 'listchannels' is nevertheless empty:

{ "channels" : 
    [  ] }

I also can't find a route to the peer, which is strange because supposedly I'm connected to it. 'getroute 10000 1.0':

{ "code" : -1, "message" : "Could not find a route" }

getinfo output

{ "id" : "", "port" : 9735, "address" :
[ ], "version" : "v0.5.2-2016-11-21-2369-gd5d3186", "blockheight" : 1289297, "network" : "testnet" }

[cdecker]

Am I correct in assuming that this network consists solely of the two nodes connected to each other and that both have been restarted after the creation? The issue is that we didn't store gossip messages across restarts and relied on our peers to tell us about our own channels as well.

This was recently fixed in #1240. As for these channels, they will not get announced anymore, I'm sorry.

[kilimanjarolows]

Almost correct. It's a 3-nodes network, with 2 nodes connected to one main node. The one main node runs on a server and was not restarted as long as I can remember.

Anyway, we'll try to scratch everything, recompile the new version and see if it problem persists. If it doesn't, I'll close the issue.

[kilimanjarolows]

I closed the issue too early it seems. The problem repeated again today.

'list peers' shows"

{
  "peers": [
    {
      "id": "<peerid>",
      "connected": true,
      "netaddr": ["<peer address>"],
      "channels": [
        {
          "state": "CHANNELD_NORMAL",
          "owner": "lightning_channeld",
          "short_channel_id": "<short channel id>",
          "channel_id": "<channel id>",
          "funding_txid": "3d99ad5669ac43d3b9fee8ec053289fe0bb06e66afcbbb5e2a83f92aedee33f5",
          "msatoshi_to_us": 5830702,
          "msatoshi_total": 40000000,
          "dust_limit_satoshis": 546,
          "max_htlc_value_in_flight_msat": 18446744073709552000,
          "channel_reserve_satoshis": 0,
          "htlc_minimum_msat": 0,
          "their_to_self_delay": 6,
          "our_to_self_delay": 6,
          "to_self_delay": 6,
          "max_accepted_htlcs": 483,
          "status": [
            "CHANNELD_NORMAL:Reconnected, and reestablished.",
            "CHANNELD_NORMAL:Funding transaction locked. Waiting for their announcement signatures."
          ],
          "in_payments_offered": 0,
          "in_msatoshi_offered": 0,
          "in_payments_fulfilled": 0,
          "in_msatoshi_fulfilled": 0,
          "out_payments_offered": 0,
          "out_msatoshi_offered": 0,
          "out_payments_fulfilled": 0,
          "out_msatoshi_fulfilled": 0
        }
      ]
    }
  ]
}

But 'listchannels' is nevertheless empty, and I can't find a route to the peer.

'getinfo' output:

{
  "id": "03568304bb3a30dfc59ee4fac27bc5e36bb6bccfe9db065fe142f85e08bc298457",
  "port": 9735,
  "address": [ ],
  "version": "v0.5.2-2016-11-21-2431-ge25297d",
  "blockheight": 1289690,
  "network": "testnet"
}

It is a network of two nodes, where one of the nodes was restarted.

[kilimanjarolows]

update

After a while listchannel stopped being empty. Now it has 2421 channels. However, the one with the peer is not there.

[cdecker]

Who is the counterparty of the channel? It seems that there was a restart in-between the funding being created and it reaching the announcement depth: Funding transaction locked. Waiting for their announcement signatures.

Are both sides of the channel c-lightning daemons?

[kilimanjarolows]

I run both end of the channels, both running c-lightning daemons. One from a remote server, and one from a local computer.

I'm not sure what do you mean by restart, so I'll just describe the actions.

• I connected to a lightning node in my server.
• I funded a channel with the node
• When the channel appeared in the 'listchannels' list, I made two successful lightning payments with the channel.
• I stopped lightnind locally (but kept it running on the server)
• I relaunched lightnind locally

After the last action I arrived to the situation described.

It may be that within those 12 hours the c-lightning daemon on the server was restarted (since it drops sometimes, I got a script to automatically restart it). If it makes a difference, I could try to check it out later.

[cdecker]

Hm, strange, so the server was not restarted? It should have remembered the channel announcement and channel updates and told the client about it. And since PR #1240 the network view (channels + nodes) should be remembered across restarts as well.

Was there some point in which both local and server were restarted without having a connection in-between?

[kilimanjarolows]

It will take me a while to check it (some people are on a holiday)

But I don't understand the logic. listpeers shows that the client is aware of the channel. Why does it need an additional telling by the server?

[kilimanjarolows]

I checked (was easier than I thought). The server was not restarted since the funding of the channel.

[ZmnSCPxj]

But I don't understand the logic. listpeers shows that the client is aware of the channel. Why does it need an additional telling by the server?

Like all good software, lightningd is schizophrenic and one sub-component does not know what the other sub-component is doing.

listpeers is handled by the primary gossipd subdaemon main component. listchannels is handled by the gossipd subdaemon routing component. Normally the main gossipd gets updated of channels first, but only informs its routing component when a channel announcement is made, which happens after funding lockin. (the reason for this is to unify all code that informs the routing component of channels: all new channels being made (whether those made locally or those we hear about from node gossip) are informed to the routing component via channel announcement)

In fact the routing component knows nothing about local state. If you have a local channel that has "msatoshi_to_us" : 0 and ask for a getroute the routing component will happily use that channel in a discovered route (even if such a route is doomed as we have no money in that channel to use for sending!); actually paying will trigger informing the routing component of the inappropriateness and will temporarily suppress the routing component using that channel.

[kilimanjarolows]

Thank you for the explanation.

Normally the main gossipd gets updated of channels first, but only informs its routing component when a channel announcement is made, which happens after funding lockin.

If this is the case, then how did it happen that I have the state CHANNELD_NORMAL:Funding transaction locked. Waiting for their announcement signatures, but the routing component is not aware of this channel?

Moreover, I did made a payment using this channel before. Doesn't it imply that the routing component was aware of the channel at some point, but then "forgot" it? What causes this amnesia?

actually paying will trigger informing the routing component of the inappropriateness and will temporarily suppress the routing component using that channel.

In this case, is it possible to know beforehand whether I can pay in a lightning store? If not, we should expect many bitter complains from future users:-)

[ZmnSCPxj]

If this is the case, then how did it happen that I have the state CHANNELD_NORMAL:Funding transaction locked. Waiting for their announcement signatures, but the routing component is not aware of this channel?

It is the announcement signatures which are needed to make the channel announcement. If the announcement signatures are not there (cough Waiting for their announcement signatures), there is no way to make the channel announcement and no way to inform the routing component.

(I think; if memory serves me we have a special path for informing the routing component for local channels that have locked in, hmmm)

Moreover, I did made a payment using this channel before. Doesn't it imply that the routing component was aware of the channel at some point, but then "forgot" it? What causes this amnesia?

Prior to the addition of gossip_store (completed as of 439dc0f9916 in master branch) the routing component started up tabula rasa. After the addition of gossip_store previous channel announcements are now persisted to the gossip_store file (@cdecker knows the details). If you had completed the channel announcement, then shut down the daemon, then updated to after that commit, then your routing component would have still started up tabula rasa (as there was no previous gossip_store to let it remember the previous channel announcement).

In this case, is it possible to know beforehand whether I can pay in a lightning store? If not, we should expect many bitter complains from future users:-)

Unless you make a direct channel to the store, no. Your node is never privy to the state of any channel (such as whether it is capable of sending your requested amount in a particular direction) as this is sensitive financial information that each node will not want to have broadcast. The routing component is only slightly more in extremis than your own node, as it is unaware of financial information even for its own node --- your own node is unaware of financial information for all other nodes, the routing component is unaware of financial information for all nodes. This is why this is not considered high priority --- your node will always be routing blindly as it cannot know the financial state of each channel it could try to pass through, once you go beyond a single hop.

In any case the builtin pay command automatically retries failed payments, so in practice you have a good chance of successfully paying.

[kilimanjarolows]

If you had completed the channel announcement, then shut down the daemon, then updated to after that commit

I didn't do that:-) Everything happened after the commit.

Thanks for the explanations. It clarified many things. I'm closing the topic for now. I'll reopen it in the future in case I'd succeed in recreating the bug.