Xud: LND: status in xud stuck

[kilrau]

If lnd is restarted (either manually or by supervisord) xud status for lndbtc is stuck at lnd-BTC is Unlocked even though lndbtc is up and running fine. xud restart fixes this.

[sangaman]

Is lnd unlocked manually or by xud after it is restarted?

[michael1011]

I also had this issue and used xucli unlock

1. First start -> no macaroons generated by LND
2. Second start -> xud status is stuck
3. Third start -> everything is working

[kilrau]

Updates? @sangaman

[kilrau]

I reproduced this yesterday on testnet with full-nodes.
Steps:

1. Start testnet full setup
2. Let bitcoind/litecoind sync
3. xucli unlock
4. xud status stuck in lnd-xTC is Unlocked

Whereas they were unlocked and running just fine, responding to all calls I tested.

SO, I think it's ok to go for the sledge hammer restart L1/L2 client-way via xud-docker if clients don't come up properly. BUT I think if the status is not being detected correctly by xud - we should fix this. Here the only solution is to restart xud, which is never nice and potentially cancelling/failing orders.

[kilrau]

Here my xud-docker testnet full setup showing LND's stuck in lnd-xTC is Unlocked within xud:

testnet > status
┌───────────┬────────────────────────────────────────────────┐
│ SERVICE   │ STATUS                                         │
├───────────┼────────────────────────────────────────────────┤
│ bitcoind  │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ litecoind │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ geth      │ Syncing 99.99% (7715612/7715737)               │
├───────────┼────────────────────────────────────────────────┤
│ lndbtc    │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ lndltc    │ Ready                                          │
├───────────┼────────────────────────────────────────────────┤
│ raiden    │ Container running                              │
├───────────┼────────────────────────────────────────────────┤
│ xud       │ Waiting for lndbtc, lndltc, raiden             │
└───────────┴────────────────────────────────────────────────┘
testnet > getinfo

General XUD Info
┌───────────────┬───────────────────────────────────────────────────────────────────────┐
│ Status        │ LND-BTC: lnd-BTC is Unlocked                                          │
│               │ LND-LTC: lnd-LTC is Unlocked                                          │
│               │ Raiden: connect ECONNREFUSED 10.0.2.6:500                             │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Alias         │ CubeAcoustic                                                          │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Node Key      │ 036c0742ac1dd74a81479ca9daf6964302def944f8a393cd7064048fa48b82e7b6    │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Address       │ 036c0742ac1dd74a81479ca9daf6964302def944f8a393cd7064048fa48b82e7b6    │
│               │ @5lfnxmuzqonpcx7uno3kvzilwirp5ms2rmrgd7lbtrsk35c7yxnmliad.onion:18885 │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Network       │ testnet                                                               │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Version       │ 1.0.0-beta.2-f75f94c-dirty                                            │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Peers         │ 6                                                                     │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Pairs         │ 4                                                                     │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Own orders    │ 0                                                                     │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Peer orders   │ 0                                                                     │
├───────────────┼───────────────────────────────────────────────────────────────────────┤
│ Pending swaps │ []                                                                    │
└───────────────┴───────────────────────────────────────────────────────────────────────┘ 


Raiden info:
┌──────────┬────────────────────────────────────┐
│ Status   │ connect ECONNREFUSED 10.0.2.6:5001 │
├──────────┼────────────────────────────────────┤
│ Version  │                                    │
├──────────┼────────────────────────────────────┤
│ Address  │                                    │
├──────────┼────────────────────────────────────┤
│ Channels │ Active: 0 | Pending: 0 | Closed: 0 │
├──────────┼────────────────────────────────────┤
│ Network  │ raiden testnet                     │
└──────────┴────────────────────────────────────┘ 


LND-BTC Info:
┌──────────┬────────────────────────────────────┐
│ Status   │ lnd-BTC is Unlocked                │
├──────────┼────────────────────────────────────┤
│ Version  │                                    │
├──────────┼────────────────────────────────────┤
│ Address  │                                    │
├──────────┼────────────────────────────────────┤
│ Alias    │                                    │
├──────────┼────────────────────────────────────┤
│ Channels │ Active: 0 | Pending: 0 | Closed: 0 │
├──────────┼────────────────────────────────────┤
│ Network  │                                    │
└──────────┴────────────────────────────────────┘ 


LND-LTC Info:
┌──────────┬────────────────────────────────────┐
│ Status   │ lnd-LTC is Unlocked                │
├──────────┼────────────────────────────────────┤
│ Version  │                                    │
├──────────┼────────────────────────────────────┤
│ Address  │                                    │
├──────────┼────────────────────────────────────┤
│ Alias    │                                    │
├──────────┼────────────────────────────────────┤
│ Channels │ Active: 0 | Pending: 0 | Closed: 0 │
├──────────┼────────────────────────────────────┤
│ Network  │                                    │
└──────────┴────────────────────────────────────┘ 

[sangaman]

I haven't had any success at reproducing this, unfortunately, starting from testnet docker and waiting for bitcoind/litecoind to sync then unlocking.

However, I stepped closely through the entire flow of unlocking lnd and setting the status to connected and I suspect a possible culprit. One of the first things we do after we make a successful GetInfo call to lnd, but before we update the client's status to Connected, is we call the logic to update the capacity for that client. It's done this way so that we know how much capacity we have for that currency before we attempt to start using it.

For lnd this involves a ChannelBalance call and a ListChannels call. If either of these two calls hang, then xud will never finish setting the status to connected. Since we're making these calls almost immediately after unlocking lnd, I wouldn't be shocked if lnd may not be fully initialized nor ready (maybe it's still trying to figure out whether local channels still exist, for instance) at this point and fails to respond. Since I haven't reproduced yet, I can't say this is what's happening for sure, but that's my suspicion.

I opened #1493 which is a quick fix that makes sure we don't wait more than a few seconds for the first attempt to update capacity immediately after lnd is unlocked. I'd be interested to see if that makes the issue go away for you.

[kilrau]

I tested this with xud master 1e20723 , this is still on. Since this prevents simnet from getting channels we have to bump this to P0 @sangaman :

simnet-1: Opening BTC channel to SuspectNaive (02d50056c51365f4c7288485074096eb4a9c005577b2edaaff4dc04763a6c49526)
simnet-1: Could not open BTC channel to SuspectNaive (02d50056c51365f4c7288485074096eb4a9c005577b2edaaff4dc04763a6c49526): rpc error: code = Unknown desc = failed to open channel with nodePubKey: 02d50056c51365f4c7288485074096eb4a9c005577b2edaaff4dc04763a6c49526, currency: BTC, amount: 500000000, message: peer not connected to swap client

[sangaman]

I tested this with xud master 1e20723 , this is still on. Since this prevents simnet from getting channels we have to bump this to P0 @sangaman :

simnet-1: Opening BTC channel to SuspectNaive (02d50056c51365f4c7288485074096eb4a9c005577b2edaaff4dc04763a6c49526)
simnet-1: Could not open BTC channel to SuspectNaive (02d50056c51365f4c7288485074096eb4a9c005577b2edaaff4dc04763a6c49526): rpc error: code = Unknown desc = failed to open channel with nodePubKey: 02d50056c51365f4c7288485074096eb4a9c005577b2edaaff4dc04763a6c49526, currency: BTC, amount: 500000000, message: peer not connected to swap client

Have you confirmed that the peer here is running a version of xud with the patch related to this issue? Are there any other issues on that node? It'd be interesting to see the logs from SuspectNative if you have those.

[sangaman]

@kilrau Should we close this issue, or is this still something you're experiencing?

[kilrau]

No, this was solved by the restarts we implemented in https://github.com/ExchangeUnion/xud-docker/pull/429. I already put the closing statement there and it will automatically close this issue here once it gets merged into master.

No need to do anything :)