Lnd: [Feature Request]: Wait for bitcoin-cli to warmup before crashing

[NicolasDorier]

If lnd start before bitcoind is fully started (like for example if bitcoind is checking blocks), lnd will crash.

Can lnd wait automatically in a similar way to bitcoin-cli -rpcwait getblockchaininfo?

[Roasbeef]

Well bitcoind should be started before lnd itself. What do you mean by "crash" here? We have an exponential back off for the RPC connection, but it may be the case now that our DNS query fails early?

[NicolasDorier]

I did not experienced it yet, but this will probably happen when I run LND on mainnet.

I carefully looked your code (lnd and rpcClient), and could not find any place where you handle RPC error code -28 (RPC_IN_WARMUP).

LND fails to start when I get wrong RPC credentials (which is expected), so I am assuming (wrongly?) that it also would if calls to bitcoin rpc returns RPC_IN_WARMUP error.

[NicolasDorier]

As I was expecting: here the error

btcpayserver_lnd_litecoin | LND_CHAIN=ltc
btcpayserver_lnd_litecoin | LND_ENVIRONMENT=mainnet
btcpayserver_lnd_litecoin | Added litecoin.active and litecoin.mainnet to config file /data/lnd.conf
btcpayserver_lnd_litecoin | Attempting automatic RPC configuration to litecoind
btcpayserver_lnd_litecoin | Automatically obtained litecoind's RPC credentials
btcpayserver_lnd_litecoin | 2018-07-11 08:52:18.623 [INF] LTND: Version 0.4.2-beta commit=f349707d213129672e199149f77956d250f583ba
btcpayserver_lnd_litecoin | 2018-07-11 08:52:18.636 [INF] LTND: Active chain: Litecoin (network=mainnet)
btcpayserver_lnd_litecoin | 2018-07-11 08:52:18.684 [INF] CHDB: Checking for schema update: latest_version=1, db_version=1
btcpayserver_lnd_litecoin | 2018-07-11 08:52:18.684 [INF] RPCS: Generating TLS certificates...
btcpayserver_lnd_litecoin | 2018-07-11 08:52:18.718 [INF] RPCS: Done generating TLS certificates
btcpayserver_lnd_litecoin | 2018-07-11 08:52:18.887 [INF] LTND: Primary chain is set to: litecoin
btcpayserver_lnd_litecoin | 2018-07-11 08:52:18.888 [INF] LTND: Initializing litecoind backed fee estimator
btcpayserver_lnd_litecoin | unable to create chain control: -28: Loading block index...
btcpayserver_lnd_litecoin | 2018-07-11 08:52:18.890 [INF] LTND: Shutdown complete
btcpayserver_lnd_litecoin | -28: Loading block index...
btcpayserver_lnd_litecoin exited with code 1

[sangaman]

I came up with a solution for this in the btcwallet code and it resolves this issue nicely in my tests. I couldn't come up with an elegant way to address the issue within the lnd codebase.

[guggero]

Would this indirectly be fixed with the backend healthcheck that is now implemented? Causing lnd to gracefully shut down if the backend isn't ready yet so it can be restarted by docker/kubernetes/systemd?

[NicolasDorier]

The way I fixed it is, I have my UTXO tracker NBXplorer writing some file when the node is ready.
Our LND's fork docker entrypoint just have an infinite loop waiting for this file to be created.