Lnd: Consider avoiding the need for txindex=1

[Sjors]

I'm running into serious performance issues when using txindex=1 on Bitcoin Core, even without pruning. btcd is also taking weeks to do an IBD although I can't tell if that's because of txindex=1 or some other reason.

It would be sad if users can't (super easily) use their own full node because of this. To quote @sipa: "what?" :-)

Can you explain what this index is used for? Maybe I can help brainstorm some ways to avoid that.

[Roasbeef]

btcd is also taking weeks to do an IBD although I can't tell if that's because of txindex=1 or some other reason.

txindex while downloading doesn't really make btcd much slower. It's just that compared to bitcoind it doesn't have as many cache optimizations, optimizations in general, and generally C++ is faster than Go.

lnd doesn't _unconditionally_ require using txindex as we have the neutrino operating mode, which uses a light client backend. This is the easiest way to run lnd.

Can you explain what this index is used for? Maybe I can help brainstorm some ways to avoid that.

We require the index currently to handle "historical confirmation notification dispatch". This is the case where we register for the confirmation of a transaction, but by the time we register, it has already received a single confirmation. An alternative scenario is that the node has been running independently for some time, then we restart lnd and need to see if something has been confirmed (or just received an initial confirmation).

A similar case applies to nodes that are running in a pruned form, If lnd is offline long enough, then it's best known block has fallen beyond the prune horizon. In this case the node doesn't even _have_ those blocks in order to check to see if something has confirmed. A workaround here, is that for a pruned node, the BlockChainIO implementation would just fetch the blocks from the network and manually scan.

In order to still handle historical conf dispatches w/o a txindex, we'd need to manually fetch+scan each block (either since the notification was requested, or since the last scan attempt), essentially mirroring the chain internally for scanning purposes. All notification requests have a heightHint parameter which gives the ChainNotifier implementation a height to bound how far back in the chain it attempts its scanning attempt.

Modifying that code fragment I linked earlier to support non-txindex'd is pretty straight forward: attempt to use the txindex, if that fails, then scan forwards from the heightHint scanning each block to see if the transaction has been confirmed or not. Alternatively, it can talk directly to the p2p network, and fetch gcs filters. This is a lower bandwidth route.

[Sjors]

lnd doesn't unconditionally require using txindex as we have the neutrino operating mode, which uses a light client backend. This is the easiest way to run lnd.

But the current btcd implementation of neutrino also requires txindex=1. I don't know if the neutrino specifications require it, or whether these is defined fallback behavior for pruned nodes. That's a separate issue of course.

We require the index currently to handle "historical confirmation notification dispatch".

Does lnd need the full transaction details, or only cares if it exists and in what block?

Would a new bitcoind RPC method like exists [transaction_id] [after_block] be useful?

For nodes without txindex=1 this would take care of scanning blocks for that transaction. It could maintain a cache of which transaction ids exist (in which block), down to recently queried after_block values. Users of that method would have be good citizens and use the most recent after_block value they can safely get away with.

This would presumably fail for a node that's pruned to far, in which case lnd could tell the user there's a problem (it's just one of the usability problems with pruned nodes that need a better solution, probably at the bitcoind level).

[Sjors]

Another approach could be to add an optional after_block argument to getrawtransaction and then make that work for pruned nodes. I don't know if there are performance tradeoffs between merely knowing if a transaction exists, knowing which block it's in and knowing the full raw transaction.

[Sjors]

See also some back and forth about this on https://github.com/bitcoin/bitcoin/issues/3220#issuecomment-354309662 , #bitcoin-core-dev, and #lightning-dev.

[Roasbeef]

But the current btcd implementation of neutrino also requires txindex=1

No, it doesn't.

Does lnd need the full transaction details, or only cares if it exists and in what block?

It doesn't need the full tx details. It only requires the hash of the block, the block height, and the index of the tx within the block.

Would a new bitcoind RPC method like exists [transaction_id] [after_block] be useful?

A new RPC added to bitcoind isn't required to implement what I described above. Instead there just needs to be a fall back of:

• First, try the txindex, if it doesn't exist then...
• Try to fetch the block raw over RPC, if you can't then...
• Connect to some of the full-node's peer's (using RPC to look them up), and fetch the block directly from the p2p network.

The btcsuite set of libraries if fully featured and makes that final fallback point rather easy to implement.

[wpaulino]

I'd like to give this a try if no one else is working on it.