electrs improvement proposal

[romanz]

Current issues

The current version of electrs has several open issues:

RPC overhead

electrs issues RPCs to bitcoind to index the blocks' data (after the initial indexing is over).

In addition, it uses bitcoind to verify that the returned transactions are in the valid chain.

Index structure

The current version uses multiple indices:

• scrtipthash -> funding txid
• funding txid -> spending txid
• txid -> confirmed height

So every scripthash query requires multiple RocksDB reads.

Also, the resulting DB takes 65 GB (as of Oct. 2020).

Inconsistent history

When returning address status, the transactions need to be ordered according to their appearence in a block.

Other

The current version doesn't use the latest Rust features (e.g. async) and probably can benefit from updating its dependencies (e.g. RocksDB and rust-bitcoincore-rpc).

Proposal

By utilizing the "block undo" data (as done in #14053 and #14121, we can add a new getblocklocations RPC to bitcoind.

It would allow electrs to get the actual file system locations of bitcoind Block and BlockUndo data structures.

By reading this data directly from the file system, it is possible to directly build the following index:

| Script Hash Prefix | blk*.dat file index | blk*.dat file offset |
| -------------------- | --------------------- | ---------------------- |
| SHA256(script)[:8] | 16-bit integer | 32-bit integer |

Today, Bitcoin mainnet has ~2300 blk*.dat files, each taking ~128MB.

This index allows retrieving an address' history using a single RocksDB read, followed by file system reads (which can be done concurrently if the blocks are stored on an SSD).

In order to make sure the transactions are in the valid chain, we also keep an in-memory mapping from each block location to its length and block hash. Given a transaction location, this mapping can be used to find the block hash in which the transaction was confirmed (since the blocks are stored consecutevily).

Benefits

• indexing and history queries don't require any RPCs to bitcoind (except the initial retrieval of block locations).
• the resulting "scripthash index" takes ~23GB (as of Oct. 2020).
• "txindex" can be added similarly, taking ~6GB (as of Oct. 2020).
• much better utilization of OS caching (on subsequent queries).
• consistent ordering of confirmed transactions within the same block.

Can I try it?

Please take a look at the next branch - and let me know what you think :)

Make sure to use a custom version of bitcoind, built with the following patch for the proposed getblocklocations RPC:

$ bitcoin-cli getblocklocations 00000000c937983704a73af28acdec37b049d214adbda81d7e2a3dd146f6ed09 10

In order to build the Bitcoin mainnet index, run:

$ cargo build --release
$ export RUST_LOG=electrs=debug 
$ ./query.sh mainnet

The indexing takes around 1.5 hours (using an AMD Ryzen 5, 16GB RAM and a HDD).

To query the index for a specific address history and balance, run:

./query.sh mainnet $BITCOIN_ADDRESS

To run an experimental Electrum server (based on async-std), run:

$ ./server.sh mainnet

in case you are running electrs on devices with limited memory (e.g. Raspberry Pi), please use the following command (it would take longer but use less RAM):

$ ./server.sh mainnet --low-memory

Please let me know about any questions/bugs/issues/suggestions!

[romanz]

CC: @Kixunil @lukechilds @openoms @rootzoll @mynodebtc @craigraw @jlopp @shesek @stadicus

[shesek]

This looks very good, impressive work! The storage savings are significant and its great that you got the bitcoind rpc out of the way of querying. The patch for bitcoind seems simple and useful enough to have a good chance of getting merged relatively soon™ (fingers crossed 🤞). I'll go over the next branch and report back if I have any more specific feedback, hopefully in the next few days.

[Kixunil]

Thanks a lot for heads up! This looks cool and I'd love to use it on my machine with sufficient storage. Unfortunately I can already see some issues with my deployments, my Debian repository and future plans:

Firstly, in order for electrs to access bitcoind data files, I would have to make sure data files have correct permissions. As I run bitcoind and eletrs under different users (in[my Debian repository as well as on an older setup), this would force me to use sysperms=1 in bitcoind, which currently implies disabling the wallet. I'd still like to use that as then even the current version of electrs syncs faster. Ideally sysperms=1 should not disable the wallet, but just be ignored in case of the wallet. The wallet is needed for Eclair, possibly for other projects. As an alternative I could have a separate wallet daemon with exactly same RPC interface as bitcoind and use that one. However, I'm not aware of such daemon existing. There's one written in Go (I think it's used inside lnd) but the interface is not exactly same as the goals of it are different. Also, pinging @nixbitcoin @jonasnick who use isolation in their project too.

The second issue is there's a very nice PR against btc-rpc-proxy which, among other things, allows to run a pruned node and pretend it's not pruned. It does so by fetching blocks from the network. (This is obviously slower, but for some users it may be worth it.) I'd very much like to allow using electrs together with this feature. Obviously this won't work if the blocks are not actually on disk. I'm not sure what's the most efficient way to solve this, but I guess if electrs was able to work the "old" way it'd help. An alternative (insane?) idea is to store the blocks in a special-purpose FUSE that remembers the overall structure but doesn't store transaction data and it does the fetching when needed.

Finally, as far as I understand it this destroys my previous idea of using electrs as txindex=1 but maybe it's not too bad since the index is not stored twice as before. The question is if electrs-style optimization of storing only part of the block hash would be more helpful.

Also, non-issue optimization idea: maybe use SipHash(script, secret) instead of SHA256(script)[:8]? My understanding is that SipHash is specifically designed to be used in hash tables and it should be impossible for an attacker to create collisions (with the intention to flood electrs; although fees could be sufficient) and it's faster. I'm not aware of any guarantees that prefix of SHA256 provides. Disclaimer: I'm not a cryptographer.

[romanz]

Thanks for the replies, much appreciated!

Regarding using SipHash: unfortunately, Electrum protocol forces me to use script_hash = SHA256(script_pubkey) as it is used for RPCs - i.e. instead of using the script_pubkey, the client is subscribing to a script_hash:
https://electrumx-spesmilo.readthedocs.io/en/latest/protocol-methods.html#blockchain-scripthash-subscribe

[Kixunil]

Ah, didn't notice that. Might be nice to compare collisions between script_hash[:8] and SipHash(script_hash) and maybe even patched version of protocol/client that uses SipHash instead of SHA256 to know if it's worth pursuing further.

[romanz]

Sounds good, thanks for the suggestion!

[lukechilds]

Wow, impressive stuff, 23GB of storage in return for an Electrum server is a pretty great tradeoff!

Out of interest, how do these changes affect perf? Specifically I'm interested in:

• Time to build initial index
• Runtime perf of queries
• Runtime RAM usage

Do you have any rough figures/estimates?

This index allows retrieving an address' history using a single RocksDB read, followed by file system reads (which can be done concurrently if the blocks are stored on an SSD).

Assuming the user is using an SSD, this sound like queries would be substantially faster.

Would removing the RPC dependency at query time remove (or at least significantly reduce) the DoS risk with electrs?

We're shipping electrs by default in @getumbrel. However we're currently building out the app store architecture and will likely have to enable txindex due to many third-party apps requiring it. For this reason we are considering changing to ElectrumX for improved perf and to make the most of txindex.

If these changes you mention will bring improved perf to electrs with lower storage requirements than ElectrumX then there would be no reason for us to switch.

[craigraw]

Testing now. I've built the next branch successfully, but I can't seem to find query.sh or server.sh?

I'd like to start by indexing testnet, and specify the db folder so I can put it on an SSD.

[romanz]

I can't seem to find query.sh or server.sh?

Sorry about it - added in https://github.com/romanz/electrs/commit/eb413a0117b950a9baa74d3de05e37cf655e3315

I'd like to start by indexing testnet, and specify the db folder so I can put it on an SSD.

You can run it using:

$ target/release/electrs_query --network testnet --db-dir /path/to/db/ --daemon-dir $HOME/.bitcoin < /dev/null

[romanz]

Out of interest, how do these changes affect perf? Specifically I'm interested in:

* Time to build initial index

* Runtime perf of queries

If everything works as intended, you should be bottle-necked on disk I/O :)

* Runtime RAM usage

There is a trade-off between indexing throughput and RAM usage - could you please share the amount of RAM you have on the device?
The new code is reading whole blk*.dat files into RAM - but we can add a "slower" indexing mode to reduce RAM consumption.

Do you have any rough figures/estimates?

I can share Prometheus graphs for my machine - would it help?

>

Would removing the RPC dependency at query time remove (or at least significantly reduce) the DoS risk with electrs?

The new code is maintaining an in-memory blockheaders' list (syncing with bitcoind each time a new block is found).
Each query is using this list to filter out transaction from blocks that are not in the active chain:
https://github.com/romanz/electrs/blob/82c50feea2ee87bb9c49cb14ef8bd6e93381748b/electrs_index/src/index.rs#L149-L166

We're shipping electrs by default in @getumbrel. However we're currently building out the app store architecture and will likely have to enable txindex due to many third-party apps requiring it. For this reason we are considering changing to ElectrumX for improved perf and to make the most of txindex.

BTW, it is possible to add an (optional) txindex to electrs - it should be smaller (by using txid 8 byte prefix, together with ZSTD compression of RocksDB) and hopefully faster to build than the bitcoind one.

[shesek]

BTW, it is possible to add an (optional) txindex to electrs - it should be smaller (by using txid 8 byte prefix, together with ZSTD compression of RocksDB) and hopefully faster to build than the bitcoind one.

One interesting possibility is making btc-rpc-proxy emulate a txindex-enabled bitcoind node by delegating getrawtransaction requests to electrs's txindex, which would enable the txindex-dependent third-party apps mentioned by @lukechilds without keeping a duplicated index.

[craigraw]

Unfortunately I can't seem to get electrs-query to do anything. I'm starting the process as the same user as bitcoind with

electrs_query --network mainnet --db-dir /mnt/ext/electrs-next/db --daemon-dir ~/.bitcoin

The process runs without exiting, but there is no output to the console, logging in syslog or visible CPU usage. I've tried with the patched bitcoind synced on both mainnet and testnet with my RPi4. The configured db directory is empty too.

[romanz]

Sorry for the incorrect instructions above - please use the following command :

electrs_query --network mainnet --db-dir /mnt/ext/electrs-next/db --daemon-dir ~/.bitcoin < /dev/null

It uses stdin to read the list of Bitcoin addresses to query - will fix soon to use command line arguments instead.

[Kixunil]

Does anyone know if pruned bitcoind keeps whole height -> blockhash index? Depending on it, electrs could also have additional optional index for fetching blocks from the network.

[craigraw]

Thanks - starts up now but after a few seconds exits with the following assertion error:

> electrs_query --network mainnet --db-dir /mnt/ext/electrs-next/db --daemon-dir ~/.bitcoin < /dev/null
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`', electrs_index/src/types.rs:146:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`', electrs_index/src/types.rs:146:13
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `9930`,
 right: `0`', electrs_index/src/types.rs:146:13
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `0`', electrs_index/src/types.rs:146:13

[lukechilds]

One interesting possibility is making btc-rpc-proxy emulate a txindex-enabled bitcoind node by delegating getrawtransaction requests to electrs's txindex, which would enable the txindex-dependent third-party apps mentioned by @lukechilds without keeping a duplicated index.

Phwoar, that's a killer idea @shesek!

If everything works as intended, you should be bottle-necked on disk I/O :)

This is pretty exciting. I'm gonna be really busy with Umbrel for the next week but looking forward to testing this out when some time clears up.

There is a trade-off between indexing throughput and RAM usage - could you please share the amount of RAM you have on the device?
The new code is reading whole blk*.dat files into RAM - but we can add a "slower" indexing mode to reduce RAM consumption.

We advertise 2GB as the minimum requirements but recommend 4 GB. There are also 8GB Raspberry Pi's available but not many users opt for that.

The RAM requirements during the initial index aren't too important, we can dedicate a large part of RAM to electrs to get the initial index completed quickly. What's important is that the memory requirements after the index is initially built aren't too high.

We already set Bitcoin Core's dbcache as a % of the users total RAM on their device for IDB and then reduce after. Would it be possible to do something similar with electrs? Like maybe a ELECTRS_INDEX_CACHE=u64 config option to set the maximum RAM size in MB?

I can share Prometheus graphs for my machine - would it help?

Maybe if you could just share rough % perf improvements between master and next on your machine? Just super rough so I can get an idea what we can expect on Umbrel hardware.

Would removing the RPC dependency at query time remove (or at least significantly reduce) the DoS risk with electrs?

The new code is maintaining an in-memory blockheaders' list (syncing with bitcoind each time a new block is found).
Each query is using this list to filter out transaction from blocks that are not in the active chain:

The reason I ask is because we expose the user's Electrum server via a Tor hidden service for them to easily connect from remotely. It was my understanding that due to depending on Bitcoin Core RPC at query time, it would be quite easy to DoS an electrs server. So for that reason I've been advising users that it's not safe to share their electrs hidden service (unless you really trust whoever you share it with) to prevent DoS attacks.

It sounds like electrs queries bypassing RPC and instead doing direct fs reads could make this a non-issue so it would be safe for Umbrel users to share their Electrum server hidden service with friends and family. Is my assumption correct or would that still be an attack vector?

[romanz]

after a few seconds exits with the following assertion error:

It seems that the block undo data files have a bit different format than I expected :(

Do you run on testnet blocks?
If so, try with:

electrs_query --network testnet --db-dir /mnt/ext/electrs-next/db --daemon-dir ~/.bitcoin < /dev/null

[romanz]

Does anyone know if pruned bitcoind keeps whole height -> blockhash index? Depending on it, electrs could also have additional optional index for fetching blocks from the network.

I don't think that bitcoind prunes this map:
https://github.com/bitcoin/bitcoin/blob/d67883d01e507dd22d1281f4a4860e79d6a46a47/src/chain.cpp#L11-L21

[romanz]

What's important is that the memory requirements after the index is initially built aren't too high.

Currently, the new Electrum server takes ~590MB of RAM on startup, which slowly grows to ~1GB (probably due to RocksDB caching) on my machine.

[romanz]

If I can make it use consistently less than 1GB of RAM - would it be OK?

[romanz]

BTW, we can use https://gitter.im/romanz/electrs if you prefer more chat-like UX :)

[romanz]

Maybe if you could just share rough % perf improvements between master and next on your machine? Just super rough so I can get an idea what we can expect on Umbrel hardware.

I have tested the query performance by running the following command twice (the index & the blocks are on HDD):

$ target/release/electrs_query --network mainnet --db-dir ./db --daemon-dir ~/.bitcoin 1HxRoekwzi2m1vAAFkhfzMkxf8BLTGJ8Qo
[2020-10-25T16:48:42.598Z INFO  electrs_index::metrics] serving Prometheus metrics on 127.0.0.1:4224
[2020-10-25T16:48:44.372Z INFO  electrs_index::index] reading 0 new blocks from 0 blk*.dat files
[2020-10-25T16:48:44.373Z INFO  electrs_index::index] indexed 0 new blocks, 0 DB rows, took 0.124s
[2020-10-25T16:48:44.562Z INFO  electrs_index::map] verified 654258 blocks, tip=0000000000000000000309664d770356c53dfb637f0543754f01d0caf293d711 @ 2020-10-25T16:28:53+00:00
[2020-10-25T16:48:59.065Z INFO  electrs_query] 617a188f4f50f4e7583152991b731039851a954a40f5245b4a143955d600a9ae:26              0.07611825 @ 654205 1HxRoekwzi2m1vAAFkhfzMkxf8BLTGJ8Qo
[2020-10-25T16:48:59.065Z INFO  electrs_query] total: 0.07611825 BTC
^C

$ target/release/electrs_query --network mainnet --db-dir ./db --daemon-dir ~/.bitcoin 1HxRoekwzi2m1vAAFkhfzMkxf8BLTGJ8Qo
[2020-10-25T16:49:04.608Z INFO  electrs_index::metrics] serving Prometheus metrics on 127.0.0.1:4224
[2020-10-25T16:49:06.174Z INFO  electrs_index::index] reading 0 new blocks from 0 blk*.dat files
[2020-10-25T16:49:06.174Z INFO  electrs_index::index] indexed 0 new blocks, 0 DB rows, took 0.118s
[2020-10-25T16:49:06.368Z INFO  electrs_index::map] verified 654258 blocks, tip=0000000000000000000309664d770356c53dfb637f0543754f01d0caf293d711 @ 2020-10-25T16:28:53+00:00
[2020-10-25T16:49:06.406Z INFO  electrs_query] 617a188f4f50f4e7583152991b731039851a954a40f5245b4a143955d600a9ae:26              0.07611825 @ 654205 1HxRoekwzi2m1vAAFkhfzMkxf8BLTGJ8Qo
[2020-10-25T16:49:06.406Z INFO  electrs_query] total: 0.07611825 BTC
^C

1HxRoekwzi2m1vAAFkhfzMkxf8BLTGJ8Qo has almost 1K transactions.
It takes ~15 seconds to get the balance with cold cache, and ~40ms with hot cache :)

[lukechilds]

Thanks @romanz, I've responded on Gitter.

[romanz]

I pushed a few fixes to https://github.com/romanz/electrs/issues/308#issuecomment-716170912 into next branch.

[romanz]

BTW, querying https://blockstream.info/address/147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq with ~4.6k transactions takes (with hot cache) ~80 ms:

+ target/release/electrs_query --network mainnet --db-dir ./db --daemon-dir ~/.bitcoin 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:41.801Z INFO  electrs_index::metrics] serving Prometheus metrics on 127.0.0.1:4224
[2020-10-25T17:47:43.506Z INFO  electrs_index::index] reading 0 new blocks from 0 blk*.dat files
[2020-10-25T17:47:43.506Z INFO  electrs_index::index] indexed 0 new blocks, 0 DB rows, took 0.118s
[2020-10-25T17:47:43.701Z INFO  electrs_index::map] verified 654262 blocks, tip=000000000000000000045f0f166d2a09ecafce154f1c65a683942140d8abb12c @ 2020-10-25T17:38:18+00:00
[2020-10-25T17:47:43.781Z INFO  electrs_query] 0fb799dd28a4fddd21d45f4bbba294529fca312941aa78c63cb7cd4752cdc3c3:0               7.44527391 @ 654122 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 876cf36a54d8b107f6ea9c3ec650b49624c45cdec49b4cf6aa2af750feba04a6:0               6.47247206 @ 654124 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 29c17e335f0b753da12507441071e216b9f94c6031c4d02722672b1a359c0136:0               7.37480838 @ 654133 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] f4478c81c527dc3bc356bd8ec746e83ab04d00b16dc02190f59a728dccec692c:0               6.59722536 @ 654145 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 4b9c29ff202b0e6b8f466161988421f7e87acea381971cfb7ba897b9d1a362d6:0               6.50676138 @ 654170 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 9d3b8bc149a9ba8713e4718bf275601b407614f8e4459622755689e75361fed1:0               7.11656844 @ 654176 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 5d39dd399888a0dd982311236e5dc2ff53d02cd315bfdcfdcde1e44f53f9c784:0               6.36556083 @ 654192 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] ed0f3cef57205968b53afdf22beb6105bb6f6f33b5447fe34bafc3a92f9c58af:0               7.08014241 @ 654223 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 86a0acbbcddba86333d62edaf5c37f80059e6707a51b5a08240b66513411c413:0               7.36787778 @ 654228 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 5f59704fb4523480bea7fddc6b52a800c58ba7ff1473b21af888f4064b66c04d:0               6.30491289 @ 654243 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 7aa3f3e9163927e1277662aea992ba415898714982519a72983d530879075aaa:0               6.55745442 @ 654248 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 660da7a70833154c4bcd5215ee3385b634e8472f5537611e73cd4a3d1d7363ec:0               7.46737724 @ 654257 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] 1adf665761258a0e7594e72db56b51ca6d9c267bb5cef93e00e7ee3bf5660217:0               7.67903151 @ 654258 147SwRQdpCfj5p8PnfsXV2SsVVpVcz3aPq
[2020-10-25T17:47:43.781Z INFO  electrs_query] total: 90.33546661 BTC

[Kixunil]

I don't think that bitcoind prunes this map:

Then I guess optionally storing the block height would be a sufficient for fallback to fetching blocks. Although they'd have to be re-scanned. Unless offset in block is stored too (u32 is enough).

[romanz]

It does so by fetching blocks from the network. (This is obviously slower, but for some users it may be worth it.) I'd very much like to allow using electrs together with this feature. Obviously this won't work if the blocks are not actually on disk. I'm not sure what's the most efficient way to solve this, but I guess if electrs was able to work the "old" way it'd help.

Good point, thanks!
Will try to see how we can support this mode of operation with the new index structure.

[romanz]

FTR, adding a txindex (to support blockchain.transaction.get from unsubscribed addresses) requires an additional ~6GB by using the following DB schema:

| Transaction ID Prefix | blk*.dat file index | blk*.dat file offset |
| --------------------- | --------------------- | ---------------------- |
| txid[:8] | 16-bit integer | 32-bit integer |

To handle prefix collisions, the server filters the transactions after reading them from the file system.

[Kixunil]

Not too bad considering that the chain has 300G already. Can be optional, I guess but not sure if it's very useful.

[lukechilds]

If anyone wants a simple way to test this I've put together a Docker configuration: https://github.com/lukechilds/electrs-next

One command and it'll build romanz/bitcoin:locations and romanz/electrs:next from source, start a mainnet Bitcoin Core node and then run the electrs indexer.

[lukechilds]

Just some stats if it's helpful for anyone, I tested this with the --low-memory flag on a Raspberry Pi 4B.

Indexed to block height 657472 with electrs efeddaa11ceb13d05be3cdf12ea1ad13b5a91841.

Indexing took ~28.5 hours.
RAM usage fluctuated between 400MB-900MB during indexing.
Index grew to ~63G before being compacted to 30GB.

[romanz]

Just some stats if it's helpful for anyone, I tested this with the --low-memory flag on a Raspberry Pi 4B.

Cool, many thanks!

[romanz]

I have opened https://github.com/bitcoin/bitcoin/pull/20702 to add getblocklocations RPC to Bitcoin Core.
Review and comments would be appreciated :)

[Kixunil]

Since Core devs are unwilling to export block locations even as an unstable API, I suggest brainstorming the alternatives here. Firstly, my own goals with electrs are allowing individuals or families to run Electrum using self-hosted server with good UX. If anyone has goals that directly conflict with this, please let me know.

• Give up next and continue with the old approach
• Copy the blocks, bitcoind blocks can be pruned
• Bypass getblocklocations by scanning the files directly and storing the information from them. All the warnings by Core devs apply, we may just decide to ignore them, risk it anyway and deal with potential breakage later if it ever happens
• Create a separate service that gets the blocks from bitcoind and stores them in a custom, efficient format. electrs would then access this service. An advantage of this is it can be reused by btc-rpc-proxy to pretend the chain is not pruned without accessing the network. I quite like this approach.

Old approach

Pros:

• works today, no changes needed
• no stability concerns, especially with --jsonrpc-import
• should work with fake full chain out of the box (didn't test yet)

Cons:

• some people find the performance insufficient - are there other ways to optimize it?
• the resulting index is larger than the one in next branch

Copying blocks

Pros:

• should be straightforward
• we can use our own encoding - even the compression mentioned by gmaxwell
• maybe fetching performance could be even better if we do something smart about storing the blocks?

Cons:

• anyone who wants to run electrs and LND will have to store the chain twice unless tricks like btc-rpc-proxy are used
• alternative Bitcoin Core UIs can't use btc-rpc-proxy now. Perhaps unlikely that someone will try to use them and Electrum at the same time but I wouldn't dare to bet on it. I would be willing to take a look at this if required, but it'll take some time.
• who knows what kind of future features will be unavailable because of pruned bitcoind

Ignoring instability of bitcoind block locations

Pros:

• I guess modifying next to do this should be easy
• Performance and storage savings
• No ridiculous issues with other services

Cons:

• Lack of stability as warned by Core devs - are we determined enough to update the code when storage format changes?
• Without official versioning, even detecting the format could be difficult and error-prone
• If Bitcoin Core doesn't adopt the mentioned compression code, we can't bypass it
• Risk of alienating Core devs who might become less willing to cooperate in the future

Creating separate service

Pros:

• Clean architecture which solves many problems
• We can start fresh and use modern Rust (still 1.41) features
• We can compress the blocks from start so even people who don't want to use electrs, just e.g. LND will benefit from it
• We can do any kind of storage/versioning we wish
• If the storage and interface prove themselves over time maybe we can somehow get it into Bitcoin Core

Cons:

• Possibly lot of work, especially if we need advanced features
• The complexity of another service harms users who refuse to use automated dependency management or an OS that supports it
• Proxying overhead in some scenarios
• People who need non-pruned bitcoind will have to either
  
  
  
  • double their storage
  
  
  • use btc-rpc-proxy (hopefully with direct support for this new service)
  
  
  • wait for a version of such service that has a similar feature to btc-rpc-proxy - fetching blocks from alternate sources
  
  
  • wait until either btc-rpc-proxy or this new service implements an alternative to rescan, if the wallet they use uses internal Core wallet
  
  

I'm not really sure what's the best approach. I like the last one but I'm afraid it will be a shitton of work to solve all the problems associated with it. Perhaps looking into optimizations is saner but is there anything meaningful remaining?

[craigraw]

Seems like a comprehensive summary!

Old Approach

• some people find the performance insufficient

This is correct, but I think understates the problem. The most common Sparrow user issues I have appear to be electrs related. Many of them may be performance driven (with sufficiently large wallets, electrs stops responding to queries), but there are other issues as well which I've filed separately. Particularly, I've noticed that script hash subscription notifications are not reliable on electrs 0.8.6 - again, whether a result of load or not, I'm unsure.

On the other hand, I've been using the next branch for a few months quite intensively and it has been flawless so far, handling multiple large wallets that electrs 0.8.6 struggles with. I don't know how much of the code changed, but it's certainly a great improvement, the size of the index aside. So, I'd really like to see the next codebase have a future, even if the index size is larger than current.

[Kixunil]

Thanks for your insight! I wonder do you have any regtest-based test cases for those failures? Perhaps I could look into it and see if there's something that would improve the situation without having to run an unsupported fork of Bitcoin Core.

Another approach I'm thinking about is taking the new codebase and only changing the index to work similarly to what was before to see if the problem reappear. It could be that some kind of other rearchitecting helped more than the storage changes.

[craigraw]

Unfortunately, particularly with the script hash notifications, the issue is intermittent. Anecdotally, it seems to happen most often with transactions in the mempool. For example, after broadcasting a transaction, someones all of the subscribed script hashes connected to the transaction are notified, sometimes a subset, and sometimes none.

Another approach I'm thinking about is taking the new codebase and only changing the index to work similarly to what was before to see if the problem reappear.

This sounds promising. One of the most reliable ways to see if there are issues is to load a large wallet (300+ subscribed script hashes) and view the CPU usage on a RPi4 or similar device. With electrs 0.8.6 this will run at 100%, while the next branch will use hardly any CPU at all.

[Kixunil]

Anecdotally, it seems to happen most often with transactions in the mempool.

In this case I agree my idea is promising. Very promising. Mempool is not stored in db. I'd even start looking at the code difference first - there's a good chance it'll be easier to compare than rewrite and test the code. Also, what you describe should be quite easy to simulate on regtest.

Meanwhile I came up with another approach that is a combination of old one and the new one and could be a reasonable tradeoff.

Index:
script_hash_prefix -> file, offset, confirmed_height, spent_height - 0 means unspent - nothing was spent in genesis ;)
txid_prefix -> file, offset, confimed_height, spent_height (optional table)
height -> block_id (optional table, falls back to bitcoind RPC, lower performance, lower storage usage)

The transactions belonging to the last block file would be stored separately, without locations as we expect the last file to be more likely to change.

Indexing would work by scanning the files old-style and remembering the locations in files, but also block height for fallback purposes. If anything goes wrong with reading the files, it'd fall back to RPC. Less performant but at least not broken. If the data read from a block file is not a valid transaction, then reindex of locations is triggered. (We assume the block height doesn't change for sufficiently buried transactions.) If data in the block files seems to be invalid (e.g. compressed), then fall back to RPC.

Perhaps height -> block_id could be stored in a big file instead of rocksdb. (Height doesn't need to be stored!) Reading would be fast and trivial:

block_list_file.seek(SeekFrom::Start(height * 32))?;
block_list_file.read_exact(&mut block_id)?;

Plus maintain a small, optimized in-memory cache as well.

There might be other possible optimizations, especially around deduplicating information between txid and script hashes. I'd have to think about it more.

Edit: came up with such optimisation. :)

We could take advantage of the fact that transactions 100 blocks deep are so unlikely to change that we will just refuse to support this scenario. We could store indexes from last 100 blocks separately (even in RAM!). Once a block is buried, we write the common data (offsets, height...) to the end of a huge file and write scripthash_prefix -> position_within_big_file and txid_prefix -> position_within_big_file to a database.

Since we store transactions within block together, we can probably replace block file numbers and hashes with a simple relative offset and store them in a header instead.

Quick and dirty estimate says the large files would be around 3G. I don't dare to estimate size of DB. In a hurry, so maybe I made a mistake somewhere.

[Kixunil]

I took a (not incredibly deep) look at it and the mempool code is rewritten quite a bit more than I expected. But that may be a good news. I don't see any obvious thing that'd make it different but the algorithm seem to have changed which might be it.

I will try to copy the mempool code to the old version. I see possibilities for even more optimizations (removing bunch of allocations and copying) so I will try to do those as well, as a separate commit so that we can compare them.

If anyone wants to help, I'd be happy to see a few files as snapshots of mempool (hex-encoded txes separated by newlines would be fine; mainnet/regtest/whatever) and example queries against those cases.

Edit: looking at where Mempool/Tracker is used it could be locking. I have some idea of how to improve locking, so that could help even more.

[craigraw]

Sounds good! Would this potentially be a reason the next code can handle larger numbers of subscribed script hashes so much better?

[Kixunil]

Writing an answer caused me to think and analyze so much that I'm now rewriting it second time. :laughing:
But I also have a better understanding of the code.

The new version seems to improve several things regarding mempool:

• Uses external crate for bitcoind RPC, which doesn't use Mutex internally
• Keeps mempool transactions in memory, the old one fetches them from the daemon when requested. This involves some locks, including Mutex above
• Uses an algorithm that looks cleaner I did not have the time to understand the difference properly yet

Since the change I attempted is not trivial, I'm thinking about trying to improve locks first (may be surprising but I do believe it's a simpler change). Although the algorithm could fix the unreliability mentioned in https://github.com/romanz/electrs/issues/308#issuecomment-756763638

For example, after broadcasting a transaction, someones all of the subscribed script hashes connected to the transaction are notified, sometimes a subset, and sometimes none.

I need to do something else now. @romanz do you think it's worth changing the locks first or there's a good reason to continue with previous attempt?

[romanz]

The new version seems to improve several things regarding mempool:

The original mempool code was indeed rewritten in next branch - it uses different data structures and way for synchronization.

For example, after broadcasting a transaction, someones all of the subscribed script hashes connected to the transaction are notified, sometimes a subset, and sometimes none.

I think that this may happen due to https://github.com/romanz/electrs/issues/297 - but not sure.

[romanz]

Regarding https://github.com/romanz/electrs/issues/308#issuecomment-756225037 - I think that a "separate service" approach can give us all the benefits we need:

• the blocks can be fetched via P2P (should be faster than JSONRPC).
• after blocks are fetched, the local node can pruned (using pruneblockchain RPC).
• the transactions can be efficiently stored and indexed by electrs.

WDYT?