Electrs: Electrs running out of memory on testnet Umbrel node with RockPro64

[thunderbiscuit]

Hi there,

I have been hacking away at making my Umbrel node work with the bitcoin testnet on a RockPro64 SBC. The bitcoin daemon syncs up nicely, but as soon as the electrs server starts, my process runs out of memory and restarts. I attach screenshots of what is happening below. After discussing with the Umbrel devs, they suggested I open an issue here to maybe figure out what is going on. The process downloads the block headers fine, but everytime it tries to apply the new headers, that's where it runs out of memory. This process loops infinitely until I kill the electrs service.

Here are some more info on my setup: my RockPro64 has 4GB of memory, 2.5 of which is typically used by the bitcoin daemon. I can see there is an electrs.toml file with some options in there... is there maybe a way for me to cap the amount of memory used by the server?

[Kixunil]

Yes, you can cap the memory by adding jsonrpc_import = true. From the log we can clearly see it's not set.
I believe it's the responsibility of Umbrel developers to make the default configuration sane, so it's their bug.
Thus it may be better to report it to them and let them fix it according to their integration policy so that you don't mess up their config (unless they specifically told you to modify electrs.toml to your liking).

[thunderbiscuit]

Thank you! Will link up with them and see what is the best way forward. @lukechilds this thread might be of interest to you.

Can you tell me more about jsonrpc_import = true? How does it help electrs know how much memory to allocate?

[Kixunil]

It just switches the algorithm to fetch the blocks over Bitcoin RPC instead of reading them directly from disk which happen to also require less memory. (IDK why, @romanz may be able to explain.) Note that this is slower but less bad then not working at all.

[romanz]

It just switches the algorithm to fetch the blocks over Bitcoin RPC instead of reading them directly from disk which happen to also require less memory.

Exactly :)

The original code used getblock RPC for get blocks from bitcoind, but it was quite slow - so I switched to reading whole blk*.dat files into RAM and parsing them using rust-bitcoin crate (which becomes I/O-bound on regular HDDs).
This works much faster but require more RAM (since each blk*.dat file takes ~128MB).

[thunderbiscuit]

I'll give that a try and keep you posted. Cheers!

[lukechilds]

I'm not quite sure why but electrs works fine reading blocks from disk on the recommended Umbrel setup (Raspberry Pi), it seemed like something specific to this SBC that caused the memory issue.

Although we do have swap configured on Umbrel OS so I guess maybe users are swapping quite heavily during the initial electrs index. I didn't notice that during testing but it could be the case.

Seems like hopefully https://github.com/romanz/electrs/issues/308 with --low-memory could be the perfect solution.

Apologies @romanz I said I'd test that a while ago but I've been struggling to find the time. I'll try and put some time aside this weekend to test it out.

[Kixunil]

@romanz 128M doesn't seem that much to me, that's strange. Could it be reimplemented by using a smaller buffer that would load it piece-by-piece? Or maybe try to allocate the buffer with C alloc (since Rust alloc doesn't support fallible allocation yet) and if it fails, fall back to jsonrpc_import.

@lukechilds if there's swapping involved there's a good chance it'll lead to a greater performance loss than using RPC, so I'd suggest measuring it.

[romanz]

128M doesn't seem that much to me, that's strange.

Good point - I kept the "slow" mode in master to work with JSON RPC (in case electrs doesn't has read access to the blk*.dat files), so when developing the "fast" mode I tried to optimize for disk read speed.
Therefore, In addition to reading whole files into RAM, it also uses a pipeline for reading, parsing, indexing and writing to RocksDB (using multiple threads) - resulting in an even more RAM usage.

Could it be reimplemented by using a smaller buffer that would load it piece-by-piece?

This is indeed the current "low-memory" mode in the next branch (it just reads the blk*.dat file block-by-block):
https://github.com/romanz/electrs/blob/883d284474eab2edd2c0eb14a2144bdca87940c3/electrs_index/src/index.rs#L349-L369

[romanz]

Please let me know if this is still an issue :)

[thunderbiscuit]

Not an issue for me anymore! I have applied the recommendations from @lukechilds and @Kixunil:

# electrs.toml
index_batch_size = 10
jsonrpc_import = true

My node now runs nicely. Not sure if Umbrel has anything else they'd like to add, but on my end at least, this issue is ready to be closed.