Raspiblitz: [REQ] Electrum Server

[annaparker]

Here is the pretty lightweight Electrum Server:

https://bitcointalk.org/index.php?topic=4589797.0

Do you think can we implement it? That would be great.

[cedricwalter]

what for in the our case? can you elaborate how it will fit with raspiblitz?
Thanks

[annaparker]

If you are using Electrum (https://electrum.org/) as Bitcoin wallet, then you are able to connect to your own Electrum server (by localhost if raspiblitz is on the same network), which is most secure option instead to use unknown Electrum servers, which can store logs, etc.

[cedricwalter]

You may want to use https://github.com/chris-belcher/electrum-personal-server instead, there is numerous project doing this. I don't think that adding electrum on raspiblitz is a good idea, raspberry pi are cheap. I wouldn't merge it in raspiblitz but maybe official project maintainer are interested in the idea?

[rootzoll]

I havent had the time to try an additional electrum server on the raspiblitz - it could be to heavy weight, not sure, might work. It would be great if someone from the community could try it out and make some benchmarking and report back.

Also I talked shortly with Thomas (programmer of electrum wallet) and he says that it should not be so much difference between electrum-personal-server and electrumX https://github.com/kyuupichan/electrumx from the perfomance side - especially if we have full blockchain (non-pruning) and txindex running on the raspiblitz anyway. And I like the idea, that the raspiblitz could serve more than just personal needs - being a backend/infrastructure for friends and family.

[TheCharlatan]

I have tried running ElectrumX on a couple of arm boards (raspberry 3, odroid XU4, pine64), and it was quite painful. It takes a long time so sync and you will probably run out of memory at some point.

[rootzoll]

@TheCharlatan thanks for reporting your experience.

The syncing of bitcoind is a pain raspiblitz users are used too ;) .. we have ftp/torrent prepared blockchain downloads to cope with this particular problem. The interesting part is how much extra ressources electrumx server eats up in addition to bitcoind and lnd and could all three run on a small powered arm board like the raspberrypi3+?

[TheCharlatan]

I can restart some arm devices end of the week to get a proper gauging, but on my dev environment ElectrumX running in docker eats between 0.5 and 1GB on testnet.

[romanz]

Please take a look at the following issues (adding electrs support for embedded devices):
https://github.com/romanz/electrs/issues/38
https://github.com/romanz/electrs/issues/58
https://github.com/romanz/electrs/issues/65

[hopitas]

I've run electrumx on low spec virtual server. It was difficult. The syncing of electrum db on top of the bitcoin blockhain took long time. So when I did sync I just beefed up the vserver, which is not possible in raspis case...

[Emzy]

I have it running on the raspiblitz.
The first sync takes 1,5 days. Every restart takes 2 hours, because it dose a full compaction of the database on every startup.

This are my startup parameters:
cargo run --release -- -vvvv --index-batch-size=10 --jsonrpc-import --db-dir /mnt/hdd/electrs --cookie="raspibolt:PASSWORD" --electrum-rpc-addr=0.0.0.0:50001

[Emzy]

It's not a bug. As fare as I looked at the source, it is intended.
I think the problem is the slow IO to the usb hard disk on the PI. Because it seems all the 35GB of db is read and written at the full compaction.

[duckartes]

Transferring a fully synced ElectrumX DB to a Raspberry Pi3 works, as long as the Pi stays always online it can keep up with incoming blocks (as far as I recall.) I could even connect to it over Tor from a remote site, although querying took a a little while.

I've since moved to Electrs to test as I can actually sync it fully on the Pi without having to transfer a synced DB. I haven't tested remotely connecting to it to query yet.

[romanz]

@Emzy AFAIU, full compaction should happen once (after the initial import is over), and since later updates come at a low rate, the "natural" RocksDB auto-compactions should take care of them quite well. Therefore, electrs should skip the full-compaction stage on subsequent restarts (unless there is a bug in the implementation...).

EDIT: found the bug and fixed it at https://github.com/romanz/electrs/commit/1dcc8ac575963120a520e8356d8a89295f75a573. Thanks for reporting this issue :)

[Emzy]

@Emzy AFAIU, full compaction should happen once (after the initial import is over), and since later updates come at a low rate, the "natural" RocksDB auto-compactions should take care of them quite well. Therefore, electrs should skip the full-compaction stage on subsequent restarts (unless there is a bug in the implementation...).

EDIT: found the bug and fixed it at romanz/electrs@1dcc8ac. Thanks for reporting this issue :)

Yes this fixed it!
Starts fast now.

[Emzy]

Now let's see what will happen wenn we have a full mempool.
Will the PI handle it.

[rootzoll]

@Emzy any long term observations? Did it ran stable for you?

[Emzy]

Yes, running fine on the PI 3 B+ for over 20 days without restart.

[rootzoll]

If anybody is at the 35c3 over the next days ... lets meet at the bitcoin assembly, exchange experiences and maybe build a test integration for the RaspiBlitz :) Hope to see you there.

[dikkechill]

As raspiblitz is heavily using raspibolt anyway, have a look at the raspibolt guide for the Electrum Personal Server: https://github.com/Stadicus/guides/blob/master/raspibolt/raspibolt_64_electrum.md
It might be relatively easy to add this as an option to the raspiblitz installer. You should probably think about the lifecycle and e.g. how to get in (security) updates, but that is a similar issue for other components.

[fluidvoice]

As raspiblitz is heavily using raspibolt anyway, have a look at the raspibolt guide for the Electrum Personal Server: https://github.com/Stadicus/guides/blob/master/raspibolt/raspibolt_64_electrum.md
It might be relatively easy to add this as an option to the raspiblitz installer. You should probably think about the lifecycle and e.g. how to get in (security) updates, but that is a similar issue for other components.

I don't see the reason to include Electrum in this project. The objective of this project is to make it cheap and EASY to run your own Lightning node.

[raumi75]

IMHO it is important to also take care of your on chain transactions. Running your own Electrum server increases your privacy and reliability. I already use my RaspiBlitz for my SPV-Wallet (BRD on iPhone) to increase privacy. We already have the full blockchain on our drive. Why not get the most out of it?

[fluidvoice]

IMHO it is important to also take care of your on chain transactions. Running your own Electrum server increases your privacy and reliability. I already use my RaspiBlitz for my SPV-Wallet (BRD on iPhone) to increase privacy. We already have the full blockchain on our drive. Why not get the most out of it?

How does running your own Electrum server increase your privacy and reliability over using other wallets with your Raspiblitz full node?

[raumi75]

I have to admit that I don't know how Electrum stacks up against other wallets. What I meant to say was if you are using Electrum, your privacy increases if you connect to a personal server.

[fluidvoice]

I have to admit that I don't know how Electrum stacks up against other wallets. What I meant to say was if you are using Electrum, your privacy increases if you connect to a personal server.

Again, Raspiblitz is a Lightning node (on RPi). Hence it's focused on Lightning functionality and wallets.
Only if Electrum wallets could be supported easily without much extra work would it make sense IMO. Otherwise, it's prob best to fork this code/repo and modify it for Electrum.

[rootzoll]

@raumi75 @fluidvoice RaspiBlitz is also a Bitcoin Fullnode on a RPi - so its valid to also provide some additions on that level. A lot of people ask about the electrum feature and it could be a nice service to family and friends letting them use your electrum server to run their lightweight electrum clients agianst. Thats why its on my list of ideas, but somehow scheduled after version 1.0 release and not with high prio.

I am open for PullRequests on this one. A first contribution would be adding a config script (/home/admin/config.scripts) that by parameter would install/emove the electrum server .. check the already existing config scripts as templates.

[DaReaLDeviL]

Whould love this feature too (in Services). Keep up the good work. Like already stated, please help to implement it https://github.com/Stadicus/guides/blob/master/raspibolt/raspibolt_64_electrum.md

[Emzy]

Results of my last test:
electrs took nearly 2 days to sync and works just fine.

Electrum Personal Server needs a few minutes to generate the addresses from your XPUB key.
After that you can use it, if your Electrum wallet if unused/new.
If you have old transactions in it. It will take about 4 hours to rescan 2 years. The rescan is performed by the bitcoind, you have to enable the wallet function.

For both options there is no txindex needed.

I would advice to use the Electrum Personal Server on raspiblitz. It is more complicated for the user. Because you have to provide your XPUB key. And you have to rescan to use a wallet already with history on it.
But it is faster and more lightweight.

I used this guide to run it: https://github.com/Stadicus/guides/blob/master/raspibolt/raspibolt_64_electrum.md

Any questions or additions?

[openoms]

I will start to test electrs on the Odroid HC1/HC2/XU4. My Rock64 is stable yet. Please share your experiences if you are running a node on DietPi https://github.com/openoms/raspiblitz/issues/26

[rootzoll]

@Emzy much thanks for runnig these tests :D

So I think we will go with electrum personal server for the v1.1 while at the same time switching the txindex to 0. providing the xpubkey should be possible to automate in the config script (I hope so). And its fair to assume that there are no transactions older than a year for rescan - because there was no RaspiBlitz a year ago ;)

[openoms]

Got the electrs working on my RPi as well. It is stable once built and the database is ready. Great advantage that it does not require scanning fro every XPUB vs. Electrum Persoanl Server. You can just point your Electrum and open any wallet with full privacy!
Screenshot with htop details: https://twitter.com/openoms/status/1108278035277639680
(should say that everything was maxed out during building the code and the database).
Tutorial and script: https://github.com/openoms/bitcoin-tutorials/blob/master/electrs/README.md

I would vouch for the option to have electrs in the RaspiBlitz (especially for better equipped boards, like the Odroid HC1 and XU4).

With any Electrum server implemented we presume the use of an external Electrum wallet. Might as well not limit to only the given XPUBs.

[openoms]

Big news from AcinQ today, the Eclair wallet is enabled for receiving payments: https://twitter.com/acinq_co/status/1111322747169357824
It is served by an Electrum server and you can set it to be a custom one.

You can test the Electrum Server in Rust by running this one line on your RaspiBlitz terminal:
wget https://github.com/openoms/bitcoin-tutorials/raw/master/electrs/electrs_install_on_RaspiBlitz.sh && sudo chmod +x electrs_install_on_RaspiBlitz.sh && ./electrs_install_on_RaspiBlitz.sh

[openoms]

I get this though when connecting Eclair (my server is working with the Electrum Wallet well):
INFO - [192.168.1.146:37421] connected peer TRACE - RPC Done DEBUG - [192.168.1.146:37421] shutting down connection ERROR - [192.168.1.146:37421] receiver failed: invalid request - maybe SSL-encrypted data?: [22, 3, 1, 0, 158, 1, 0, 0, 154, 3, 3, 116, 183, 21, 237, 62, 174, 177, 60, 162, 132, 221, 35, 230, 36, 0, 20, 1, 103, 181, 28, 173, 194, 24, 164, 193, 67, 144, 230, 175, 45, 221, 3, 0, 0, 16, 192, 44, 192, 43, 192, 47, 192, 19, 192, 20, 0, 156, 0, 47, 0, 53, 1, 0, 0, 97, 255, 1, 0, 1, 0, 0, 0, 0, 33, 0, 31, 0, 0, 28, 68, 105, 101, 116, 80, 105, 46, 99, 97, 98, 108, 101, 46, 118, 105, 114, 103, 105, 110, 109, 101, 100, 105, 97, 46, 110, 101, 116, 0, 23, 0, 0, 0, 13, 0, 20, 0, 18, 4, 3, 8, 4, 4, 1, 5, 3, 8, 5, 5, 1, 8, 6, 6, 1, 2, 1, 0, 5, 0, 5, 1, 0, 0, 0, 0, 0, 11, 0, 2, 1, 0, 0, 10] INFO - [192.168.1.146:37421] disconnected peer
Will test setting up ssl: https://github.com/romanz/electrs/blob/master/doc/usage.md#ssl-connection

[rootzoll]

Being able to provide the RaspiBlitz as a social trusted Full Node for friends and family is one of the great goals :) This fits in nicely. I can see the RaspiBlitz that provides to friends and family that just run a thin mobile wallet like eclair the bitcoin info via Electrum and also provides channel capacity and a good route into the rest of the network, so that they dont need much care on channels etc.

@openoms - the SSL ties into this issue, that would plan to have one NGINX server with a Lets Encrypt SSL that forwards to all the other HTTP based services. https://github.com/rootzoll/raspiblitz/issues/255

[rootzoll]

@openoms please coordinate with @Emzy on which Electrum server will work with the eclair mobile app.

[Emzy]

So we are all more on the side of electrs? Maybe the better alternative.

We can also have a torrent for the database?
So there are not a 18+ hours sync time.

I will test the script "https://github.com/openoms/bitcoin-tutorials/raw/master/electrs/electrs_install_on_RaspiBlitz.sh"
I will also test NGINX and Eclair.

[openoms]

I managed to make my electrs (on DietPi, Odroid XU4) serve the Eclair wallet! Works even from the outside with a dynamic DNS.
This pointed to the right directions, soon to be implemented in the electrs readme. https://github.com/romanz/electrs/pull/126/commits/18890b78e3541f9f5a6f58638ba818891c724443
Would like to clarify the security and privacy drawbacks of the setup a bit and start a more detailed guide for the RaspiBlitz.

Curious to see how the Raspberry takes the beating. My setup is now serving electrs, nginx and certbot on top of the RaspiBlitz stuff with Tor and RTL. Free Mem: 1191M / 1994M

[romanz]

Many thanks for testing and documentation efforts, much appreciated!
Please let me know if there is anything I can do to help this use-case :)

[openoms]

Thanks @romanz for keeping an eye on us!

I have set up the whole thing electrs, nginx and certbot with systemd services on my two running nodes, Odroid XU4 and HC1. Seen Free Mem 716M / 1994M only with Eclair connected. We might run into issues with the RPI, I am very curious. There is a swapfile set up, so running out of memory would cause a serious slow down, maybe not errors. I will spin up my RPi node as well.

Please have a look and maybe test this script with the RaspiBlitz specific settings: https://github.com/openoms/bitcoin-tutorials/blob/master/electrs/electrs_automation_for_Eclair.sh
(it is completing the previous https://github.com/openoms/bitcoin-tutorials/blob/master/electrs/electrs_install_on_RaspiBlitz.sh)

[rootzoll]

TODO: https://github.com/openoms/bitcoin-tutorials/tree/master/electrs

[fluidvoice]

TODO: https://github.com/openoms/bitcoin-tutorials/tree/master/electrs

to be optional, right? Raspiblitz should always default to the minimal install (least RAM resources possible) so to give room for dev's who might want to run their own apps on top of a bitcoin full-node.

[rootzoll]

@fluidvoice optional correct - something you can switch on/off and see if it works for you.

[openoms]

The intermittent running of electrs seems to be feasible even on the RPi. Great test results here from @Himbeergeld: https://github.com/openoms/bitcoin-tutorials/issues/2#issuecomment-486417803

[openoms]

The Electrs is serving well on the RPi4 4GB and flawless for months on the Odroid HC1 2GB.
Installing Rust and building the Electrs binary from source is a lengthy process, so I suggest we could include those in the SDcard image (build_sdcard.sh) and trigger to build the database build from the Services menu as an option.

I had multiple problems with installing Nginx and Certbot to serve remote SSL connections, mainly because of their many dependencies breaking during setup.
The better option would be to offer a hidden Tor service to connect to remotely: https://github.com/openoms/bitcoin-tutorials/tree/master/electrs#tor-hidden-service

I am thinking that there could be a separate downloadable script with a dialog based GUI (like the rest of the RaspiBlitz menu) to help testing integration.

Other specifics:

• electrs should be owned and run by the user bitcoin
• needs to copy the RPC password to either the electrs.service file or to config.rs. See

# to preserve settings:
# see https://github.com/romanz/electrs/blob/master/src/config.rs
# sudo nano $HOME/electrs/src/config.rs 
# change the lines:
# 73: from: .takes_value(true), to: .default_value("raspibolt:PASSWORD B"),
# 132: from .default_value("Welcome to electrs (Electrum Rust Server)!") to your custom message

[Emzy]

The Electrs is serving well on the RPi4 4GB and flawless for months on the Odroid HC1 2GB.

I startet yesterday with testing your 1_electrs_on_RaspiBlitz.sh script from https://github.com/openoms/bitcoin-tutorials/tree/master/electrs

It is using the admin user and is still syncing (PI4 4GB with USB3-HDD). I will test the new one afterwards.

Installing Rust and building the Electrs binary from source is a lengthy process, so I suggest we could include those in the SDcard image (build_sdcard.sh) and trigger to build the database build from the Services menu as an option.

The better option would be to offer a hidden Tor service to connect to remotely: https://github.com/openoms/bitcoin-tutorials/tree/master/electrs#tor-hidden-service

I agree.

I am thinking that there could be a separate downloadable script with a dialog based GUI (like the rest of the RaspiBlitz menu) to help testing integration.

Other specifics:

• electrs should be owned and run by the user bitcoin
• needs to copy the RPC password to either the electrs.service file or to config.rs. See

# to preserve settings:
# see https://github.com/romanz/electrs/blob/master/src/config.rs
# sudo nano $HOME/electrs/src/config.rs 
# change the lines:
# 73: from: .takes_value(true), to: .default_value("raspibolt:PASSWORD B"),
# 132: from .default_value("Welcome to electrs (Electrum Rust Server)!") to your custom message

[openoms]

@Emzy I have done some major refactoring making the config file generation safer and introduced the user electrs to run the service.

The SSL setup got simplified, but it is probably best to use the Tor hidden service to connect: https://github.com/openoms/bitcoin-tutorials/blob/master/electrs/Tor_Hidden_Service_for_Electrs.md

[Emzy]

I will test it again. The last version I testet is running electrs with user-id admin.
Tnx for the effort!

[Emzy]

I tested 1_electrs_on_RaspiBlitz.sh and 2_electrs_systemd_service.sh and it worked super fine. Great work!
The certbot at 3_Nginx_and_Certbot_for_SSL.sh makes it complicated. You need Dyndns and a port forward working.
Electrum works fine with a self signed certificate. And even cache it. So you are save after the first connect.
I would recommend to have certbot at a option and just use a self signed certificate as default.

What do you think? @openoms

[openoms]

@Emzy thank you for the suggestion, a self signed certificate would make a lot of sense. It still encrypts the communication, but there is no need for a central authority to verify the certificate as we are connecting to a known and trusted RaspiBlitz.
I think all three scripts could be made to one (with certbot and it's requirements removed).
As an extra I could add the automatic setup of a Hidden Service for Electrs too if Tor is enabled on the node already.

[openoms]

Update: https://github.com/openoms/bitcoin-tutorials/blob/master/electrs/electrs_to_RaspiBlitz.sh
The setup process got hugely simplified.
The electrs script is now unified and uses a self signed certificate.
It installs:

• Rust
• Electrs
• Nginx for connecting through SSL with a self-signed certificate
• Tor Hidden Service if Tor is active
• runs as a dedicated user: electrs

[Emzy]

Testet it. Runs fine.
I added a check for an already generated certificate.
And a Tor example.
https://github.com/openoms/bitcoin-tutorials/pull/17

[openoms]

made the script to match the config.scripts style.
Tested on and off, works.
https://github.com/openoms/raspiblitz-extras#electrum-rust-server

I suggest this could be implemented to the SERVICES menu and be part of the 1.4 release.
Only thing that we should somehow separate the menu to avoid recommending this to be run on an RPi3 / 1 GB RAM device.

[Emzy]

I had no issues running it on a RPi3. I will do a new test on a RPi3.

[openoms]

PR is now in: https://github.com/rootzoll/raspiblitz/pull/868

@emzy great to hear. No need to restrict the RPi3-s then. Other projects run it on that hardware too. I think memory is an issue with numerous LN channels, but with 1GB RAM one is limited on that anyway.

[Emzy]

@openoms I switched to a Raspberry PI 3+ to test electrs again. The initial sync took 2,5 days with a HDD.
Opening an Electrum wallet is slow but works. A wallet with 20 transactions took 15 seconds and one with 100 transactions took 1 Minute.
My LND has only one channel open, maybe that helps.

After the first sync of electrs I had to reboot. Because it was too slow to open a wallet in Electrum. I think something else was saturating the USB2 of the HDD.

I would recommend a PI4 for it, but IP3+ works.

[openoms]

@Emzy great test results, thank you for sharing! Will include them in the related guide.

[openoms]

@rootzoll if the electrs database is already present the install script could move it to the new directory to avoid manual steps or reindexing as mentioned in this tweet: https://twitter.com/rootzoll/status/1209047432237068288?s=20

Whoever used my guide and script( https://github.com/openoms/bitcoin-tutorials/tree/master/electrs) would have the electrs datbase in : /mnt/hdd/electrs/db owned by electrs:electrs .

[openoms]

Currntly the electrs install script shows too detailed / redundant info. The Hidden Service QR code is accessible from the tor menu and should not be shown at the end of the setup.

Will clear and will show a nice summary in a message box.

[rootzoll]

On testing the install and the indexing worked so far ... but at the end the
nginx service seems dead. @openoms any clue?

nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2020-01-22 02:17:50 GMT; 4 days ago
     Docs: man:nginx(8)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

[theotherside]

My error is slightly different that rootzoll.

[openoms]

On testing the install and the indexing worked so far ... but at the end the
nginx service seems dead. @openoms any clue?

nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2020-01-22 02:17:50 GMT; 4 days ago
     Docs: man:nginx(8)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

For any Nginx config failure running
sudo nginx -t
points to the problematic file and line.

[openoms]

The output on my test node:

sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Do you get any errors @rootzoll?

[rootzoll]

@openoms tahts on my machine after runnin the electrs install script

nginx: [emerg] open() "/etc/nginx/sites-enabled/btcpayserver" failed (2: No such file or directory) in /etc/nginx/nginx.conf:62
nginx: configuration file /etc/nginx/nginx.conf test failed

[openoms]

Got this.

There is a symbolic link in /etc/nginx/sites-enabled/:

lrwxrwxrwx 1 root root   39 Jan 21 04:13 btcpayserver -> /etc/nginx/sites-available/btcpayserver

The bonus.btcpayserver.sh only removed the sudo rm -f /etc/nginx/sites-available/btcpayserver so that broke.

Run:
sudo rm -f /etc/nginx/sites-enabled/btcpayserver
to fix.

Adding it to the script now.

[rootzoll]

OK ElectRS tested and works so far. Will be part of v1.4 release. Closing issue.