Electrs: Failed to create RocksDB directory

Created on 22 May 2020 · 18Comments · Source: romanz/electrs

Hi,

I've built the docker image successfully and tried to run the container/server with:

docker run --network host \
           --volume /mnt/clouddrive/.bitcoin:/home/user/.bitcoin:ro \
           --volume /mnt/clouddrive/electrs:/home/user \
           --rm -i -t electrs-app \
           electrs -vvvv --timestamp --db-dir /home/user/db

Here is the output with RUST_BACKTRACE=full:

Config { log: StdErrLog { verbosity: Trace, quiet: false, show_level: true, timestamp: Millisecond, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/home/user/db/mainnet", daemon_dir: "/home/user/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), electrum_rpc_addr: V4(127.0.0.1:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 6, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.3 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
2020-05-21T22:12:28.280+00:00 - DEBUG - Server listening on 127.0.0.1:4224
2020-05-21T22:12:28.285+00:00 - DEBUG - Running accept thread
2020-05-21T22:12:29.760+00:00 - INFO - NetworkInfo { version: 190001, subversion: "/Satoshi:0.19.0.1/", relayfee: 0.00001 }
2020-05-21T22:12:31.521+00:00 - INFO - BlockchainInfo { chain: "main", blocks: 631229, headers: 631229, verificationprogress: 0.99999647717456, bestblockhash: "00000000000000000011d5c21b0125664be4052e10b95e35ef1bf375244d48a7", pruned: false, initialblockdownload: false }
2020-05-21T22:12:31.523+00:00 - DEBUG - opening DB at "/home/user/db/mainnet"
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { message: "Failed to create RocksDB directory: `Os { code: 13, kind: PermissionDenied, message: \"Permission denied\" }`." }', src/store.rs:60:17
stack backtrace:
   0:       0x5569a06298 - backtrace::backtrace::libunwind::trace::h88e4781b95be5e7d
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1:       0x5569a06298 - backtrace::backtrace::trace_unsynchronized::h27424f3324e761be
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2:       0x5569a06298 - std::sys_common::backtrace::_print_fmt::h7d3510b3a4048fe7
                               at src/libstd/sys_common/backtrace.rs:77
   3:       0x5569a06298 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hc9d57e2bd1e53ca8
                               at src/libstd/sys_common/backtrace.rs:59
   4:       0x556987f538 - core::fmt::write::h3a82d9a133640e63
                               at src/libcore/fmt/mod.rs:1052
   5:       0x5569a05a48 - std::io::Write::write_fmt::h0d8caa20275259fe
                               at src/libstd/io/mod.rs:1426
   6:       0x5569a05688 - std::sys_common::backtrace::_print::ha706fb52b493e47b
                               at src/libstd/sys_common/backtrace.rs:62
   7:       0x5569a05688 - std::sys_common::backtrace::print::h62d24dbbbcdb47e7
                               at src/libstd/sys_common/backtrace.rs:49
   8:       0x5569a05688 - std::panicking::default_hook::{{closure}}::heba3e973eafbb1d9
                               at src/libstd/panicking.rs:204
   9:       0x5569a04df4 - std::panicking::default_hook::he515c32b9ced0a2d
                               at src/libstd/panicking.rs:224
  10:       0x5569a04df4 - std::panicking::rust_panic_with_hook::h580ddde2c8bb727f
                               at src/libstd/panicking.rs:472
  11:       0x5569a04a98 - rust_begin_unwind
                               at src/libstd/panicking.rs:380
  12:       0x556987d704 - core::panicking::panic_fmt::hf152853fda7ac566
                               at src/libcore/panicking.rs:85
  13:       0x5569882be0 - core::option::expect_none_failed::h6b4d39870d82acf6
                               at src/libcore/option.rs:1199
  14:       0x5569896bbc - electrs::main::h3aee48bad518ed0d
  15:       0x556988bf1c - std::rt::lang_start::{{closure}}::h52a53a942887a108
  16:       0x556988f0a0 - main
  17:       0x7f9b505d24 - __libc_start_main
  18:       0x5569854160 - <unknown>

I see that /mnt/clouddrive/electrs exists and has empty .bitcoin dir inside. It's owned by root with 755 permissions.

question

Source

OvoNemaSmisla

Most helpful comment

He means don't make it publicly accessible at all.

So if you want it accessible over the internet, put some auth in front like an SSH tunnel/VPN/Tor etc.

lukechilds on 27 May 2020

👍3

All 18 comments

Looks like something is messed up with the docker image. Isn't docker supposed to deal with these kinds of things? I never used it, but people tried to sell it to me as a magic solution to these problems. ;)

Kixunil on 22 May 2020

😄1

Well I'm pretty new to Docker world myself, so I wouldn't know 😀
I've recently managed to run ElectrumX server as a Docker container with a lot of help form @lukechilds, so I wanted to try Electrs.

It's a strange error. You would think that a write premission is missing on the /mnt/clouddrive/electrs, but how come it's able to create the dir in the first place and then create .bitcoin dir inside?

OvoNemaSmisla on 22 May 2020

@Kixunil Docker is great at many things, but handling permissions is definitely not one of them!

From the Dockerfile:

RUN adduser --disabled-login --system --shell /bin/false --uid 1000 user

USER user

So the electrs process inside the container is running with uid 1000 which doesn't have permission on the host to write to that dir.

@OvoNemaSmisla can you try running:

$ sudo chown -r 1000:1000 /mnt/clouddrive/electrs

And then run the container again?

lukechilds on 22 May 2020

👍1

Hi Luke!

Setting the ownership to 1000:1000 resolved the issue, but the first run crashed:

Config { log: StdErrLog { verbosity: Trace, quiet: false, show_level: true, timestamp: Millisecond, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/home/user/db/mainnet", daemon_dir: "/home/user/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), electrum_rpc_addr: V4(127.0.0.1:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 6, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.3 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
2020-05-22T13:20:24.578+00:00 - DEBUG - Server listening on 127.0.0.1:4224
2020-05-22T13:20:24.583+00:00 - DEBUG - Running accept thread
2020-05-22T13:20:24.653+00:00 - INFO - NetworkInfo { version: 190001, subversion: "/Satoshi:0.19.0.1/", relayfee: 0.00001 }
2020-05-22T13:20:24.657+00:00 - INFO - BlockchainInfo { chain: "main", blocks: 631301, headers: 631301, verificationprogress: 0.9999953067958438, bestblockhash: "0000000000000000000590b5d438b3ec9b78ffd39b3d26de5e6d8bdbefc0b4f2", pruned: false, initialblockdownload: false }
2020-05-22T13:20:24.659+00:00 - DEBUG - opening DB at "/home/user/db/mainnet"
2020-05-22T13:20:24.699+00:00 - TRACE - latest indexed blockhash: 0000000000000000000000000000000000000000000000000000000000000000
2020-05-22T13:20:24.700+00:00 - INFO - listing block files at "/home/user/.bitcoin/blocks/blk*.dat"
2020-05-22T13:20:24.824+00:00 - INFO - indexing 2084 blk*.dat files
2020-05-22T13:20:24.824+00:00 - DEBUG - found 0 indexed blocks
2020-05-22T13:20:24.830+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:20:32.045+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:20:39.150+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:20:46.194+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:20:53.030+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:20:59.897+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:21:06.658+00:00 - TRACE - downloading 31302 block headers
2020-05-22T13:21:12.990+00:00 - DEBUG - applying 631302 new headers from height 0

The second run of the container/server crashed again:

Config { log: StdErrLog { verbosity: Trace, quiet: false, show_level: true, timestamp: Millisecond, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/home/user/db/mainnet", daemon_dir: "/home/user/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), electrum_rpc_addr: V4(127.0.0.1:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 6, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.3 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
2020-05-22T13:25:43.020+00:00 - DEBUG - Server listening on 127.0.0.1:4224
2020-05-22T13:25:43.029+00:00 - DEBUG - Running accept thread
2020-05-22T13:25:43.034+00:00 - INFO - NetworkInfo { version: 190001, subversion: "/Satoshi:0.19.0.1/", relayfee: 0.00001 }
2020-05-22T13:25:43.037+00:00 - INFO - BlockchainInfo { chain: "main", blocks: 631302, headers: 631302, verificationprogress: 0.999998125560482, bestblockhash: "0000000000000000000d04d8cfffde7c36fa556cbba39f5fc30a9c5acd1facf0", pruned: false, initialblockdownload: false }
2020-05-22T13:25:43.038+00:00 - DEBUG - opening DB at "/home/user/db/mainnet"
2020-05-22T13:25:43.129+00:00 - TRACE - latest indexed blockhash: 0000000000000000000000000000000000000000000000000000000000000000
2020-05-22T13:25:43.130+00:00 - INFO - listing block files at "/home/user/.bitcoin/blocks/blk*.dat"
2020-05-22T13:25:43.344+00:00 - INFO - indexing 2084 blk*.dat files
2020-05-22T13:25:43.344+00:00 - DEBUG - found 0 indexed blocks
2020-05-22T13:25:43.349+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:25:50.065+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:25:56.937+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:26:03.640+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:26:10.437+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:26:17.145+00:00 - TRACE - downloading 100000 block headers
2020-05-22T13:26:23.976+00:00 - TRACE - downloading 31303 block headers
2020-05-22T13:26:30.383+00:00 - DEBUG - applying 631303 new headers from height 0
2020-05-22T13:26:45.211+00:00 - TRACE - indexed "/home/user/.bitcoin/blocks/blk00000.dat": 1712260 rows

The third one is still ongoing:

Config { log: StdErrLog { verbosity: Trace, quiet: false, show_level: true, timestamp: Millisecond, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/home/user/db/mainnet", daemon_dir: "/home/user/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), electrum_rpc_addr: V4(127.0.0.1:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 6, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.3 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
2020-05-22T13:30:33.299+00:00 - DEBUG - Server listening on 127.0.0.1:4224
2020-05-22T13:30:33.310+00:00 - DEBUG - Running accept thread
2020-05-22T13:33:01.200+00:00 - INFO - NetworkInfo { version: 190001, subversion: "/Satoshi:0.19.0.1/", relayfee: 0.00001 }
2020-05-22T13:33:01.248+00:00 - INFO - BlockchainInfo { chain: "main", blocks: 631229, headers: 631301, verificationprogress: 0.9996043629171736, bestblockhash: "00000000000000000011d5c21b0125664be4052e10b95e35ef1bf375244d48a7", pruned: false, initialblockdownload: true }
2020-05-22T13:33:01.249+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631229 progress=0.9996043629171736
2020-05-22T13:35:40.288+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631234 progress=0.9996162949406716
2020-05-22T13:37:02.544+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631236 progress=0.9996182773545936
2020-05-22T13:39:40.836+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631242 progress=0.9996440008299003
2020-05-22T13:42:04.595+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631248 progress=0.999695053282724
2020-05-22T13:42:34.366+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631250 progress=0.9997039560656186
2020-05-22T13:44:38.099+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631255 progress=0.999721728579314
2020-05-22T13:47:38.358+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631261 progress=0.999742907237386
2020-05-22T13:51:46.365+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631267 progress=0.9997655408222448
2020-05-22T13:55:39.895+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631273 progress=0.999807284324652
2020-05-22T13:57:47.871+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631279 progress=0.9998476233506122
2020-05-22T14:00:26.383+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631285 progress=0.9998767606715268
2020-05-22T14:02:52.224+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631291 progress=0.9999025617099012
2020-05-22T14:04:53.545+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631297 progress=0.9999482910386618
2020-05-22T14:05:52.089+00:00 - WARN - wait until IBD is over: headers=631301 blocks=631301 progress=0.9999759380320448
2020-05-22T14:05:56.532+00:00 - DEBUG - opening DB at "/home/user/db/mainnet"
2020-05-22T14:05:57.022+00:00 - TRACE - latest indexed blockhash: 0000000000000000000000000000000000000000000000000000000000000000
2020-05-22T14:05:57.036+00:00 - INFO - listing block files at "/home/user/.bitcoin/blocks/blk*.dat"
2020-05-22T14:05:57.366+00:00 - INFO - indexing 2084 blk*.dat files
2020-05-22T14:05:57.367+00:00 - DEBUG - found 0 indexed blocks
2020-05-22T14:05:57.439+00:00 - TRACE - downloading 100000 block headers
2020-05-22T14:13:00.796+00:00 - TRACE - downloading 100000 block headers
2020-05-22T14:26:52.783+00:00 - TRACE - downloading 100000 block headers

OvoNemaSmisla on 22 May 2020

Are you sure it crashed? There are no crash messages in the log. :thinking:

Kixunil on 22 May 2020

It stopped working, because I'm back at the command line and docker ps doesn't show any active container:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

In the mean time the last run also crashed and here are the last lines from the command line:

2020-05-22T15:05:07.963+00:00 - TRACE - downloading 100000 block headers
2020-05-22T15:17:54.308+00:00 - TRACE - downloading 100000 block headers
2020-05-22T15:18:01.154+00:00 - TRACE - downloading 100000 block headers
2020-05-22T15:18:08.452+00:00 - TRACE - downloading 31302 block headers
2020-05-22T15:18:14.718+00:00 - DEBUG - applying 631302 new headers from height 0

Something similar was happening with ElextrumX container and Luke advised it might be because there was not enough RAM available for the server.
Btw, I'm running this on Odroid N2 4GB RAM and 2TB USB 3.0 HDD.

Is there a way to set a RAM limit for the server? Some argument?

OvoNemaSmisla on 22 May 2020

@OvoNemaSmisla you can see all the available options here: https://github.com/romanz/electrs/blob/master/config_spec.toml

I'm not sure it's crashing due to memory but this option might be worth a try to rule it out:

[[switch]]
name = "jsonrpc_import"
doc = "Use JSONRPC instead of directly importing blk*.dat files. Useful for remote full node or low memory system"

So run with the --jsonrpc-import param (to electrs not docker run).

lukechilds on 23 May 2020

It stopped working, because I'm back at the command line and docker ps doesn't show any active container:

When you say it stopped working, what exactly happened? The log output stopped and it returned to your host shell prompt?

So something like:

$ docker run ...
Config { log: StdErrLog { verbosity: Trace, quiet: false, show_level: true, timestamp: Millisecond, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "/home/user/db/mainnet", daemon_dir: "/home/user/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), electrum_rpc_addr: V4(127.0.0.1:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: false, index_batch_size: 100, bulk_index_threads: 6, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.3 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
2020-05-22T13:20:24.578+00:00 - DEBUG - Server listening on 127.0.0.1:4224
2020-05-22T13:20:24.583+00:00 - DEBUG - Running accept thread
...
2020-05-22T13:21:06.658+00:00 - TRACE - downloading 31302 block headers
2020-05-22T13:21:12.990+00:00 - DEBUG - applying 631302 new headers from height 0
$

Like that?

lukechilds on 23 May 2020

@lukechilds one doesn't have to study config spec. electrs supports --help and has a man page too. :)

Kixunil on 23 May 2020

👍1

When you say it stopped working, what exactly happened? The log output stopped and it returned to your host shell prompt?

Exactly like that and it's definitely memory. Every time it stopped, both memory and cpu peaked. One time even bticoind crashed.

Instead of trying out the --jsonrpc-import and letting Odroid create the indexed db on it's HDD which would take ages, I've ran the container/server on my desktop machine (AMD R5 3600, 16GB RAM, SSD).
I mounted a samba share to the .bitcoin directory on Odroid and used it as the .bitcoin volume, while the other volume was a db directory on SSD.
It took around 2.5 hours to create the indexed db of 61.9GB.
I then copied the db to Odroid HDD and ran the container/server on Odroid with --jsonrpc-import.
So far it is up and running for 11 hours.

I'll try the suggested usage (without --jsonrpc-import), but first I want to make sure it runs fine as it is.
Also, the next step is to set up SSL connection using NGINX as an SSL endpoint following the instructions from the man page.

Thanks both @lukechilds and @Kixunil for helping me out.

OvoNemaSmisla on 24 May 2020

👍3

I've managed to set up SSL connection using NGINX as an SSL endpoint as well as Tor hidden service.

Odroid runs smooth so far:

Aside port forwarding, is there anything else I need to do in order to make Electrs server available for others?

OvoNemaSmisla on 24 May 2020

Great to hear that!

I don't know who you mean by "others". If your family/friends, then everything should be fine. If the whole world, then maybe you want to reconsider that. As far as I know, electrs is more optimized for personal use rather than public use.

Kixunil on 25 May 2020

Yeah, I wanted to open the server to anyone outside my network.
I did the same with ElectrumX server and it worked fine so I though I could do the same with Electrs.

Is the problem to do so related to performance or security?

OvoNemaSmisla on 26 May 2020

electrs was written for the personal use-case (i.e. assuming "non-malicious" users).
It was not meant to handle denial-of-service attacks, and its performance may suffer significantly if there are many concurrent connected clients.
I wouldn't recommend exposing it to the world without any authentication.

romanz on 26 May 2020

👍3

It's worth noting that on the other hand, it' much more performant than ElectrumX for the personal use case. :)

Kixunil on 26 May 2020

👍1

I wouldn't recommend exposing it to the world without any authentication.

Is there a way to enable authentication?

OvoNemaSmisla on 26 May 2020

He means don't make it publicly accessible at all.

So if you want it accessible over the internet, put some auth in front like an SSH tunnel/VPN/Tor etc.

lukechilds on 27 May 2020

👍3

Got it.
I've set up Tor hidden service and disabled port forwarding, so it's accessible outside my network only over Tor.

OvoNemaSmisla on 27 May 2020

🎉1 👍1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

question about store.rs

kenshin-samourai · 4Comments

Would it be possibe to swap rocksdb for somethig else?

Kixunil · 3Comments

Litecoin support

darklow · 8Comments

Crashes on initial sync/index

sangaman · 8Comments

rpc - listen on multiple interfaces

sangaman · 8Comments