Electrs: fatal runtime error: memory allocation failed

[duckartes]

On my Raspberry Pi 3 I'm receiving this error.

fatal runtime error: memory allocation failed
Aborted

sometimes it throws

2018-07-22T12:25:45.772+00:00 - ERROR - server failed: Error: RPC failed: {"id":3,"method":"getbestblockhash","params":[]}
Caused by: disconnected from daemon

but works re-running it.

[greenaddress]

The second error is due to https://github.com/romanz/electrs/issues/18

The first one is just because you have limited RAM on the rpi - for that either you wait for https://github.com/romanz/electrs/issues/30 or consider reducing the ram that core uses.

[duckartes]

How can I limit core? maxmempool and dbcache are set to 50.

[greenaddress]

you could set blocksonly=1

that would remove the mempool entirely until you are fully synchronized. if you are already synchronized then i'm not sure, for now just wait for #30 or maybe look around for other ways to reduce ram usage on the rpi (remove uneeded services)

You could also try to reduce slightly the maximum number of outgoing connections to 5

[duckartes]

From previous tests blocksonly doesn't disable the mempool cache (could have changed in newest versions, or PEBKAC.)

As far as connections I just have a connect= set to sync locally.

Fresh boot takes up less than 40-50M not too much I can shave off I don't think.

[greenaddress]

I guess wait for #30 is the only option. Shouldn't be too hard, means exposing some of the rockdb parameters via command line i think

[romanz]

@duckartes BTW, when did the fatal runtime error: memory allocation failed happen?
Is it during the initial indexing or after it?

[duckartes]

Initial indexing.

[romanz]

Actually, we can slow down the initial indexing to use less memory -> #40

[romanz]

@duckartes Please try again with the latest commit (https://github.com/romanz/electrs/commit/2f9b529da89ddf2a379ba48149980eb89de25308) using the new --skip-bulk-import flag.

P.S.
Using this flag, electrs doesn't require access to blk*.dat files - only to bitcoind JSONRPC :)

[greenaddress]

@romanz nice!

[duckartes]

Still run into it :(

[romanz]

When did the crash happen?
Could you please attach the log?

If it's still happening during the initial indexing, I'd suggest running the initial indexing on a machine (with 2GB+ RAM), and then copying the DB directory to Raspberry Pi's storage.

[romanz]

Meanwhile, I'll work on #30.

[romanz]

@duckartes Please try again with https://github.com/romanz/electrs/commit/fcb926a8adc36c5ece9951c360f19fdf96763606 (it should reduce the memory requirements - but would take more time to finish the initial indexing).

[duckartes]

Letting it run again and I'll post a log.

I want to add I haven't even been able to fully sync on my main rig.

Plenty of processing power. Dies on RPC error, 74273 left to index is the farthest I've gotten so far.

Always have to restart. Perhaps a retry loop X number of times / or indefinite?

EDIT: It died after posting this on an RPC error but went up to 65576 left to index.

[duckartes]

Last log on my Pi:

https://pastebin.com/tjDXQteB

[romanz]

Always have to restart. Perhaps a retry loop X number of times / or indefinite?

This probably is the best solution for disconnection errors -> #18

https://pastebin.com/tjDXQteB

First of all - how did you get the detailed RocksDB log?
It is very useful for performance analysis :)

Did the Pi fail due to an out-of-memory error or just disconnected from bitcoind?

[duckartes]

Disconnected.

I run: cargo run --release -- -vvvv --timestamp --skip-bulk-import --db-dir /mnt/storage/.pin0de/electrs/ --daemon-dir /mnt/storage/.pin0de/bitcoin/

Log is from: /mnt/storage/.pin0de/electrs/mainnet/LOG

(May have too many v? lol, but I thought I saw it listed as a command somewhere here for the most verbose, not shown in --help)

[romanz]

Disconnected.

Well, at least the OOM errors are gone :)
It should be OK to wrap the electrs in a "loop-forever" bash script for rerunning automatically.

Log is from: /mnt/storage/.pin0de/electrs/mainnet/LOG

Thanks, TIL :)

Regarding the -vvvv: it's currently the highest verbosity setting, but it's safe to pass more.

[romanz]

42 should help with the disconnections handling :)

[romanz]

@duckartes Could you please test on Raspberry Pi again with the latest version?
(--skip-bulk-import should reduce the RAM usage and #42 should reconnect automatically during initiali indexing)

[duckartes]

Just got some time to start from scratch last night.

I eventually got it to sync fully by nudging it along the way over and over again, but starting from scratch last, I still get the "fatal runtime error: memory allocation failed." (which aborts the run.sh script)

Could you suggest a temporary file host so I could upload the LOG file?

[duckartes]

Found the suggestion from a previous post:

https://expirebox.com/download/5c45bc3d1127b053907cadc259da7eff.html

LOG from a fresh start.

[romanz]

Thanks a lot for the testing!
Downloaded the file - will investigate today/tomorrow...

$ sha256sum ~/Downloads/LOG
b823d5f700ce008bb94d023499020181e6590396b8df52310c4643241d07d148  /home/roman/Downloads/LOG

[duckartes]

Electrs may have not been absolutely up-to-date.

I pulled and rebuilt today and it's been going ever since. Sitting at about 18G worth of work so far, we'll see what happens.

Love how fast it indexes.

[romanz]

Love how fast it indexes.

Thanks :)

[romanz]

Could you please run it with the latest scripts/run.sh?
In case of a panic, it should print a detailed backtrace :)

[duckartes]

I'll have to figure out how to add the fmt option.

I just used: curl https://sh.rustup.rs -sSf | sh from https://rustup.rs/ with a default install on my system.

[duckartes]

Actually just found it.

I moved to nightly so the command to get fmt seems to be: rustup component add rustfmt-preview --toolchain nightly

https://github.com/rust-lang-nursery/rustfmt

[duckartes]

On 26G now with no memory error. Gonna let it finish and restart making sure everything is up to date.

[duckartes]

Yep, around 13-14G worth of indexing it seems to fail.

https://expirebox.com/download/6c38f3430dd09f7a15ac76cade31890e.html

[romanz]

Could you please also attach the backtrace?

[duckartes]

Where does it save that?

[romanz]

Currently, electrs prints it to stderr - so it may be lost :/
-> #45

[duckartes]

Word. Just keep updating and I'll keep testing :P

[romanz]

stderr is now persisted to /tmp/electrs.log at https://github.com/romanz/electrs/commit/87b4df57472ad6fdb1b1f09cff0dbd03c29b822d (when run via ./scripts/run.sh).

[romanz]

Could you please also track electrs memory usage during the initial indexing (using the following command)?

top -d1 -b -p `pgrep electrs` | ts | grep electrs

[romanz]

Also, how much free memory does your system have for electrs (could you please run the following commands and attach the results)?

$ sudo lshw -c memory
$ free -h

Are you are also running bitcoind on the same Raspberry Pi?

[romanz]

I'll try to reproduce the OOM errors on my machine, given the answers to the questions above :)

[duckartes]

What package / tool is ts apart of?

bash: ts: command not found``

sudo lshw -c memory
  *-memory                  
       description: System memory
       physical id: 4
       size: 976MiB

free -h
              total        used        free      shared  buff/cache   available
Mem:          976Mi       489Mi       122Mi       1.0Mi       364Mi       432Mi
Swap:         1.9Gi       112Mi       1.8Gi

Yes, The bitcoind is on the Pi.

No issues running on my main rig now, can sync fully no issues.

[romanz]

ts is part of the moreutils package.

[romanz]

So I guess that the target memory usage will be <500MB for the initial indexing phase.

[romanz]

Thanks a lot @duckartes !
I'll track further optimizations at https://github.com/romanz/electrs/issues/30.

[romanz]

@duckartes Please try https://github.com/romanz/electrs/commit/9dcec984ead3847c60cde1c7d46f44e70419e7ab - it should take <550MB and run for ~4.5 hours (on my x86 machine):

$ git fetch origin
$ git checkout low-mem
$ cargo run --release -- -vvvv --timestamp --network=mainnet \
                         --index-batch-size=10 --skip-bulk-import

[romanz]

Rebased low-mem branch on the latest master -> 875f11b846d78b6b84900849595a1f2e9e678a59.
@duckartes please try to run this version, with:

$ cargo run --release -- -vvvv --timestamp --network=mainnet \
                         --index-batch-size=10 --jsonrpc-import

Note: I've renamed --skip-bulk-import to --jsonrpc-import.

[duckartes]

Did:

$ git fetch origin

$ git checkout low-mem

Ran:

$CARGO run --release -- -vvvv --index-batch-size=10 --jsonrpc-import --timestamp --db-dir=/mnt/storage/.pin0de/electrs/ $* 2>> "$LOG"

du -h /mnt/storage/.pin0de/electrs/

56G /mnt/storage/.pin0de/electrs/mainnet
56G /mnt/storage/.pin0de/electrs/

Currently.

No failures as of yet though.

[duckartes]

du -h /mnt/storage/.pin0de/electrs/
38G /mnt/storage/.pin0de/electrs/mainnet
38G /mnt/storage/.pin0de/electrs/

Looks like it's fully synced and no errors.

I'll restart from scratch in another day or two and verify that it works completely.

Awesome!

[romanz]

Great news!
Many thanks for the testing - and please let me know how the verification goes so I can publish a new version on https://crates.io :)

[duckartes]

Took about 2ish days but no errors. Synced fully.

I limit my bitcoin.conf to:

dbcache=50
maxmempool=50

to reduce memory usage along with the settings from https://github.com/romanz/electrs/issues/38#issuecomment-419201852

[romanz]

Cool :)
Will merge soon, and release it as 0.4.0 version.

[romanz]

Merged to master at #58

[romanz]

Released at v0.4.0.