Electrs: Indexing errors

[Emanresueht]

Hi There,

Was running the indexing twice with different options in the command. Do you know what the issue might be?

debianvm1@DebianVM:~/electrs$ cargo run --release -- -vvvv --index-batch-size=10 --jsonrpc-import --db-dir ./db --electrum-rpc-addr="127.0.0.1:50001"
    Finished release [optimized] target(s) in 7.36s
     Running `target/release/electrs -vvvv --index-batch-size=10 --jsonrpc-import --db-dir ./db '--electrum-rpc-addr=127.0.0.1:50001'`
Config { log: StdErrLog { verbosity: Trace, quiet: false, timestamp: Off, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "./db/mainnet", daemon_dir: "/home/debianvm1/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), electrum_rpc_addr: V4(127.0.0.1:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: true, index_batch_size: 10, bulk_index_threads: 8, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.4 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
DEBUG - Server listening on 127.0.0.1:4224
DEBUG - Running accept thread
INFO - NetworkInfo { version: 190100, subversion: "/Satoshi:0.19.1/", relayfee: 0.00001 }
INFO - BlockchainInfo { chain: "main", blocks: 636804, headers: 636804, verificationprogress: 0.9999993185142182, bestblockhash: "0000000000000000000f0f3bdbbe4c68ddc37222d0e4e7054d89eb0b481b58ad", pruned: false, initialblockdownload: false }
DEBUG - opening DB at "./db/mainnet"
TRACE - latest indexed blockhash: 0000000000000000000000000000000000000000000000000000000000000000
TRACE - downloading 100000 block headers
TRACE - downloading 100000 block headers
TRACE - downloading 100000 block headers
TRACE - downloading 100000 block headers
TRACE - downloading 100000 block headers
TRACE - downloading 100000 block headers
TRACE - downloading 36805 block headers
INFO - best=0000000000000000000f0f3bdbbe4c68ddc37222d0e4e7054d89eb0b481b58ad height=636804 @ 2020-06-29T00:42:03Z (636805 left to index)
TRACE - closing DB at "./db/mainnet"
ERROR - server failed: Error: getblock RPC error: {"code":-1,"message":"Block not found on disk"}

debianvm1@DebianVM:~/electrs$
-------------------------------------------end-first-----------------------------------------------------------------------

debianvm1@DebianVM:~/electrs$ cargo run --release -- -vvvv --index-batch-size=10 --jsonrpc-import --timestamp --db-dir ./db --electrum-rpc-addr="127.0.0.1:50001"
    Finished release [optimized] target(s) in 2.23s
     Running `target/release/electrs -vvvv --index-batch-size=10 --jsonrpc-import --timestamp --db-dir ./db '--electrum-rpc-addr=127.0.0.1:50001'`
Config { log: StdErrLog { verbosity: Trace, quiet: false, timestamp: Millisecond, modules: [], writer: "stderr", color_choice: Auto }, network_type: bitcoin, db_path: "./db/mainnet", daemon_dir: "/home/debianvm1/.bitcoin", daemon_rpc_addr: V4(127.0.0.1:8332), electrum_rpc_addr: V4(127.0.0.1:50001), monitoring_addr: V4(127.0.0.1:4224), jsonrpc_import: true, index_batch_size: 10, bulk_index_threads: 8, tx_cache_size: 10485760, txid_limit: 100, server_banner: "Welcome to electrs 0.8.4 (Electrum Rust Server)!", blocktxids_cache_size: 10485760 }
2020-06-29T13:19:37.603+01:00 - DEBUG - Server listening on 127.0.0.1:4224
2020-06-29T13:19:37.603+01:00 - DEBUG - Running accept thread
2020-06-29T13:19:37.624+01:00 - INFO - NetworkInfo { version: 190100, subversion: "/Satoshi:0.19.1/", relayfee: 0.00001 }
2020-06-29T13:19:37.625+01:00 - INFO - BlockchainInfo { chain: "main", blocks: 636864, headers: 636864, verificationprogress: 0.9999951889853614, bestblockhash: "0000000000000000001163e7b8547c60c3a357a611260c3737f1afdd08cc17e7", pruned: false, initialblockdownload: false }
2020-06-29T13:19:37.626+01:00 - DEBUG - opening DB at "./db/mainnet"
2020-06-29T13:19:38.123+01:00 - TRACE - latest indexed blockhash: 000000000000000009898b42978e4d4381da0d0db08ecdfd356f9c2b356c7722
2020-06-29T13:19:43.150+01:00 - DEBUG - applying 365850 new headers from height 0
2020-06-29T13:19:43.356+01:00 - DEBUG - downloading new block headers (365850 already indexed) from 0000000000000000001163e7b8547c60c3a357a611260c3737f1afdd08cc17e7
2020-06-29T13:20:51.153+01:00 - TRACE - downloaded 271015 block headers
2020-06-29T13:20:51.574+01:00 - INFO - best=0000000000000000001163e7b8547c60c3a357a611260c3737f1afdd08cc17e7 height=636864 @ 2020-06-29T12:08:05Z (271015 left to index)
2020-06-29T13:20:52.041+01:00 - TRACE - closing DB at "./db/mainnet"
2020-06-29T13:20:52.045+01:00 - ERROR - server failed: Error: getblock RPC error: {"code":-1,"message":"Block not found on disk"}

debianvm1@DebianVM:~/electrs$
----------------------------------------------------------------end-second---------------------------------------------------

Thanks

[romanz]

Thanks for reporting the issue!
It seems that some of the blocks are missing from the "blk*.dat" files.
Could you please attach the result of:

$ ls -l /home/debianvm1/.bitcoin/blocks/blk*.dat

[Emanresueht]

Here the output into a txt: https://ufile.io/8wct3izt

[Kixunil]

This usually means corrupt data. Maybe because the node crashed, power went down... Unfortunately, you will have to run bitcoind with -reindex and waiting for re-sync

[romanz]

I agree with @Kixunil - it seems that raw block data got corrupted (it fails during Block deserialization).

[romanz]

Did you experience other data corruption/loss on this file system?

[Emanresueht]

Thanks @romanz and @Kixunil so far. Yes, my node went down once whilst running initial -txindex but after restart all looked good. Also btc-rpc-explorer (https://github.com/janoside/btc-rpc-explorer) was running without issues so did not look deeper into anything. Running -reindex right now and will come back with results.

[Emanresueht]

Hey @romanz and @Kixunil, its done, all works, thanks a lot guys! Reindexing the node took ages but it was worth it:) After that electrs ran without any issues.

Cheers!

[romanz]

Cool, glad to hear :)