Electrs: question about store.rs

[kenshin-samourai]

hey

i apologize in advance for this stupid question but i don't understand the purpose of this line
https://github.com/romanz/electrs/blob/master/src/store.rs#L77

do we try to set self.opts.bulk_import?

[romanz]

Thanks for the question!
Actually, it's indeed not clear enough from the code - so I'll try to explain:

bulk_import is a flag for choosing the initial block import mode.
If it's true, we read the blk*.dat files and write them into the DB (with some options being different for performance reasons - e.g. no flushing to disk and disabled compaction).
After we finish the initial block import, we flip it to false, flush the writes and compact the DB:
https://github.com/romanz/electrs/blob/1ca25531b7d74d7bcd7f5cbd9d0bcaf763474220/src/store.rs#L184-L188

This mode is used at the first time electrs runs, so in the next runs it should use regular block import (via bitcoind JSONRPC interface).

Hope this helps!

[kenshin-samourai]

thanks for this explanation @romanz. the rationale makes sense.

if my understanding is correct:

• after the initial block import, writes should be flushed to disk automatically (https://github.com/romanz/electrs/blob/master/src/store.rs#L157)
• when electrs is launched, the dbstore object is always initialized with bulk_import set to true (https://github.com/romanz/electrs/blob/master/src/store.rs#L69).
  this value should be set to false when enable_compaction() is called.

i'm not not a rust expert (far from that actually) but my understanding is that line 77 sets bulk_import to false on a clone of self.opts and not on self.opts itself. thus, flushes to disk are never activated in write(). do i miss something about cloned objects in rust?

i've made some tests on an instance with jsonrpc_import set to true and the compaction flag stored in database, At each startup, electrs processes again all the blocks created after the compaction flag was stored in database, as if they were never written on disk.

edit:

i think that i was able to solve my issue by modifying the flush() method.

the new code is:

fn flush(&self) {
    self.db.flush().unwrap();
}

it seems that previous code wasn't flushing the memtables to disk.

[romanz]

Many thanks for finding this!
I think it may be the cause for https://github.com/romanz/electrs/issues/249.

[kenshin-samourai]

good catch! thanks for you work. electrs is a great project!