Sled: Empty database load time/file size

[rubdos]

Hi!

I am very new to this library (as is... everyone, I think? Since it's new...), and I really like the interface.
I integrated it a bit with an application I am writing (which is a lot more alpha than sled, no worries there). The ConfigBuilder::new().temporary(true) databases, which I use in my (unit and integration) tests work very well.

Now, I am trying to create an on-disk (SSD) database, and that seems to take a while:

$ time my_app
my_app  1.90s user 2.39s system 81% cpu 5.301 total

Especially since this is actually just empty!

    use sled::ConfigBuilder;
    let cache_path = [...];
    let config = ConfigBuilder::new()
        .path(cache_path)
        .cache_capacity(10_000_000_000)
        .use_compression(true)
        .flush_every_ms(Some(1000))
        .build();

In addition, the file size is 8MB per these settings (which is okay for me, but could be an indicator for you?)

I'm on sled 0.15.2.

[spacejam]

Hey, @rubdos ! The 8mb part is OK, because the database preallocates 8mb segments of disk space on startup. This can be turned down by changing the config setting io_buf_size if you want it to use less. But be careful about setting it too low, because when values are larger than 1/4 (also tunable by min_items_per_segment) of the size of a segment, it will cause a write to fail currently, until we implement off-page storage for large values.

The more concerning thing to me is the startup time for an empty database, which should not be so long. I'll drill into this a bit more.

[rubdos]

The 8mb part is OK,

Yap, for me too atm.

If I ever make an embedded version, I'll keep the 1/4th rule in mind.
This also means that writing values > 2MB can fail to write concurrently with default settings, right?

[spacejam]

@rubdos yeah, exactly. The short-term idea is to use something like http://persy.rs/ to handle off-page blobs, but this is not implemented yet.

[spacejam]

OK I've finally had some time to pop open the flamegraphs on this, and I've identified 3 optimizations that should make recovery quite speedy once complete:

1. don't ever read one byte at a time to determine how long a failed flush is (this is where almost all of the time is spent during recovery of a mostly-empty segment
2. don't ever read one byte at a time when we hit EVIL_BYTE, just instantly return a Corrupted read to signal that we've outrun our log
3. read entire segments at a time. this is less urgent, but in the long run will be a nice optimization for speeding up recovery

[spacejam]

@rubdos I've released sled 0.15.6 which includes above optimizations 1 and 2. I've measured this to greatly reduce the startup hiccups. Please let me know if this fails to address your issue, as this is an issue that needs to be addressed.

[rubdos]

I seem to have a backwards compatibility issue now:

thread 'main' panicked at 'assertion failed: buf.len() >= 1', /home/rsmet/.cargo/registry/src/github.com-1ecc6299db9ec823/sled-0.15.6/src/tree/prefix.rs:25:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at libstd/sys_common/backtrace.rs:59
             at libstd/panicking.rs:206
   3: std::panicking::default_hook
             at libstd/panicking.rs:222
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:400
   5: std::panicking::begin_panic
             at /checkout/src/libstd/panicking.rs:363
   6: sled::tree::prefix::prefix_decode
             at /home/rsmet/.cargo/registry/src/github.com-1ecc6299db9ec823/sled-0.15.6/src/tree/prefix.rs:25
   7: sled::tree::tree::Tree::path_for_key
             at /home/rsmet/.cargo/registry/src/github.com-1ecc6299db9ec823/sled-0.15.6/src/tree/tree.rs:671
   8: sled::tree::tree::Tree::get_internal
             at /home/rsmet/.cargo/registry/src/github.com-1ecc6299db9ec823/sled-0.15.6/src/tree/tree.rs:560
   9: sled::tree::tree::Tree::get
             at /home/rsmet/.cargo/registry/src/github.com-1ecc6299db9ec823/sled-0.15.6/src/tree/tree.rs:135

Not that my data mattered, but I might as well say so. Startup time is instantaneous now. Thanks :-)

[spacejam]

Awesome, glad to hear it's fixed for you!

Yeah, backwards compatibility is absolutely a requirement for future stable releases. we are doing some funky things on the storage side of things that have yet to be formally verified, and occasionally bugs are still popping out around correct recovery in practice, so it's not clear how much more the storage format will need to evolve before 1.0. But before then, backwards compatibility is not on the table because it would slow down development efforts.

Thanks for opening this issue, and I'm eager to hear if you hit any snags in the future :)

[rubdos]

But before then, backwards compatibility is not on the table because it would slow down development efforts.

It is not for my application either. A lot of data structures that are meaningless at the moment, and such. So I don't care yet.

Thanks for opening this issue, and I'm eager to hear if you hit any snags in the future :)

I will. I'm gonna stress test this thing (through my application), so I'll certainly let you know ;-)

I also have snapper on my home directory, so I actually keep every version of the database for a week or two. If you ever need something from my archive, let me know. Always small files here.