Sled: Optimal value size

[brechtcs]

I'm wondering whether there's already anything you can say about the optimal size for values stored in sled? I know at this stage any statements regarding performance will be mostly theoretical, but it would be interesting if you already had an idea whether sled will work best for small key-value pairs or whether it will be able to efficiently deal with large chunk of data as well.

Either way, I'm definitely looking forward to having a stable embedded database available in Rust soon. Keep up the good work! :)

[spacejam]

@brechtpm thanks for the kind words :)

There is currently an io buffer that all updates must pass into before being written sequentially into the log. I tend to keep it configured around 8mb in my tests. But, off the bat, you can't store anything larger than that in the system right now.

This buffer size maps 1:1 to the concept of a "segment" which is sort of like a mix between a generational GC and a slab allocator. Too large, and it may increase the overhead of managing the segment, but to be honest I have not measured workloads configured with very large segment sizes, and it will be interesting how large this can go before it causes issues.

Because the tree is logically similar to a B+ tree, the naive approach (currently implemented) stores the entire key on each page that references it, so if you have a deep tree and huge keys, you get a little write amplification for the time being.

In the future, large keys will probably be addressed with configurable prefix encoding. Large values could be addressed by punting large writes to the filesystem, which is pretty good at being a blob store, but this is purely theoretical.

All this is to say that I want this to work well with a wide variety of workloads, but we don't have things like prefix encoding on keys or vfs punting for big values yet. I want to implement these things, and if I do, I want to do so in a configurable way that does not over-restrict the use cases for the system.

In general, I would love to hear about certain workloads that you need to run on embedded databases, so that I can turn these into benchmarks that I pay attention to.

[Ralith]

I'm experimenting with content-addressed storage of dynamically inserted/deleted game assets, which have widely varying sizes, from a handful of bytes (metadata) to tens or even hundreds of MB (textures, geometry, video, ...). I can implement fallback to the filesystem myself easily enough, but it would be convenient if I could use sled and have it Just Work for everything in a nice uniform interface.

[Ralith]

Any progress on lifting the restriction on value sizes?

[spacejam]

@Ralith yeah, large values are now stored in their own files, off-log, so the limits are pushed to the filesystem now