Sled: disk space blowup

[spacejam]

https://github.com/Kerollmops/sled-disk-load/blob/master/src/main.rs

Writing 5kb values over and over for 3000 keys causes a space leak to pop out when using a single thread. This might be because of no longer explicitly flushing the local crossbeam epoch guard after each high level operation, causing segments not to be released for freeing as quickly.

[Kerollmops]

Notice that when every insert call is immediately flushed, or each 100 inserts, I can see the database shrink.

[spacejam]

When I run this with a single thread and set it to burn-in constantly, it goes up and down but tends to be stable without going above a couple gb, dipping down to a few hundred mb. I see rapid blowup issues when multiple threads are involved, even when flushing EBR guards, making me think it might be an issue with EBR not being able to release epochs. Going to talk with Stjepan about good ways of introspecting this and possibly tuning crossbeam-epoch in sled

[spacejam]

fixed on master

[Kerollmops]

Do you plan to publish a new version on crates.io with this fix?

[spacejam]

@Kerollmops I've cut a release candidate, but it's still burning in and I believe I've found an issue I need to address before a full release which happens after several hours of intense, highly-contended writes - but if you'd like to try out the release candidate I'd love to know how it works for you!

[Kerollmops]

I tested it a little bit but I encountered important RAM consumptions, I then tried the same program with RocksDB and I encountered data corruption 🤦

So I backup to LMDB with heed, one thing that I wanted to try is using the merge operators to enable parallel computation. I used my little MTBL Rust portage not easy to use but it work really well.