Sled: Database size, memory and performance issues when used with 'criner'

[Byron]

This is not a bug report, but something I would like to share so sled can get better for my use case.

Context

For context, criner is a tool to mine crates.io. For that, it uses sled (v0.31.0) as sole source of truth. Criner does not keep cashes of any kind, but uses Sled to access all data knowing it's fast and save.

When running 'criner', it will store ~210000 crate versions, ~36000 crates, as well as ~210000 tasks and up to ~210000 downloaded crates.

All objects are encoded with rust messagepack, except for downloads which are data blobs as downloaded from crates.io.

Sizes on disk after having downloaded ~100000 crates is as follows

  […]
  16.09 GB assets      # this is a separate file-based copy of all downloads
  27.26 GB criner.db. # the DB with all objects, including downloads

Motivation

I would like to use sled for all data, including downloaded files, as I would hope it performs better than using files on disk. Criner should run on raspberry-pi class computers with 512MB of RAM and possibly externally attached SSD drives.

Sled seems to tick all the boxes, and on top of that is very pleasant to work with, thus I want to keep using it. Hence this post to learn more and get a shared understanding on how sled should be used.

I am absolutely motivated to involve myself with Sled because I want it to be the awesome store behind 'criner' running on tiny and inexpensive devices, doing the work of big servers :).

Observations

When running criner over night for the first time, the program was 'not running' in the morning after just ~66000 crates downloaded. There are no logs (_which I am going to fix_), and there was no panic visible. As Criner is running forever, I have no explanation for this, yet, but suspect an OOM kill.

When starting criner again, there was a noticeable delay till the TUI was displayed, It took enough time to open the activity monitor to see that criner was reading at speed of 1.3GB/s, and it was doing its fair share of writes as well.

The memory consumption peaked at about 6GB.

When the GUI showed up, criner started to download crates once again, and there was still 100's of MB of writes per second, even thought the downloads per second can't exceed 7MB/s.

After letting it run for ~2h without issues…

…it is now stable at around 5GB of memory…

…and the following disk stats:

Concerns/Confusion

What I am unclear about is whether the amount of memory used could be an issue on smaller devices. I am aware that this might be memory-mapped memory entirely, and not 'real' memory, and don't think I know what I am doing when looking at the memory statistics.

Furthermore, I hope the startup delay can be avoided somehow, even though it's minor given that criner would run forever usually.

I would be glad to learn what you think about this.

Versions

• sled 0.31.0
• OS MacOS
• rustc v1.41.0

How to reproduce

git clone https://github.com/Byron/crates-io-cli-rs cio
cd cio
git checkout 755b764a49d9532b983ed3bd1920e59cd17e907d
make  tests/fixtures/index-bare
cargo run --release -- --downloads-directory ./assets  mine --concurrent-downloads 25 -r tests/fixtures/index-bare criner.db/

This will run criner with GUI. For logs, only, use the --no-gui flag and turn on logging.

RUST_LOG=info cargo run --release -- --downloads-directory ./assets  mine --no-gui --concurrent-downloads 25 -r tests/fixtures/index-bare criner.db/

[Byron]

There might be a memory leak - after three hours, it's up to 7GB.

It's entirely unclear if this is indeed related to Sled, or related to reqwest, which also sees all downloaded data. For now I hold back, but it might be worth cutting out reqwest and just generating a workload similar to what criner currently does.

[Byron]

Just for completeness, it seems to be down again. I will also stop posting these updates, and hope that what's here is useful.

[spacejam]

Hey @Byron, thanks for the detailed reports! I think you're running into at least 2 issues. This info is really great for me to see because it incentivizes me to spend some effort over the next week or so on memory and recovery issues, which I believe will result in some nice improvements for your workload :)

1. One thing that sled has some outstanding issues around is keeping the cache size down. as it's currently using the size on disk to estimate the size in memory, which is really fast and cheap for the system but quite a poor metric. The alternative approach is to recursively find out how large all of the sizes involved in storing a particular tree node are, which is exact but very expensive. Maybe a nice solution is to do the expensive calculation every 10,000 requests or so, and add 2 atomic numbers: log size and real in-memory size. Then when calculating cache size for eviction purposes, we can just read those 2 atomic numbers and figure out the average ratio, and keep using the cheap metric. Perhaps a better solution is to use a compact key representation, maybe even the fst crate which can work off of a Vec and the in-memory size does not require hardly any computation to figure out.
2. sled is quite slow during recovery because it uses a pretty naive, non-parallelized recovery algorithm on 0.31. The solution is planned out and fairly clear, but most people aren't running sled for large data sets yet, so this has not yet bitten many folks and I haven't made it a huge priority. I actually just thought of an even better solution involving "fuzzy checkpointing" where periodically a background thread will read the current stable logged offset, then traverse the page table without locking anything and build a map of data (this is the same map that is used on recovery) and then on startup we only look at the file segments that are later than that "starting checkpoint location" and use several threads to merge new page data. This can function in a very parallelizable way, as a last-write-wins CRDT where the latest version of a page recovered wins.

Now that more and more things in sled are getting to be stabilized, it feels like the number of usecases that sled doesn't suck for is going down, but it's definitely got some rough edges still, as you're experiencing. The worst things I'm currently aware of relate to disk usage, memory usage, and recovery times. The current master branch has some improvements for disk and memory usage (disk usage then indirectly causes recovery time to speed up, but the core algorithm has not changed yet) and I would be curious to know if the same workload fares any better, but I can try it out on my own system to get a sense of progress.

My motivation comes from the kinds of issues people open, and this is quite a nice one! Thank you again, and I'll post updates in here as I hack on recovery, memory usage, and disk usage more.

[Byron]

Thanks so much for sharing your thoughts :)! I am totally keen to try the latest greatest, and will switch to sled-master to always get the bleeding edge.
As I also expect to run a database migration at some point, I realized that all data is going to be held in memory for a moment. That will be increasingly harder, if not impossible, if I keep dumping 'big' data into sled.
In order to avoid that from happening, I will migrate my data into a form that stores binary blobs on disk (only downloads for now), and keeps all meta-data in sled. This means a lot of small objects, and probably equates to a nice data set for you to test with.
What's even better about this approach is that I should be able to share my dataset with you much more easily, so you don't have to download crates.io yourself all the time :D.

Looking forward to seeing how all this develops (and in a few months, I should be ready to put criner on a Raspberry Pi 3, so stay tuned, challenges will keep coming :D).

[Byron]

Correction: I just saw that the export/import functionality is implemented using an iterator, so I presume memory wouldn't be an issue. For now I will migrate away from storing blobs, but am absolutely willing to give it another shot some time later.

Generally I think using files for blobs is beneficial, as using the filesystem will be something I don't get around to doing, and that is fully async these days which is really good for my workload which is likely to be IO bound for report generation.

[Byron]

Here is my experience report :).

• When using the master for migration, it would not open the database, and provided a very nice and polite error message. I will run the sled migration next to get to master.
• when dropping the tree with ~160000 downloads, it consumed about 45GB of real, non-virtual memory. I am surprised my machine managed to do that, given it _only_ has 16 GB of RAM 😅. I was holding my breath…
  
• It seems to not have released any disk space (please note that I moved the assets directory into the criner.db)
  
  
  
  
  
  • I will run a normal export/import cycle once to see how it fares, and hope I will get below 1Gb
  
  
• opening the database is quick now, that's nice.

[Byron]

And here the final update. The migration worked without any hitches and with low memory footprint (but please note that the big tree was dropped before).

Old (without assets) vs New (with assets):

Now the sled database weighs just about 630MB, and probably has around 650k smaller objects.
Thus far, everything works alright and I am tracking master.

[Byron]

As an update, I noticed that now I am in a state where opening the database alone takes about a minute (didn't measure it precisely) on a 2017 MacBook 13", while reading seemingly half the database once. The DB itself is about 11GB is size, and it reads nearly 6GB.

[Byron]

Please feel free to close this issue if it's not substantial enough for being fixed directly. Thanks for all your work on Sled, please keep going and make it the go-to database for Rust projects!

[spacejam]

fixed on master