Sled: Question

[goldenMetteyya]

What is the difference here of handling files your self versus using mmap?

[spacejam]

Hey! Good question :) mmap is a great choice for most stateful systems, because it can really simplify the handling of data in files. The main reason why I chose not to use it is to simplify the path to using O_DIRECT, which tells the kernel not to use any memory to cache data that is read or written to disk. Very often, you benefit a lot from the kernel doing this caching for you. But with this database, we are doing the caching ourselves, and there are a few reasons that our cache can outperform the kernel eventually. Note that it's not there yet, and it takes real work to beat the kernel's cache, but we haven't switched to O_DIRECT yet so we are still taking advantage of it for a small set of cases where it helps to have the disk data cached.

Here are two reasons why userspace caching benefits this database:

1. compression. the kernel's cache will store the compressed version of our data, and if we enabled compression, we still have to decompress the data for it to be useful to our application. in our cache, we store the decompressed version of the data, to prevent having to use CPU every time we access the data
2. serialization. the kernel's cache will store the serialized version of data, just like we wrote it. we would have to create a more low-level and probably buggy data access mechanism if we wanted to avoid deserialization costs on every access

It's also tricky to have multiple threads that write data in mmap databases sometimes. But the main reason I'm not using it is because you can't use it with O_DIRECT, which lets me just do the caching of the deserialized, decompressed objects myself, and use way less disk space.

I should document these decisions better! Thank you for asking about this important trade-off 👍

PS this blog post series talks a lot about choices around using or not using mmap, I thought it was pretty fun to read, and maybe you would like it: https://medium.com/@ifesdjeen/on-disk-io-part-1-flavours-of-io-8e1ace1de017

[goldenMetteyya]

Hello! Thanks for your response:) I think that you have made the right trade off here for a longer term performance direction and I like it. I have been looking into the project to help out where I can:) Those series of posts looks very good I will read through them:) Also is O_direct only a unix thing or are you aware of a windows counterpart so it may help to port sled ?

[spacejam]

I'd love to work together :) At the moment, things are a bit chaotic after a huge storage refactor, and the dust is still settling. After it does, I'm going to write a bunch of docs that will help prospective contributors get more familiar with the codebase, do some much-needed refactoring to ease the learning process, and do a more thorough roadmap that shows how folks may be able to help out. I apologize for the current state of things, as it may be a bit contributor hostile for the next week or two. There are so many directions this project could go in, and I need to focus on making sure the core is as reliable and high performance as possible, so I would like to mentor folks and give them ownership over higher-level parts of the system that they are interested in. Stay tuned!

O_DIRECT is available on a few OS's, including linux, FreeBSD, NetBSD, and anything that is fully POSIX.1-2008 compliant, I think. Its implementation varies slightly across systems. For OSX, we can use the F_NOCACHE flag to fnctl, and on windows we can use the FILE_FLAG_NO_BUFFERING flag for a similar result.