Sled: Question: Is Sled multi-process safe?

[ckaran]

I have access to a cluster system, but the nodes on the cluster can only synchronize through the filesystem. flock does work on the system, so I've been using it to ensure that any shared files aren't corrupted, and I could continue to do the same when using sled, but if sled is already multi-process safe, then there's no need for all this code (and it would likely slow sled down a lot). So, is sled multi-process safe?

[spacejam]

sled uses advisory locks, but I haven't subjected them to particularly intense concurrency checking, and I know that in general this is an aspect of filesystems that sometimes is buggy. Importantly, you have to start sled any time you want to use it. It's not possible as of now for multiple processes to share the same open file, and data loss will happen if you do something that lets multiple sled processes work on the same file while bypassing the internal locking logic somehow.

[ckaran]

Got it, thank you! And I think you've managed to address a separate question that's bugged me on and off for years, that is 'why do databases insist on using sockets when the filesystem could be faster'. Now I know!

[itamarst]

I gather from the comment above that the answer is "no, it's not multi-process safe". I had the same question and couldn't find the answer in the documentation, so it might be nice to document this.

[visig9]

I think the answer is "sled cannot run in multi-process situation, but If the OS / FS support advisory lock, and implement correctly, datafile would not be corrupted". So it's safe when non-intended run multiple process in the sametime. If I'm wrong please correct me.

Check here:

https://github.com/spacejam/sled/blob/v0.34.6/src/config.rs#L580-L608

[ckaran]

@visig9 I would be very careful in relying on this, because as @spacejam said earlier, "...I haven't subjected them to particularly intense concurrency checking, and I know that in general this is an aspect of filesystems that sometimes is buggy." I know that one of the goals of sled is to not wake up operators in the middle of the night, which means a lot of effort has been put into making it operate correctly if you use it as intended. Sled is not intended to be run in the way that this question originally asked (two wholly separate processes opening up the same database at the same time), so there is less testing being done along this path.

Put another way, assume that flock works perfectly on the filesystem that you're using, and so you design your code around this assumption. Then an OS upgrade comes along that introduces a bug into flock, or your code is ported to a different OS/filesystem that runs sled but has a buggy flock. Sled isn't being tested aggressively against flock, so no-one notices that the database is now getting corrupted. Now what?

Using code (including sled) in new ways is fun, right up until someone calls you up at 2 a.m. screaming at you to 'fix it right now!' With a database, it's better to find out not only what is possible to do, but was is considered normal and expected usage of the database. That way you get less interrupted sleep. :wink:

[spacejam]

Yeah, I would be extremely distrustful of the advisory locks working as something more than a last-ditch attempt to prevent corruption. I do often consider that it would be nice to create a unix socket listener for the first process to open the database, and to allow other processes that can access the unix socket file to issue commands that are serialized by the first process. Transactions become a bit more interesting in this case however, and some changes need to happen in the transaction subsystem to facilitate this kind of usage in a seamless way.

[ckaran]

@spacejam I like the idea of a unix socket, but I think that you may wish to make that a separate crate. That way sled can remain highly portable.

As for implementing transactions, what would happen if you make Batch serializeable? External processes could send their batches to the process that owns sled, and it would apply them in whatever order they happen to come in. The advantage of this method is that it completely isolates sled from the networking side of things; batches could be coming in from unix sockets, named FIFOs, TCP, UDP, whatever, and it wouldn't matter to you. The only thing you might need to do is to expand what Batch is capable of doing, so that the rest of the operations that are possible on database or tree are exposed to it.

[spacejam]

The tricky thing with transactions is that you need to be able to interactively read data in between arbitrary user code inside a transaction closure, and after the closure, you need to perform concurrency control before atomically persisting any changes from the transaction, and then possibly have the client retry etc... The atomic writebatch part is pretty easy because it just needs to call sled's existing batch code in a non-interactive way.

[vi]

Shall documentation be updated somehow to address discussion in this issue?

Something like "Sled does not support sharing database between separate programs. Although Sled uses advisory locking to prevent data corruption if opened by multiple processes, it is not recommended to rely on it in production."

[andreivasiliu]

I agree. I picked up sled (via bindb) as I was looking for an alternative to Python's shelve, as a way to cache objects to disk in a way that doesn't involve reading the whole (500mb) database to RAM first. It worked pretty well, until, to my surprise, I wasn't able to open the database from two processes even as just read-only, and the documentation said nothing about this being the case.

[ckaran]

@vi, @andreivasiliu, you're right that it should be documented; have you considered making a pull request that updates the documentation? Since @spacejam is essentially the only developer on this project (he's made around 99% of all commits), any help we can give him would likely be viewed favorably. And since documentation of this type is relatively simple, it's well within all of our abilities to write it.

@spacejam, I see your point about transactions being difficult. Maybe instead of expanding Batch to cover everything that you can do to a database or a tree, we use a chosen subset. I looked at the code for Batch, and I don't think it can be expanded as-is. In my mind, Batch is actually a builder of a sequence of operations. So, something like the following (all of this is just banged out at the keyboard, no testing, probably has syntax bugs + semantic bugs, etc., etc., etc.):

use serde::{Serialize, Deserialize};
use thiserror::Error;

#[derive(Debug, Clone, Serialize, Deserialize)]
enum BatchElement{
    Insert{key: IVec, value: IVec},
    Remove{key: IVec, return_old_value: bool},
    Merge{key: IVec, value: IVec},
    ...
}

#[derive(Debug, Clone, Serialize, Deserialize, Error)]
pub enum BatchError {
    #[error("Oops")]
    InsertErr{"Useful Information Here"},
    ...
}

#[derive(Debug, Default, Clone, Serialize, Deserialize)]
pub struct Batch {
    pub(crate) writes: Vec<BatchElement>
}

impl Batch {
    pub fn insert(self, key: AsRef<[u8]>, value: AsRef<[u8]>) -> Self 
    {
        self.writes.push(BatchElement::Insert{key: key.clone(), value: value.clone()});
        self
    }

    pub fn remove(self, key: AsRef<[u8]>, return_old_value: bool) -> Self 
    {
        self.writes.push(BatchElement::Remove{key: key.clone(), return_old_value});
        self
    }

    pub fn merge(self, key: AsRef<[u8]>, value: AsRef<[u8]>) -> Self 
    {
        self.writes.push(BatchElement::Merge{key: key.clone(), value: value.clone()});
        self
    }
}

Once the sequence is built, it is serialized, sent to the DB process, deserialized and applied. At the first error encountered, the batch is aborted, and the error returned (BatchError above). Since this is a transaction, only if the entire transaction applies cleanly is it committed.

While this doesn't give you the full power of in-process transactions, it does give you a lot of power, all while avoiding executing arbitrary code from potentially untrusted processes. Would this kind of an interface work for you?

[vi]

Shall this issue remain open until documentation is updated?