Sled: Record/Table Tree overlay

[spacejam]

A nice feature for more structured queries, especially once transactions are in place on the Tree, is to be able to define a relatively thin Record or Table-like overlay where a rust structure is mapped into an ordering of types, some of which existing as a component of a primary key (maybe just defaulting to the first item in the struct) and the rest being in the value. This will allow us to write code similar to:

#[derive(Record)]
struct Post {
  id: u64,
  author: String,
  content: String,
  author_ip: Ipv6Addr,
}

fn main() -> Result<(), ()> {
  let db = Db::start_default(path)?;
  let posts = db.open_table::<Post>(b"posts")?;

  p.update(|post| post.content = post.content.to_uppercase())
   .where(|post| post.author == "alice");
}

[Kerollmops]

We could probably specify the primary key by specifying it using a container attribute like serde already do.

#[derive(Record)]
#[sled(primary_key = id)]
struct Post {
  id: u64,
  author: String,
  content: String,
  author_ip: Ipv6Addr,
}

[spacejam]

@Kerollmops yeah! I was thinking this could also work for things like auto_increment, unique, foreign_key(other_table.column) that can be set on individual members.

[spacejam]

I don't want to get too deep into SQL territory but I consider the above things kind of table stakes ;)

[Kerollmops]

I think the Record proc_macro should just be a convenient maner to implement a trait, this trait should implementable by hand, like serde Serialize/Deserialize. This way it could be possible to support dynamic types (e.g. json values, HashMaps).

[tomhoule]

I have been playing around with the idea for the prisma-sled connector I have been thinking about, but just noticed this issue, so I put together a very rough working sketch here: https://github.com/tomhoule/sled-record

The interesting part is mostly the test/example with a rountrip: https://github.com/tomhoule/sled-record/blob/master/sled-record/tests/generic_tests.rs

With field annotations, we could even support automatic secondary indexes (and unique), and generate methods on the struct, something like (if we had a unique index on User.email): User::find_by_email(&db, "[email protected]").

One thing I wonder about is whether the implementation would be any different if it were inside sled, since I think we have all the primitives we need in the public API. It could live as a separate crate. Edit: just thought about the idea of caching the deserialized representation of a record at the sled level (https://github.com/spacejam/sled/issues/770) - then it would make sense to integrate more. Are there other reasons?

I'll try to be on discord tomorrow (berlin time) to discuss :)

[D1plo1d]

@tomhoule Very interesting!

I've put together a prototype of Sled record (de)serialization as well but with a focus on zero downtime migrations.

My thinking is that if we store each record as an enum of revisions that can be migrated on retrieval then an embedded application has a path forward for migrating legacy data. The specifics of the migration strategy are left to the application so this is very much just tools to (de)serialize forwards-compatible data to sled. Here's what it looks like in practice:

https://github.com/tegapp/teg/blob/bccbd75fcd7a2a52eb7308a025e2ff56fe73c800/packages/teg-auth/src/print_queue/tasks/models/mod.rs

One thing that interested me but I have zero experience in is indexes. How are you handling those in sled-record?