Serde: Derive stateful serialization/deserialization

[Marwes]

I started looking into if it were possible to serialize/deserialize shared references in a way that preserves the reference graph and encountered this proof of concept https://github.com/TyOverby/bincode/issues/139#issuecomment-287259643. While working on generalizing that implementation I started thinking that while I could probably remove most of the duplication by using macros there would still be a non trivial amount of code left which I'd rather not have to write.

So I propose some extensions to serde so it is possible to easily create stateful deserializers and serializers. While this could maybe be implemented as a separate crate I think it would incur a fair amount of duplication if the crate were to properly handle attributes such as #[serde(rename ...)].

Deserialization

pub struct PassThroughSeed<S, T> {
    pub seed: S,
    _marker: PhantomData<T>,
}

#[derive(DeserializeSeed)]
#[serde(deserialize_seed="Seed")]
pub struct MyType {
    // Marking a field with a visitor which is constructed from `self.seed`
    #[serde(deserialize_seed_with="SomeTypeVisitor::new")]
    field: SomeType,
    // `NestedType` has `impl DeserializeSeed for PassThroughSeed<Seed, NestedType>`
    #[serde(deserialize_seed)]
    field2: NestedType,
    // If no deserialize_seed_with attribute is set just use the normal `Deserialize` implementation
    field3: i32
}

// Deriving `DeserializeSeed` generates a passthrough implementation
// (It does not use the seed itself but just serves as a provider of the seed to the fields)
impl DeserializeSeed for PassThrough<Seed, MyType> {
    // Normal struct derive except `*_seed` functions are called
}

Serialization

trait SerializeSeed {
    type Seed;
    fn serialize_dag<S>(&self,
                        serializer: S,
                        seed: &Self::Seed)
                        -> Result<S::Ok, S::Error>
        where S: Serializer;
}


pub struct Wrapper<'a, S, T> {
    pub state: S,
    pub value: &'a T,
}

#[derive(SerializeSeed)]
#[serde(serialize_seed="Seed")]
pub struct MyType {
    #[serialize_seed_with="TypeVisitor::new"]
    field: Type,
    // Wraps `self.field2` using `Wrapper`
    #[serialize_seed]
    field2: Type,
    // Normal serialization
    field3: i32,
}

If the implementation seems sound I will try and implement it (wheter that be in serde or in separate cratte) however it might not be a top priority for me for a bit.

[dtolnay]

Yes! This would be amazing. Your approach seems sound to me. I bet it would eliminate >80% of @dnorman's handwritten Unbase serialization code.

[Marwes]

serde_state and serde_derive_state has been published (I replaced seed with state to get rid of the ugly DeserializeSeedEx name). The documentation could use some improvement and there is probably some things in the API that does not need to be there but everything that is needed should be working now.

Example usage can be found at https://docs.rs/serde_state/0.2.0/serde_state/. More complex usage (tracking shared references) can be seen in https://github.com/gluon-lang/gluon/pull/279 (look in base/src/types.rs, base/src/serialization.rs, vm/src/serialization.rs, vm/src/value.rs) as well as the added tests de tests, ser tests.

https://crates.io/search?q=serde_state

At this point I have everything I for https://github.com/gluon-lang/gluon/pull/279 (merging it after some cleanup work) so for my purposes this does everything I need for the moment. It may still lack for things for other people so at this point the crate(s) really just need more people trying it.

Pinging some people that have shown some interest in this addition:

@dnorman
@xrl
@KodrAus
@alexander-irbis
@pratyush

[RReverser]

@Marwes Could the derive be still made compatible with native serde DeserializeSeed / SerializeSeed? Or is serde_state meant to be a complete replacement for these?

[Marwes]

Could the derive be still made compatible with native serde DeserializeSeed / SerializeSeed?

The DeserializeState/SerializeState instances can be converted to DeserializeSeed/SeriializeSeed instances via wrappers easily. The other way around is more awkward (which is why these are new traits). It comes down to that deriving the *Seed traits are really awkward when forwarding to each field since it is impossible to know which Seed type is used for that field. Whereas that *State trait don't need to mess around with a seed type per type and can just pass the same State around everywhere https://github.com/serde-rs/serde/pull/909#issuecomment-303841743.

Or is serde_state meant to be a complete replacement for these?

I wish there was a good way to omit the traits but it just makes the code more complicated. The actual cost of having two more traits is also just that, two more trait declarations. All of the other code would still need to be there regardless of the addition of these traits. Since they can also be converted to Seed types there is no loss in generality either.

So all in all, having the traits makes for less complicated code than omitting them.

[Shados]

@Marwes I recently started poking about with serde_state, and appear to have run into a bug: attempting to use #[derive(DeserializeState)] with untagged enums fails to compile, although from looking at Gluon's source, it works OK with SerializeState?

[Marwes]

The derive macros for Serialize and Deserialize are entirely different so that on its own is not that surprising, I'd expect a test to have caught it though. Do you have a minimal reproduction, ideally one that can just be pasted into the test suite?

[RReverser]

I've noticed that serde_state is currently implemented as a fork of the Serde repo (https://github.com/Marwes/serde_state) and is already >1 year behind the upstream (168 commits).

Does this have any effect on the bugs fixed in between and so on? Are there plans to merge serde_state into the main repo or, the opposite, decouple them further?

[RReverser]

Aside from theoretical concerns, the practical issue with above is that currently there is no way to report issues or feature requests to serde_state, because it's just a fork of serde, and Github doesn't show dedicated Issues tab on forks.

[RReverser]

@Marwes Can you please make it possible to report issues to serde_state somehow, maybe by splitting repos away?