Rfcs: Serializable Lambda expression

[hyunsik]

Rust basically is very strong for parallel programming. If Rust supports Serializable Lambda expression, it would be very strong even for distributed programming.

Here was a little discussion.
https://github.com/rust-lang/rust/issues/23759

Python and Java 8 already have similar concepts as follows:

• https://docs.python.org/3/library/pickle.html
• https://docs.oracle.com/javase/8/docs/api/java/lang/invoke/SerializedLambda.html

[kmcallister]

See also #668.

[Rufflewind]

One could do add some compiler magic to support arbitrary object serialization in Rust, with the caveat that such an object can only be deserialized by the same program. This would be like Any with persistence. The Haskell recently added support for these kinds of “static pointers”.

But the downside of this is that the user has no control over the serialization protocol, and it would require making a serializer that works over arbitrarily complicated types, even though much of it could be done outside of the standard library.

A more flexible approach would be to expose two new mechanisms to Rust:

• The ability to have an extensible lookup table for a selection of types, a “global type registry” if you will. This I think is the main missing ingredient for #668.

• The ability to construct and deconstruct closures into a tuple of its upvars (Closure::Data). Typically, Closure::Data would be serializable through a #[derive] mechanism.

  ~~~rust
  pub trait Closure {
  type Data;
  fn construct(data: Self::Data) -> Self;
  fn deconstruct(self) -> Self::Data;
  }

  impl Closure for /* all unboxed closures / {
  type Data = (/ upvars ... /);
  / ... */
  }

  impl Closure for /* all fn(_) -> _ / {
  type Data = ();
  / ... */
  }
  ~~~

Edit: no need store a separate function pointer since the closure type itself would suffice.

[ariesdevil]

Any updates on it? With write spark like system in rust, Serializable Lambda expression is needed.

[habnabit]

(python's pickle does not serialize functions' code to a wire format; it only serializes the name of a function.)

[ariesdevil]

@habnabit but you can implement it easier,see https://github.com/douban/dpark/blob/0.5.0/dpark/serialize.py#L250

[alecmocatta]

I created a crate serde_closure that makes closures serializable by wrapping them with the macro Fn!(...).

If the resulting serializable closure is upcast to Box<dyn serde_traitobject::Fn()> it can then be serialized and sent between identical distributed processes using serde_traitobject, though this currently requires nightly.

[Rufflewind]

Interesting. I had tried a similar idea a while back but deemed it too questionable for production use.

[alecmocatta]

@Rufflewind Yes, thanks for your crate, it was good inspiration for serde_traitobject.

I just made a rust PR that would make serde_traitobject safe: https://github.com/rust-lang/rust/pull/66113. Keen to hear if you have any thoughts!