Flatbuffers: [Rust] Question : Converting a nested struct into a flatbuffer message

[jean-airoldie]

Hi,

I'm trying to implement nested structs that can be converted to flatbuffers message. I've implemented a minimal example of what im trying to achieve here.

Let's say I have the following schema file.

namespace payload;

table Payload {
    messages: [Message];
}

table Message {
    value: uint;
}

Now lets say I want to implement the following structs.

/// A struct that can be converted to a flatbuffer `Payload`.
struct PayloadStruct {
    message_structs: Vec<MessageStruct>,
}
/// A struct that can be converted to a flatbuffer `Message`.
struct MessageStruct {
    value: u32,
}

My issue is that, in order to serialize PayloadStruct as a Payload flatbuffer message I have to intermediately convert its messages_strucs field to a vector of Message within a FlatBufferBuilder. To do so, I see two options:

• Handle all the conversion in the PayloadStruct. This was done here and it compiles. The problem is that it offers no code reuseability.
• Deleguate MessageStruct the role of writing itself to the FlatBufferBuilder and do the rest of the work in PayloadStruct. This was attempted here but I can't get it to compile.

Is it possible to make the second option compile without using a RefCell or a Cell?

[rw]

@jean-airoldie What is the compiler error for your delegated function?

[jean-airoldie]

@rw I was giving the same lifetime to the FlatBufferBuilder and to its mutable reference. This solves it.

    pub fn to_bytes_delegated<'a, 'b>(
        &'b self,
        builder: &'a mut FlatBufferBuilder<'b>
    ) -> &'a [u8] {
        // ...
    }

Thanks for the quick response nonetheless!

[rw]

@jean-airoldie Great!

[jean-airoldie]

@rw Actually, I just though about this further and my solution doesn't work. Furthermore it seems to reveal a significant problem when reusing a FlatBufferBuilder like an arena. I simplified my example and added additional comments to illustrate the problem. But here is the general concept.

The current problem is that the lifetime of WIPOffset<_> returned from FlatBufferBuilder::create_vector() must be bound to the lifetime of the builder. This is a problem when your application reuses the same builder by periodically calling reset on it because its lifetime will necessarily outlast those of objects it serializes. The only current viable way (that i could think of) would be to do the entire serialization in one function so that no WIPOffset<_> escapes the anonymous lifetime. Otherwise, any function calling this this serialization function would need the lifetime of its parameters (that will be serialized) to be bound to the builder's lifetime, which effectively renders it unusable. But doing it this way prevents all forms of code reuse and forces to needlessly clone bytes within the builder to prevent returning a byte slice bound to the builder.

Conceptually I think the WIPOffset / slice to bytes within the builder should be bound to something other than the builder and that calling FlatBufferBuilder::reset() (or the builder going out of scope) invalidates the references. This way we could bind the lifetime of a function that does the serialization to the lifetime of the bytes in the builder.

[rw]

@jean-airoldie Thanks for digging into this! I admit I'm not entirely sure I understand.

1. Lifetimes and their bounds are pretty complicated in Rust FlatBuffers. For example, here is generated code that takes a moment to grok:
   https://github.com/jean-airoldie/fbs-payload-example/blob/master/src/payload_generated.rs#L119

    pub fn create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr>(
        _fbb: &'mut_bldr mut flatbuffers::FlatBufferBuilder<'bldr>,
        args: &'args MessageArgs) -> flatbuffers::WIPOffset<Message<'bldr>> {

FWIW, understanding that function signature might be useful in solving this lifetime problem.

1. You said

The current problem is that the lifetime of WIPOffset<_> returned from FlatBufferBuilder::create_vector() must be bound to the lifetime of the builder. This is a problem when your application reuses the same builder by periodically calling reset on it because its lifetime will necessarily outlast those of objects it serializes.

From my perspective (again, maybe I don't fully understand what you're trying to do), reusing that WIPOffset<_> would be a logic error if the builder gets reset. Instead, you would want to use the generated accessor functions to read the data.

Would you please restate for me, in a sentence, how you want to use the WIPOffsets between calls to builder.reset()?

1. About WIPOffsets...

Conceptually I think the WIPOffset / slice to bytes within the builder should be bound to something other than the builder and that calling FlatBufferBuilder::reset() (or the builder going out of scope) invalidates the references. This way we could bind the lifetime of a function that does the serialization to the lifetime of the bytes in the builder.

That might be nice, but I'm not sure that would be possible with compile-time checking.

1. Overall, I'm amenable to improving this API. I just want to make sure we fully grok the implications of what any lifetime changes would mean for safety and for ergonomics.

[jean-airoldie]

@rw I reread my explanation and I agree I could be clearer and more concise.

Basically the WIPOffsets that escape the builder are bound to the lifetime of the builder. This works fine when we create a builder each time we intend to serialize something because the serialized bytes will be dropped from scope at the same time as the builder.

The problem arises when we want to reuse the builder by creating it once, keeping it in a server (some struct that lives for the entire program) and then call reset on the builder after each serialization. Such scenario is appealing in a long lived program from a performance standpoint because we don't need to create a builder (its underlying vector) each time we want to serialize something. The problem is that, in this scenario, the builder will necessarily outlive the things it serializes. Since the WIPOffsets returned from pushing a struct are bound to the lifetime of the builder, the objects we push must also have to be bound to the lifetime of the builder, which is impossible for most usage.

My usage would be store the builder in a server struct and call builder.reset() after each serialization. This would require that the lifetime of the WIPOffsets be bound somehow to the anonymous lifetime of the function in which the serialization is done.

[rw]

@jean-airoldie

1) What can you do with WIPOffset that you can't do with get_root_as_* functions?
https://github.com/google/flatbuffers/blob/master/tests/monster_test_generated.rs#L1864

2) I notice you aren't yet using the root_type directive in your flatbuffers schema. You may want to add it. Example: https://github.com/google/flatbuffers/blob/master/tests/monster_test.fbs#L127

3) I think the mismatch that is confusing to me is this: how could you use a WIPOffset in a valid way, if the builder it is bound to has been reset?

:-)

[jean-airoldie]

@rw

1. My use case is not about deserializing since we can simply call get_root without any builder. I don't understand how I could construct a flatbuffers representation using get_root_as.
2. I use get_root directly since there can only be a unique root_type per schema file (from my understanding).
3.

fn foo(&mut self) {
  // The builder is empty here.
  let mut builder = self.builder;
  {
    // Somehow the WIPOffset must be bound to this anonymous lifetime
    let offset = serialize_a_vector_of_things(thing, &mut builder);
    // use the offset when constructing a more complex flatbuffer object
    ...
    // send the serialized bytes
  }
  // The WIPOffset is out of scope so we can safely call reset. 
  builder.reset()
  // The builder is empty once again.
}

[jean-airoldie]

@rw Regarding the API I was thinking of something like this:

fn send_things(&mut self, things: Vec<Thing>) {
  // This is a contiguous vector that we keep to reduce allocations
  // which is currently empty but has reserved storage.
  let mut vec = &mut self.vec;
  {
    // This creates a new builder that captures the anonymous lifetime
    // and uses our vector as its storage
    let mut builder = FlatBufferBuilder::from_vec(&mut vec);
    let offset = serialize_a_vector_of_things(things, &mut builder);
    // use the offset when constructing a more complex flatbuffer object
    ...
    // write the serialized bytes to a `Sink`
    // builder.reset(); might be an issue if we forget so we rather have `Drop` call it
  }
  // The builder goes out of scope and so do its WIPOffsets
  // We get back our vector empty with at least as much reserved storage.
}

Or probably cleaner:

fn send_thing(&mut self, things: Vec<Thing>) {
  // This is a a `FlatBufferArena` that is currently empty but has reserved storage.
  let mut arena = &mut self.arena;
  {
    // This creates a `FlatBufferBuilder` that captures the anonymous lifetime
    // and uses storage from the arena (so no allocations required).
    let mut builder = arena.create_builder();
    let offset = serialize_a_vector_of_things(things, &mut builder);
    // use the offset when constructing a more complex flatbuffer object
    ...
    // write the serialized bytes to a `Sink`
    // builder.reset(); might be an issue if we forget so we rather have `Drop` call it
  }
  // The builder goes out of scope and so do its WIPOffsets.
  // We get back our arena which is empty.
}

[rw]

@jean-airoldie I have a guess that reborrowing from structs is what is causing the issue here. Do you think it would work (as an example for us to learn more) if you used plain functions?

[jean-airoldie]

@rw I'm not sure what you mean by that. If you mean replacing &mut self by &mut FlatBufferBuilder in send_things, then we have the same problem because the builder's lifetime is still bound to the lifetime of the struct that contains it. Therefore borrowck will require that the anonymous lifetime of any function that calls send_things (to which the &mut borrow of the builder would be bound) last as long as the lifetime of the builder (and thus the struct who owns the builder), which does not make any practical sense. By definition a serialization function is meant to be used more then once, which means that the lifetime of each function will be strictly smaller than the one of the builder.

Here is a short example:

// Lets say its basically the same from my previous example, albeit
// with a different function signature.
fn send_things(&mut builder, things: Vec<Thing>) {
  /* ... */
}

fn gen_rand_things() -> Vec<Thing> {
  unimplemented!();
}

fn main() {
  // The lifetime of the builder is bound to the lifetime of the main function.
  let mut builder = FlatBufferBuilder::new();
  let things = gen_rand_things();
  // The lifetime of the borrow must be the same as the lifetime
  // of the builder which isn't the case. Here the anonymous lifetime
  // of the `send_things` function call is clearly shorter than the
  // one of the builder.
  send_things(&mut builder, things);
  let more_things = gen_rand_things();
  // Same here.
  send_things(&mut builder, more_things);
}

edit: fixed typo in main

[rw]

@jean-airoldie I'm asking for a "struct-less" version that just uses function calls. So it would be

send_things(&mut builder, things);

not

send_things(&mut server.builder, things);

[jean-airoldie]

@rw I made a typo, I edited my previous answer. The same situation occurs, but the builder is bound to the lifetime of the main function instead of a struct.

[rw]

@jean-airoldie I made a minimal PR to your repo that may solve your problem. My changes have the following reasoning behind them:

1) In the "delegated" builder function signatures, the WIPOffset lifetimes need to be tied to the builder. The previous function signature you used did not constrain it in that way, because the lifetime was elided from the function definition.
2) The functions (probably) should not return byte slices from builder.finished_data(), because then the builder can panic if you forget to reset it. Plus, lifetimes might continue to be problematic.
3) I added a sink parameter to show how I would write the builder.finished_data() into an object that implemented io::Write.

$ cargo run --quiet
[12, 0, 0, 0, 0, 0, 6, 0, 8, 0, 4, 0, 6, 0, 0, 0, 4, 0, 0, 0, 1, 0, 0, 0, 8, 0, 0, 0, 4, 0, 4, 0, 4, 0, 0, 0]

Please let me know if that helps you do what you are trying to do :-)

[jean-airoldie]

@rw Damn, it never occurred to me that I did not need to store the builder slice. This makes perfect sense. Thanks a lot for your time!

[rw]

@jean-airoldie Would you be interested in writing a FAQ/troubleshooting section for the Rust FlatBuffers documentation, to share what you learned?

[jean-airoldie]

@rw Sure! I think some common usage examples would be interesting as well. There are a couple of concepts that required me to look at the source code to understand better.