Json: possibility to have a Streaming serializer ?

[allan-simon]

Imagine I have a huge request from my database that I would like to serialize as a json array to send it over the network

currently (if i haven't missed anything), I have to accumulate my objects into a vector , which I then will be able to serialize in one time.

it's fine most of the time but here looking to the size of the vector

• my program needs to keep a copy in memory of all the data before being able to send a single byte (so huge RAM consumption)
• it needs to wait the vector creation to be completed to sent the first byte (so huge latency)

If Serde was to provide a way to serialize things as it go, I would be able to something like that

stream.write(serializer.start_array())
for one_row from all_database_rows.iter() {
      let my_object = object_from_row(one_row);
      stream.write(serializer.array_item(my_object));
}
stream.write(serializer.end_array())

[oli-obk]

This is possible already. You need a helper type, but inside that type's Serialize impl you can call serialize_seq, which gives you an object you can keep adding elements one by one by calling serialize_element on it. Once you are done, you call the end method

[allan-simon]

thanks for your answer, as i'm still a bit new with both rust and serde I just want to confirm some point:

my helper type actually would hold for example the rust-postgres' "Rows" (which is an iterator)
and serializing it will only then unroll the iterator right ? which solve the problem of having an intermediate Vec

in that case it's great but actually only solve one part of my "problem" , the second is that now I still have an intermediate string representing the whole array serialized as json, and I would like to know if i can get rid of it completly by directlying writing in the stream rather than in the String

[oli-obk]

my helper type actually would hold for example the rust-postgres' "Rows" (which is an iterator)
and serializing it will only then unroll the iterator right ?

Yes

he second is that now I still have an intermediate string representing the whole array serialized as json, and I would like to know if i can get rid of it completly by directlying writing in the stream rather than in the String

That's possible by using to_writer instead of to_string

[dtolnay]

I wouldn't bother with a wrapper type:

let rows = /* whatever iterator */;
let out = std::io::stdout();

let mut ser = serde_json::Serializer::new(out);
let mut seq = ser.serialize_seq(Some(rows.len()))?; // or None if unknown
for row in rows {
    seq.serialize_element(&row)?;
}
seq.end()?;

[dan-fritchman]

        use serde_json::Serializer;
        let out = std::io::stdout();
        let mut ser = Serializer::new(out);
        let mut seq = ser.serialize_seq(None)?;

Yields:

    |
641 |         let mut seq = ser.serialize_seq(None)?;
    |                           ^^^^^^^^^^^^^ method not found in `serde_json::ser::Serializer<std::io::Stdout>`
    |

Should this still work?
Same applies when replacing Stdout with std::fs::File.

[dtolnay]

Yes. The full error message would have said what traits to import.

use serde::ser::{SerializeSeq, Serializer};

fn main() -> serde_json::Result<()> {
    let rows = /* whatever iterator */ "serde".chars();
    let out = std::io::stdout();

    let mut ser = serde_json::Serializer::new(out);
    let mut seq = ser.serialize_seq(None)?;
    for row in rows {
        seq.serialize_element(&row)?;
    }
    seq.end()
}