Json: Handle 64-bit types encoded as strings

[jsgf]

Because JSON interop for 64-bit integers is bad (precision loss if the impl treats them as doubles), lots of JSON serializations use strings to represent large integers. This doesn't seem to work out of the box, and I didn't see an option to allow this. It can be handled by manually specifying a serializer/deserializer, but it seems like a common case.

[dtolnay]

I would prefer not to implicitly accept strings as 64-bit numbers because it would destroy the ability for untagged enums to distinguish between strings and numbers.

#[derive(Deserialize)]
#[serde(untagged)]
enum E {
    U(u64),
    S(String),
}

Would something like the following work for your use case?

#[macro_use]
extern crate serde_derive;

extern crate serde;
extern crate serde_json;

#[derive(Serialize, Deserialize, Debug)]
struct Jsgf {
    #[serde(with = "string")]
    u: u64,
    #[serde(with = "string")]
    i: i64,
}

mod string {
    use std::fmt::Display;
    use std::str::FromStr;

    use serde::{de, Serializer, Deserialize, Deserializer};

    pub fn serialize<T, S>(value: &T, serializer: S) -> Result<S::Ok, S::Error>
        where T: Display,
              S: Serializer
    {
        serializer.collect_str(value)
    }

    pub fn deserialize<'de, T, D>(deserializer: D) -> Result<T, D::Error>
        where T: FromStr,
              T::Err: Display,
              D: Deserializer<'de>
    {
        String::deserialize(deserializer)?.parse().map_err(de::Error::custom)
    }
}

fn main() {
    let j = r#" { "u": "18446744073709551615", "i": "-9223372036854775808" } "#;

    let jsgf: Jsgf = serde_json::from_str(j).unwrap();
    println!("{:#?}", jsgf);

    println!("{}", serde_json::to_string_pretty(&jsgf).unwrap());
}

[jsgf]

Thanks! I was going to write that myself, but this saves me the trouble.

I wasn't suggesting it should happen automatically - just allowing some option so that i64 fields can be deserialized from strings. The with annotation seems fine, but it would be nice if string were a standard part of the serde_json crate (or whereever makes sense).

BTW, where is #[serde(with = "...")] documented? I noticed it used in passing in a number of the examples, but I didn't see a mention of it in the "Attributes" part of the docs.

[dtolnay]

Yep, we are planning to do a helper library of such functions in https://github.com/serde-rs/serde/issues/553.

It is documented under field attributes. https://serde.rs/field-attrs.html

[klausi]

So the Mastodon API changed at some point to use strings instead of integers in their account JSON. Currently failing test demo at https://github.com/Aaronepower/Mammut/pull/15

How can you specify that a field should either be u64 or u64 encoded as string? Is there a [#serde(u64_or_string)]?

[oli-obk]

@klausi: There is an example earlier in this issue on how to solve it (https://github.com/serde-rs/json/issues/329#issuecomment-305608405). Can you elaborate on how it does not work in your use case?

[klausi]

We want to accept both strings and integers from JSON to be more robust. The solution from the comment above only works if the data is passed as string type in JSON. Serde deserialization fails if the type is integer in JSON.

[oli-obk]

To best of my knowledge that is still somewhat involved. You need a Visitor impl: https://play.rust-lang.org/?gist=235ab908634066163ebe17c9760fce80&version=stable

[dtolnay]

There is also the untagged enum way which is less code. Playground

#[derive(Deserialize)]
#[serde(untagged)]
enum StringOrInt {
    String(String),
    Int(i64),
}

match StringOrInt::deserialize(deserializer)? {
    StringOrInt::String(s) => s.parse().map_err(de::Error::custom),
    StringOrInt::Int(i) => Ok(i),
}

[oli-obk]

That solution requires intermediate allocations which can never fully be eliminated even with Cow

[kevincox]

I would prefer not to implicitly accept strings as 64-bit numbers because it would destroy the ability for untagged enums to distinguish between strings and numbers.

I think this is enough of an edge case that we can be okay with it. Your untagged union case could always treat a native string type as the string variant.

The situation right now isn't great, and I don't think it can really be fixed without doing this in serde-json somewhere.

• You need to annotate every integer in your program. Otherwise users will be confused that some fields can be strings and some can't.
• If you use multiple serialization formats you will be forced to use a string in all of them. This is really a shame because supporting JSON alongside an efficient alternative is a common use case for me and it isn't really nice to make the efficient alternative less efficient for the JSON version.

I would propose:

• The parser can accept either format. If using untagged unions the variant will be selected as it is today.
• A serialization option/formatter is added to serialize ints larger than 32 bits as strings.