Serde: Implement prefix for ser/deserializing structure fields

[vityafx]

In my library which implements challonge rest api there are a lot of places where the service (challonge) sends the same structures with different prefixes:

struct Match {
    // ...
    player1_votes: u64,
    player1_name: String,
    player2_votes: u64,
    player2_name: String,
}

We may see a structure here, let's name it Player:

struct Player {
    votes: u64,
    name: String,
}

However we can't parse it. I propose a feature prefix which can be used to deserialize a field with a prefix. So, here is what would Match structure look like:

struct Match {
    // ...
    #[serde(flatten, prefix = "player1_"),
    player1: Player,
    #[serde(flatten, prefix = "player2_"),
    player2: Player
}

This would parse good using the same structure. However, this feature needs flattening to be implemented.

[dtolnay]

Do you know of any other APIs that follow this convention?

[vityafx]

@dtolnay I don't remember now, but I am sure I have seen this kind of serializing two times yet. Yes, this is rare case, but the feature is not really hard to implement I think (I can try to do that by myself), and without it it is really difficult to parse data serialized that way. Do you mean this feature should not be implemented because cases like this are rarely seen?

[dtolnay]

Right, I would prefer not to implement this knowing of only one or two APIs where it would be useful. Let's give other users a chance to chime in that this would be useful to them.

With flatten (#119) here is one possible workaround:

#[derive(Serialize, Deserialize)]
struct Match {
    #[serde(flatten, with = "Player1")]
    player1: Player,
    #[serde(flatten, with = "Player2")]
    player2: Player,
}

struct Player {
    votes: u64,
    name: String,
}

#[derive(Serialize, Deserialize)]
#[serde(remote = "Player")]
struct Player1 {
    #[serde(rename = "player1_votes")]
    votes: u64,
    #[serde(rename = "player1_name")]
    name: String,
}

#[derive(Serialize, Deserialize)]
#[serde(remote = "Player")]
struct Player2 {
    #[serde(rename = "player2_votes")]
    votes: u64,
    #[serde(rename = "player2_name")]
    name: String,
}

[dtolnay]

The workaround in my previous comment has been implemented and works as of Serde 1.0.44 and we have not yet heard from anyone else with this use case so I am going to close. Thanks!

[synek317]

@dtolnay please count me as 'anyone', this workaround works but is ugly. Can you imagine it when I have let say 5 instances of the same type?

And I think it does not work with Options. Or at least I cannot make it working

[dtolnay]

Here is one that can do 5 instances of the same type and Options. Requires serde 1.0.65.

#[macro_use]
extern crate serde_derive;

extern crate serde;
extern crate serde_json;

#[derive(Serialize, Deserialize, Debug)]
struct Match {
    #[serde(flatten, with = "prefix_player1")]
    player1: Player,
    #[serde(flatten, with = "prefix_player2")]
    player2: Option<Player>,
    #[serde(flatten, with = "prefix_player3")]
    player3: Option<Player>,
}

#[derive(Serialize, Deserialize, Debug)]
struct Player {
    votes: u64,
    name: String,
}

macro_rules! prefix_with {
    ($module:ident $prefix:expr) => {
        mod $module {
            use serde::de::{self, Deserialize, DeserializeOwned, Deserializer};
            use serde::ser::{self, Serialize, Serializer};
            use serde_json::{self, Map, Value};

            pub fn serialize<T, S>(object: &T, serializer: S) -> Result<S::Ok, S::Error>
            where
                T: Serialize,
                S: Serializer,
            {
                let value = serde_json::to_value(object).map_err(ser::Error::custom)?;
                match value {
                    Value::Object(map) => {
                        let add_prefix = map
                            .into_iter()
                            .map(|(k, v)| (format!("{}{}", $prefix, k), v));
                        serializer.collect_map(add_prefix)
                    }
                    Value::Null => serializer.serialize_none(),
                    _ => Err(ser::Error::custom("expected a map or none")),
                }
            }

            pub fn deserialize<'de, T, D>(deserializer: D) -> Result<T, D::Error>
            where
                T: DeserializeOwned,
                D: Deserializer<'de>,
            {
                let map = Map::deserialize(deserializer)?;
                let strip_prefix = map
                    .into_iter()
                    .filter_map(|(k, v)| {
                        if k.starts_with(&$prefix) {
                            Some((k[$prefix.len()..].to_owned(), v))
                        } else {
                            None
                        }
                    })
                    .collect::<Map<String, Value>>();
                let value = if strip_prefix.is_empty() {
                    Value::Null
                } else {
                    Value::Object(strip_prefix)
                };
                serde_json::from_value(value).map_err(de::Error::custom)
            }
        }
    };
}

prefix_with!(prefix_player1 "player1_");
prefix_with!(prefix_player2 "player2_");
prefix_with!(prefix_player3 "player3_");

fn main() {
    let m = Match {
        player1: Player {
            votes: 1,
            name: "name1".to_owned(),
        },
        player2: Some(Player {
            votes: 2,
            name: "name2".to_owned(),
        }),
        player3: None,
    };

    let j = serde_json::to_string_pretty(&m).unwrap();
    println!("{}", j);

    println!("{:#?}", serde_json::from_str::<Match>(&j).unwrap());
}

[synek317]

I really don't know what to say... One the one hand, your code is totally amazing and I wish I could be as smart as you. On the other hand, this became even more complicated and magical workaround that is now tied to json format, making it even worse.

[dtolnay]

The serde_with crate now provides a version of with_prefix! that does not use json internally.
https://docs.rs/serde_with/0.2.1/serde_with/macro.with_prefix.html
https://docs.rs/serde_with/0.2.1/src/serde_with/with_prefix.rs.html

[synek317]

Wow, this is totally amazing

[vityafx]

Yes, this is already better. Thanks.

[axos88]

@dtolnay Twilio uses the same convention for GeoData for the From and To phone number data

[scull7]

I'm stuck using a 3rd party API that uses a convention in the same vein, but doesn't fit the serde_with use case; or perhaps I'm not smart enough to see how it fits.

I have a response with the following structure:

{
  "something": [
  {
    "name": "foo",
    "annoying": [
      { "name": "thing1", "foo_cow": 1, "foo_dog": 2, "foo_cat": 3 },
      { "name": "thing2", "foo_cow": 2, "foo_dog": 3, "foo_cat": 4},
    ]
  },
  {
    "name": "bar",
    "annoying": [
      { "name": "thing1", "bar_cow": 1, "bar_dog": 2, "bar_cat": 3 },
      { "name": "thing2", "bar_cow": 2, "bar_dog": 3, "bar_cat": 4},
    ]
  },
  {
    "name": "baz",
    "annoying": [
      { "name": "thing1", "baz_cow": 1, "baz_dog": 2, "baz_cat": 3 },
      { "name": "thing2", "baz_cow": 2, "baz_dog": 3, "baz_cat": 4},
    ]
  }]
}

I want to deserialize into:

#[derive(Deserialize)]
pub struct Member {
  name: String,
  cow: u16,
  dog: u16,
  cat: u16,
}
#[derive(Deserialize)]
pub struct Group {
    name: String,
    #[serde(rename = "annoying")]
    members: Vec<Member>
}
#[derive(Deserialize)]
pub struct Collection {
  #[serde(rename = "something")]
  items: Vec<Group>  
}

Is there a macro to deserialize based on a regex field match? Any help would be sanity saving 😄

[agausmann]

FWIW, I also ran into this while trying to use serde to help parse Factorio game data - the Item prototype has two flattened IconSpecification structures, one which uses the field names icons, icon, icon_size and the other uses the field names dark_background_icons, dark_background_icon, icon_size.

This is an even more complex example, because the prefix doesn't apply to all the fields - the icon_size is shared between the two. I think the workaround using with and remotes will be the best for my usecase, thanks for pointing that out.