Serde: custom deserialization for HashMap values

[cmusser]

I have a custom deserialization for a struct, and I'd like to perform the same deserialization to a HashMap of those structs.

Currently, I have a structure like this:

#[derive(Serialize, Deserialize)]
pub struct KeyData {
    // Some stuff elided....
    #[serde(serialize_with = "u8vec_as_hex", deserialize_with = "pubkey_from_hex")]
    pub peer_public: PublicKey,
}

And what I'd like is something along the lines of:

#[derive(Serialize, Deserialize)]
pub struct KeyData {
    // Some stuff elided....

    // I don't think there's a magic annotation for this, but
    // the key is just a string, and the value is a structure *derived*
    // from a string.
    pub public_peers: HashMap<String, PublicKey>,
 }

The essence of it is that the key (the string), can come straight from the source YAML, but the value (the PublicKey), must be deserialized from a string that is a compact text representation of the hex values that make up the key.

Notes:

1. The PublicKey struct is from Sodiumoxide (the libsodium wrapper). Sodiumoxide does provide Serialize/Deserialize functions, but they produce and read from a list of hex values representing the individual key bytes and I want a more compact representation.

2. pubkey_from_hex() is this:

   fn pubkey_from_hex<'de, D>(deserializer: D) -> Result<PublicKey, D::Error>
       where D: Deserializer<'de>
   {
       use serde::de::Error;
       String::deserialize(deserializer)
           .and_then(|string| base16::decode(&string.as_bytes()).map_err(|err| Error::custom(err.to_string())))
           .map(|bytes| PublicKey::from_slice(&bytes))
           .and_then(|opt| opt.ok_or_else(|| Error::custom("failed to deserialize public key")))
   }
   

3. I don't actually need the companion serialization operation (u8vec_as_hex"). The code that uses my struct is only interested in reading the values and then using them as is.

I looked at this page in the manual https://serde.rs/deserialize-map.html, but I couldn't grasp how to apply the example to what I need. I think that the the custom deserializer code above (or some form of it) needs to be incorporated into that example, and I can't figure out where.

Is there a way to get this to work? This seems to be a specific case of the enhancement request in https://github.com/serde-rs/serde/issues/723, but I'm not sure if there is a solution there yet.

[dtolnay]

There is no convenient supported way to do this yet (as tracked in #723), but you can provide a deserialize_with function that can deserialize the full HashMap<String, PublicKey>. It would look much like the function in https://github.com/serde-rs/serde/issues/723#issuecomment-382501277.

[cmusser]

I tried to follow the stuff in that comment (and I think what were related snippets in earlier comments), but I don't understand the concepts and couldn't put the pieces together. I probably need a step by step walkthrough to know what to do.

An alternate solution is to deserialize into a HashMap and then apply a mapping function which creates a new hashmap where the values gets transformed from hex strings into the PublicKeys. That's sort of a boneheaded two-step solution that doesn't fully leverage Serde, but it might be enough to for me to get unstuck.

[dtolnay]

Here is what the wrappers from that thread would look like, but for HashMap rather than Vec. Playground

#[macro_use]
extern crate serde_derive;

extern crate serde;

use serde::{Serializer, Deserialize, Deserializer};
use sodiumoxide::*;
use std::collections::HashMap;

// provided by Sodiumoxide
mod sodiumoxide {
    use serde::{Serializer, Deserializer};

    pub struct PublicKey;

    pub fn u8vec_as_hex<S>(_key: &PublicKey, _serializer: S) -> Result<S::Ok, S::Error>
    where
        S: Serializer,
    {
        unimplemented!()
    }

    pub fn pubkey_from_hex<'de, D>(_deserializer: D) -> Result<PublicKey, D::Error>
    where
        D: Deserializer<'de>,
    {
        unimplemented!()
    }
}

#[derive(Serialize, Deserialize)]
struct KeyData {
    #[serde(serialize_with = "ser_peer_public", deserialize_with = "de_peer_public")]
    peer_public: HashMap<String, PublicKey>,
}

fn ser_peer_public<S>(peer_public: &HashMap<String, PublicKey>, serializer: S) -> Result<S::Ok, S::Error>
where
    S: Serializer,
{
    #[derive(Serialize)]
    struct Wrapper<'a>(#[serde(serialize_with = "u8vec_as_hex")] &'a PublicKey);

    let map = peer_public.iter().map(|(k, v)| (k, Wrapper(v)));
    serializer.collect_map(map)
}

fn de_peer_public<'de, D>(deserializer: D) -> Result<HashMap<String, PublicKey>, D::Error>
where
    D: Deserializer<'de>,
{
    #[derive(Deserialize)]
    struct Wrapper(#[serde(deserialize_with = "pubkey_from_hex")] PublicKey);

    let v = HashMap::<String, Wrapper>::deserialize(deserializer)?;
    Ok(v.into_iter().map(|(k, Wrapper(v))| (k, v)).collect())
}

[cmusser]

Ok, thanks for typing that up. It worked very nicely and is far more straightforward than I'd thought. The two "levels" are easy enough to understand. About the only thing that was mysterious was :

#[derive(Serialize)]
struct Wrapper<'a>(#[serde(serialize_with = "u8vec_as_hex")] &'a PublicKey);

Looks like a struct declared inside a function, but I didn't expect parens surrounding the annotation and the &'a PublicKey. But I do understand how the wrapper does the transformation. At first I wasn't clear on how things like iter() and map() worked with key-value collections, but I see they are handled by these functions just fine. It's neat: you can transform key or value as you see fit.

[cmusser]

This problem was resolved for me. The solution is likely to be useful to others with similar data structures.

[Ottawan]

Wrapper should derive some other traits, I get misleading error about missing deserialize() for HashMap otherwise, i.e.:

#[derive(Deserialize, Hash, Eq, PartialEq)]
struct Wrapper(#[serde(deserialize_with = "pubkey_from_hex")] PublicKey);

[schungx]

Thanks for all the help here!

I have the following for custom deserialization for a HashMap with numeric keys:

~~~
fn deserialize_hashmap_with_u32_key<'de, D>(deserializer: D) -> Result, D::Error>
where
D: Deserializer<'de>,
{
fn deserialize_string_to_u32<'de, D>(deserializer: D) -> Result
where
D: Deserializer<'de>,
{
let s = String::deserialize(deserializer).map_err(serde::de::Error::custom)?;
s.parse::().map_err(serde::de::Error::custom)
}

#[derive(Deserialize, Hash, Eq, PartialEq)]
struct Wrapper(#[serde(deserialize_with = "deserialize_string_to_u32")] u32);

let v = HashMap::<Wrapper, Object>::deserialize(deserializer)?;
Ok(v.into_iter().map(|(Wrapper(k), v)| (k, v)).collect())

}
~~~

Only deserialization is needed because serialization automatically converts the numeric keys to strings for serde_json.