Messagepack-csharp: Support for deserializing into existing objects

[westclif]

I would love to have support for all formatters to actually deserialize their data into an already existing object if its provided as an additional reference parameter. (if this parameter is null, we could fall back to the current implementation to create the (T)object)

This would additionally help the unity3d implementation (in case you want to serialize unity intern classes / monobehaviors) in a setup there you want to deserialize the data into an already existing object.

This could also help to reduce memory allocations as the data could be restored into already existing field / classes of the provided object.

Something like that? -->

public interface IMessagePackFormatter<T> { bool Deserialize(byte[] bytes, int offset, IFormatterResolver formatterResolver, ref T @object, out int readSize) }

[neuecc]

Thank you, interesting feature.
Protocol Buffers has instance.MergeFrom(Stream), agree, it is very useful.

Here is concept API, is it okay?

public interface IOverwriteMessagePackFormatter<T>
{
    void DeserializeTo(ref T to, byte[] bytes, int offset, IFormatterResolver formatterResolver, out int readSize);
}

public static class MessagePackSerializer
{
    public static void DeserializeTo<T>(ref T to, byte[] buffer, IFormatterResolver resolver)
    {
        // resolver.
        var formatter = resolver.GetFormatter<T>() as IOverwriteMessagePackFormatter<T>;
        if (formatter == null) throw new Exception("not supported...!");

        //
        // formatter.DeserializeTo(ref to, bytes, offset, )...
    }
}

public class ListFormatter<T> : IOverwriteMessagePackFormatter<List<T>>
{
    public void DeserializeTo(ref List<T> to, byte[] bytes, int offset, IFormatterResolver formatterResolver, out int readSize)
    {
        to.Clear(); // do clear

        var itemFormatter = formatterResolver.GetFormatter<T>();
        var overwriteFormatter = itemFormatter as IOverwriteMessagePackFormatter<T>;
        // prefer overwrite formatter...
        if (overwriteFormatter != null)
        {
            to.Add(overwriteFormatter.DeserializeTo());
        }
        else
        {
            to.Add(itemFormatter.Deserialize());
        }
    }
}

[westclif]

Love it, although i think like that this will be a bunch of work to implement for each of the current formatters. Especially the ones who Work with Emit and ExpressionTrees (the more generic ones)

Thats why i thought it would be easier to just add an additional ref parameter to the current impelementation and either use it or not if its null. Still the generic formatters would need a bit more work, to figure out the sub fields/members in an object hierachy and if they could be deserialized to through references. (to make it easier and prob faster in terms of performance? / Maybe more troublesome with multithreaded apps though)

[westclif]

For my own purposes ive currently created a kindof wrapper like this:

Im able to register "reference" deserializer for the types i where i need them, and the others just fall back to the default implementation, wrapped through a simple class and an interface which i extend for custom real reference serializers.

Its enough for my current project, in which i fiddle around with unity intern classes, especially particle systems, where the properties are just represented through some kind of struct which reflects the c++ data and can only be accessed by reference.

Also this kind of reference just works with objects/classes right now, or those special unity intern structs. So for value types i would need to use real ref params. But i expect the biggest benefit from classes, so thats fine for me right now. As value types normally allocate and free their memory fast enough on the stack / cache.

Still would love to see a deeper intergration of a similiar feature at some point. :)

public static class ReferenceFormatterCache
{
    private static readonly Dictionary<Type, object> customReferencFormatter = new Dictionary<Type, object>()
    {
        { typeof(ParticleSystem),ParticleFormatter.BaseParticleFormatter.Instance },
        { typeof(ParticleSystem.CollisionModule),ParticleFormatter.CollisionModuleFormatter.Instance }
    };

    public static IMessagePackReferenceFormatter<T> getReferenceFormatter<T>(IMessagePackFormatter<T> formatter)
    {
        IMessagePackReferenceFormatter<T> referenceFormatter = null;
        object objectReferenceFormatter = null;
        if (customReferencFormatter.TryGetValue(typeof(T), out objectReferenceFormatter))
            referenceFormatter = (IMessagePackReferenceFormatter<T>)objectReferenceFormatter;
        if (referenceFormatter == null)
        {
            referenceFormatter = new ReferenceFormatter<T>(formatter);
            customReferencFormatter.Add(typeof(T), referenceFormatter);
        }
        return referenceFormatter;
    }
}

public class CachedCurseFormatter<T>
{
    public static readonly IMessagePackFormatter<T> formatter;
    public static readonly IMessagePackReferenceFormatter<T> referenceFormatter;

    public static T Deserialize(ref byte[] bytes, ref int offset, IFormatterResolver formatterResolver)
    {
        int readSize = 0;
        T value = formatter.Deserialize(bytes, offset, formatterResolver, out readSize);
        offset += readSize;
        return value;
    }

    static CachedCurseFormatter()
    {
        formatter = CurseResolver.Instance.GetFormatterWithVerify<T>();
        referenceFormatter = ReferenceFormatterCache.getReferenceFormatter(formatter);
    }
}

public interface IMessagePackReferenceFormatter<T>
{
    int Serialize(ref byte[] bytes, int offset, T value, IFormatterResolver formatterResolver);
    T Deserialize(ref byte[] bytes, ref int offset, T @object, IFormatterResolver formatterResolver);
}

public class ReferenceFormatter<T> : IMessagePackReferenceFormatter<T>
{
    public readonly IMessagePackFormatter<T> formatter;

    public ReferenceFormatter(IMessagePackFormatter<T> _formatter)
    {
        formatter = _formatter;
    }

    public T Deserialize(ref byte[] bytes, ref int offset, T value, IFormatterResolver formatterResolver)
    {
        int readSize = 0;
        value = formatter.Deserialize(bytes, offset, formatterResolver, out readSize);
        offset += readSize;
        return value;
    }

    public int Serialize(ref byte[] bytes, int offset, T value, IFormatterResolver formatterResolver)
    {
        return formatter.Serialize(ref bytes, offset, value, formatterResolver);
    }
}

[neuecc]

Wrapper?
Overwrite (Merge) and single deserialization are different.

• Do not create new objects
  
  
  
  • This omits the constructor call (even though it is not just an empty constructor)
  
  
  • Instead, new object creation can be avoided
  
  
• Some objects can not implement overwrite
  
  
  
  • ImmutableObject (Primitives, ImmutableCollections, etc ...)
  
  
  • In that case, not to not supported, just avoid creating it and assigning it to ref?
  
  

[westclif]

Thats why for my example i implemented the overwrite just for the specific cases in which i needed it.

public sealed class CollisionModuleFormatter : IMessagePackReferenceFormatter<ParticleSystem.CollisionModule>
    {
        public static readonly IMessagePackReferenceFormatter<ParticleSystem.CollisionModule> Instance = new CollisionModuleFormatter();

        public int Serialize(ref byte[] bytes, int offset, ParticleSystem.CollisionModule value, global::MessagePack.IFormatterResolver formatterResolver)
        {
            var startOffset = offset;

            offset += global::MessagePack.MessagePackBinary.WriteArrayHeader(ref bytes, offset, 3);
            offset += CachedCurseFormatter<float>.formatter.Serialize(ref bytes, offset, value.bounceMultiplier, formatterResolver);
            offset += CachedCurseFormatter<bool>.formatter.Serialize(ref bytes, offset, value.enabled, formatterResolver);
            offset += CachedCurseFormatter<float>.formatter.Serialize(ref bytes, offset, value.colliderForce, formatterResolver);

            return offset - startOffset;
        }

        public ParticleSystem.CollisionModule Deserialize(ref byte[] bytes, ref int offset, ParticleSystem.CollisionModule value, IFormatterResolver formatterResolver)
        {
            var readSize = 0;
            var startOffset = offset;
            var length = global::MessagePack.MessagePackBinary.ReadArrayHeader(bytes, offset, out readSize);
            offset += readSize;

            value.bounceMultiplier = CachedCurseFormatter<float>.Deserialize(ref bytes, ref offset, formatterResolver);
            value.enabled = CachedCurseFormatter<bool>.Deserialize(ref bytes, ref offset, formatterResolver);
            value.colliderForce = CachedCurseFormatter<float>.Deserialize(ref bytes, ref offset, formatterResolver);
            return value;
        }
    }

I'll expect it to be more trouble then what i imagined especially as my focus was just on unity3d projects.
But yeah assigning it by ref sounds good.

[neuecc]

I've finished API surface.

// IOverwriteMessagePackFormatter requires fallback standard formatter
public interface IOverwriteMessagePackFormatter<T> : IMessagePackFormatter<T>
{
    void DeserializeTo(ref T to, byte[] bytes, int offset, IFormatterResolver formatterResolver, out int readSize);
}

public static class IMessagePackFormatterExtensions
{
    // helper extension, use overwrite or create new
    public static void DeserializeTo<T>(this IMessagePackFormatter<T> formatter, ref T to, byte[] bytes, int offset, IFormatterResolver formatterResolver, out int readSize)
    {
        var overwriteFormatter = formatter as IOverwriteMessagePackFormatter<T>;
        if (overwriteFormatter != null)
        {
            overwriteFormatter.DeserializeTo(ref to, bytes, offset, formatterResolver, out readSize);
        }
        else
        {
            to = formatter.Deserialize(bytes, offset, formatterResolver, out readSize);
        }
    }
}

public sealed class SimpleModelFormatter : IOverwriteMessagePackFormatter<SimpleModel>
{
    // ommit IMessagePackFormatter<T> codes...

    public void DeserializeTo(ref SimpleModel value, byte[] bytes, int offset, IFormatterResolver formatterResolver, out int readSize)
    {
        // omit common codes..

        switch (key)
        {
            case 0:
                // if field, we can use direct ref(fast).
                formatterResolver.GetFormatterWithVerify<Vector3>().DeserializeTo(ref value.Transform, bytes, offset, formatterResolver, out readSize);
                break;
            case 1:
                // if property, requires copy local.
                var __MyProperty__ = value.MyProperty;
                formatterResolver.GetFormatterWithVerify<MyClass>().DeserializeTo(ref __MyProperty__, bytes, offset, formatterResolver, out readSize);
                value.MyProperty = __MyProperty__;
                break;
            default:
                readSize = global::MessagePack.MessagePackBinary.ReadNextBlock(bytes, offset);
                break;
        }
    }
}

// collection sample
public sealed class ArrayFormatter<T> : IMessagePackFormatter<T[]>, IOverwriteMessagePackFormatter<T[]>
{
    // omit serialize, deserialize codes...

    public void DeserializeTo(ref T[] array, byte[] bytes, int offset, IFormatterResolver formatterResolver, out int readSize)
    {
        if (MessagePackBinary.IsNil(bytes, offset))
        {
            array = null;
            readSize = 1;
        }
        else if (array == null)
        {
            array = Deserialize(bytes, offset, formatterResolver, out readSize);
        }
        else
        {
            var startOffset = offset;
            var formatter = formatterResolver.GetFormatterWithVerify<T>();

            var len = MessagePackBinary.ReadArrayHeader(bytes, offset, out readSize);
            offset += readSize;

            // clear collection
            if (array.Length != len)
            {
                var newArray = new T[len];
                array = newArray;
            }
            else
            {
                Array.Clear(array, 0, array.Length);
            }

            var overwriteFormatter = formatter as IOverwriteMessagePackFormatter<T>;
            if (overwriteFormatter != null)
            {
                for (int i = 0; i < array.Length; i++)
                {
                    overwriteFormatter.DeserializeTo(ref array[i], bytes, offset, formatterResolver, out readSize);
                    offset += readSize;
                }
            }
            else
            {
                for (int i = 0; i < array.Length; i++)
                {
                    array[i] = formatter.Deserialize(bytes, offset, formatterResolver, out readSize);
                    offset += readSize;
                }
            }

            readSize = offset - startOffset;
        }
    }
}

// Tips, ref property is good to use DeserializeTo
Vector3 myProperty;
public ref Vector3 MyProperty { get { return ref myProperty; } }

There are still additional requirements.
There are cases of only Deserialize or DeserializeTo only

• MonoBehaviour(DeserializeTo only)
• Immutable Struct(Deserialize only)
  How should I configure it?

[Flags]
public enum GeneratingMethods
{
    Both = Deserialize | DeserializeTo,
    Deserialize = 1,
    DeserializeTo = 2,
}

[MessagePackObject(GeneratingMethods.Both)]
public class MyClass
{
    // ...props
}

What value should be set the default of "GeneratingMethods"?

Both, DeserializeTo -> It brings unexpected results to what is assumed to be using ImmutableObject or constructor.
Deserialize -> Explicit settings are necessary for things that can only be used with overwriting (such as MonoBehaviour)

There is also a way to change it for each type of object.
with empty constructor -> Both
with parameter constructor -> Deserialize

I think that this is good, but it is somewhat off from simple rules.

[neuecc]

Protocol Buffer's Merge is add for collection, not override.
Which behavior is preferred? or implements both?(how to configure?)

[neuecc]

concept of configuration per resolver.
default is merge.

public class MessagePackConfiguration : IResolverConfiguration, IMessagePackFormatter<IResolverConfiguration>
{
    public static readonly MessagePackConfiguration Default = new MessagePackConfiguration(CollectionDeserializeToBehaviour.Merge);

    public CollectionDeserializeToBehaviour CollectionDeserializeToBehaviour { get; private set; }

    public MessagePackConfiguration(CollectionDeserializeToBehaviour collectionDeserializeToBehaviour)
    {
        this.CollectionDeserializeToBehaviour = collectionDeserializeToBehaviour;
    }

    int IMessagePackFormatter<IResolverConfiguration>.Serialize(ref byte[] bytes, int offset, IResolverConfiguration value, IFormatterResolver formatterResolver)
    {
        throw new NotSupportedException("IMessagePackFormatter method is dummy, please call AsConfiguration().");
    }

    IResolverConfiguration IMessagePackFormatter<IResolverConfiguration>.Deserialize(byte[] bytes, int offset, IFormatterResolver formatterResolver, out int readSize)
    {
        throw new NotSupportedException("IMessagePackFormatter method is dummy, please call AsConfiguration().");
    }
}

public interface IResolverConfiguration
{
    CollectionDeserializeToBehaviour CollectionDeserializeToBehaviour { get; }
}

public enum CollectionDeserializeToBehaviour
{
    Merge,
    Overwrite,
}

public static class ResolverConfigurationExtensions
{
    public static IResolverConfiguration GetConfiguration(this IFormatterResolver formatterResolver)
    {
        var configuration = formatterResolver.GetFormatter<IResolverConfiguration>();
        return (configuration == null) ? MessagePackConfiguration.Default : configuration.AsConfiguration();
    }

    public static IResolverConfiguration AsConfiguration(this IMessagePackFormatter<IResolverConfiguration> configurationFormatter)
    {
        return (IResolverConfiguration)configurationFormatter;
    }
}

[westclif]

As my main use case is unity specific and though games development. I would prefer either both or the overwrite implementation. To reduce memory allocation as much as possible and also to overwrite unity intern classes.

[neuecc]

Thanks, I'll implement both(and default is changed to overwrite).

[westclif]

Another possiblity would be to provide a function as an additional argument in exchange for the default new() / constructor call which either gets or creates the object.?

[wjax]

@neuecc Hello. Any update on this topic? It is really a must to be able to reuse Objects to reduce GC pressure. Specially in IOT etc

[neuecc]

v2 has biggest changes.
Implements this feature require to finish v2 completely.

[AArnott]

Is it really a requirement that the caller specify the exact object to deserialize to? Or would it be enough if there was an Pool<T> from which objects could be reused? So you can deserialize from nothing (instantiating new objects), then when you're done with those objects, you can recycle them by adding them to the pool. Then deserialize again, and those objects are reused.

[westclif]

Thats sadly not always possible.
Depending on the type of objects we are talking about, we could probably dont even have control over the lifetime of those object we want to de-/serialize.

Thats why i still would recommend an callback/func to be able to configure which object to target. (This way you could still put a pooling system behind this)

And by default the func could just create a default instance of the object to deserialize into.

From my point of view, thats probably the most universal approach and would still be pretty fast.

[AArnott]

Adding a delegate invocation to this path would impact perf pretty significantly at least in some cases, actually. The most prominent perf cost in deserialization already is simply an interface invocation (accounting for about 33% of deserialization time in our perf traces). Adding a delegate invocation would likely double that cost.

If you can't control the lifetime of the object, how could you possibly ever want to recycle it? If it's ready to be reused in deserialization, you put it into the pool. If you don't know whether it's still in use, you'd better not put it in the pool. You could replace "put it into the pool" with "return it from your delegate" and the problem is the same: if you don't know it's lifetime, you can't recycle it.

[wjax]

I do not know exactly about the best option. But a deserialiation solution
should allow for an allocation free strategy. Garbage collector should be
avoided.

Protobuf-net follows a pooling strategy that seems quite interesting. Maybe
borrow the idea?

El vie., 10 may. 2019 20:46, Andrew Arnott [e-mail]
escribió:

Adding a delegate invocation to this path would impact perf pretty
significantly at least in some cases, actually. The most prominent perf
cost in deserialization already is simply an interface invocation
(accounting for about 33% of deserialization time in our perf traces).
Adding a delegate invocation would likely double that cost.

If you can't control the lifetime of the object, how could you possibly
ever want to recycle it? If it's ready to be reused in deserialization, you
put it into the pool. If you don't know whether it's still in use, you'd
better not put it in the pool. You could replace "put it into the pool"
with "return it from your delegate" and the problem is the same: if you
don't know it's lifetime, you can't recycle it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/neuecc/MessagePack-CSharp/issues/107#issuecomment-491393147,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACLVHUV7JVF57HDA6L2SMDPUW7KFANCNFSM4DXPPSXA
.

[westclif]

@AArnott again, i'm mostly talking about unity.
Here we have the chance to use IL2CPP to neglate the performance impact.

On the other Hand. Lambdas dont really look to bad to me regarding performance (reference https://jacksondunstan.com/articles/3605)
But to be honest this depends on the Runtime/Compiler. You could Also work with Kindof Object Getter Factory Classes/Interfaces.

Anyway i dont need to recycle my objects at all. I just want to override or merge my serialized data into their data/fields...
The memory allocation of those objects is handeled on c++ side and marshalled to be able to access them from c# side.
Thats how Unity will probably handle most their objects in the future. (Data-Orientated-Programming (DOTS/ECS) / Internal System like the Particle System/Mesh Renderer etc)

I can for sure somehow work around this but some of those objects i want to deserialize my stuff into subobjects which can only exist inside a particle system for example.
(https://docs.unity3d.com/ScriptReference/ParticleSystem.html)
If you open up the link, you see multiple properties/structs which actually reflect data handled on c++ side.

Thats there i need to explictily specify my deserialization target.
I could for sure deserialize and serialize the "whole" particlesystem and could work with pooling, but there is to much stuff happening on c++ side which could lead to performance impacts as soon as multiple fields are modified (which wouldnt be necessary in that case)

I also could keep a secondary pool which simply reflects the unity internal classes, and assign the values back to unity after the deserializen progress, to at least avoid the memory allocation. But i still would have the overhead of assigning all fields 2 times. Once in the deserialization to my pooled object and once back to the unity managed object. (Thats what i'm already doing btw)

Maybe i even missed the point why you asked about the pooling solution?
If thats the case. Sorry

#211 seems to be a solution i can use as well

[AArnott]

Interesting use case.

The reason I brought up pooling is because I thought the goal was to avoid/reduce allocations during deserialization. We already use pooling during deserialization internally, so pooling of publicly observable objects seemed like a natural next step. If reducing GC pressure were your only goal, I think object pooling would fit well here.

But ya, if you want to deserialize directly to specific existing objects, I agree pooling wouldn't be the best fit. That said, it's also a very unique requirement -- one that I don't know of any deserializer that would support. I'm not sure it's the best justification for making a general purpose msgpack library that much more complex. But when/if I am the one to look into this, I'll see if it can fit nicely into the design.

[rikimaru0345]

That said, it's also a very unique requirement -- one that I don't know of any deserializer that would support.

Ceras does. It supports that and pretty much every variation of pooling imaginable.
Take a look: Configuration Examples

Overwriting a given object, recycling objects, all sorts of pooling scenarios...
It is possible to integrate the user given function very tightly with the rest of the code.
Like taking the method the user provides and compiling a call right into the generated code, instead of jumping through 2 (or more) hoops by calling a user given delegate. That allows you to reduce the number of delegate calls!
See here: ConstructBy API

Disclaimer:
Even though I use Ceras for most things, I'm still very much interested in seeing MessagePack-CSharp supporting those features as well!
That is because Ceras is highly "specialized" to .NET and does not have any official "protocol" it implements. So creating a Ceras implementation in other language (JS, Python, ...) isn't really possible in the foreseeable future. That's why I continue to rely on MessagePack-CSharp for when I need to communicate with systems written in other languages.

I'm not sure it's the best justification for making a general purpose msgpack library that much more complex. But when/if I am the one to look into this, I'll see if it can fit nicely into the design.

I'm certain that it will require a pretty in-depth rework if you want to support the full gamut of this feature.
It's definitely not straightforward once you consider that pooling is inherently intertwined with the concept over overwriting.
Just some food for though here: Let's say after you're done deserializing an object (A) it returns to the pool but it might still hold references to objects (B & C) that are still used! So you can only reuse/overwrite object A, but not B or C.
And then what if the data says that the field A.B should be null, how/where would you even return B back to its pool? Does it have its own pool or will it become garbage, waiting for the GC to collect it? ...
(Ceras deals with that by having a DiscardObjectMethod that's called when an object is not needed anymore, but that has some disadvantages as well...)

It is definitely a very complex topic once you get into it. But I think it is worth it. :+1: from me :smile:

Maybe take some inspiration from how I dealt with all of this in Ceras when/if you decide to spend some time on it.

[neuecc]

Merge/Overwrite API sometimes exists in serializers.
For example, Unity has static void FromJsonOverwrite (string json, object objectToOverwrite) API.
https://docs.unity3d.com/Manual/JSONSerialization.html
My proposal API(DeserializeTo)'s use-case is based on it and I agree it is useful.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[blankensteiner]

Is there an ETR on this feature?

[AArnott]

not yet.

[digital-synapse]

+1 on this feature,
Games generally don't seem to use POCO's for state synchronization (Although they should whenever possible imo, many times the chosen engine doesn't support it or the underlying design prevents it from being practical to do so)

[AArnott]

@digital-synapse tip: please "thumbs up" the issue description to "+1" a feature. It's far easier to assess overall popularity of a request that way.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.