Messagepack-csharp: Cannot serialize an array of objects

[moozzyk]

I am playing with this library and I hit a wall when trying to serialize an array of objects.

Here is my type hierarchy:

    public class Address
    {
        public string Street { get; set; }
    }

    public class Person
    {
        public string Name { get; set; }
        public object[] /*Address*/ Addresses { get; set; }
    }

and here is the object I am trying to serialize:

            var p = new Person
            {
                Name = "John",
                Addresses = new []
                {
                    new Address { Street = "St." },
                    new Address { Street = "Ave." }
                }
            };

I am using the CotractlessStandardResolver like this:

MessagePack.MessagePackSerializer.Serialize(ms, t, MessagePack.Resolvers.ContractlessStandardResolver.Instance);

And the exception I am getting is:

System.InvalidOperationException occurred
  HResult=0x80131509
  Message=Not supported primitive object resolver. type:Address
  Source=<Cannot evaluate the exception source>
  StackTrace:
   at MessagePack.Formatters.PrimitiveObjectFormatter.Serialize(Byte[]& bytes, Int32 offset, Object value, IFormatterResolver formatterResolver)
   at MessagePack.Formatters.ArrayFormatter`1.Serialize(Byte[]& bytes, Int32 offset, T[] value, IFormatterResolver formatterResolver)
   at MessagePack.Formatters.MsgPackTest_PersonFormatter.Serialize(Byte[]& , Int32 , Person , IFormatterResolver )
   at MessagePack.MessagePackSerializer.Serialize[T](Stream stream, T obj, IFormatterResolver resolver)
   at MsgPackTest.Program.Nuecc[T](T t) in C:\Users\pawelka\Source\Repos\MsgPackTest\MsgPackTest\Program.cs:line 179
   at MsgPackTest.Program.ReaderTest() in C:\Users\pawelka\Source\Repos\MsgPackTest\MsgPackTest\Program.cs:line 88
   at MsgPackTest.Program.Main(String[] args) in C:\Users\pawelka\Source\Repos\MsgPackTest\MsgPackTest\Program.cs:line 64

MessagePack-cli serializes this type without errors.

[neuecc]

This is by design.
MessagePack for C# resolve serializer by type.
object is mapped to use PrimitiveObjectResolver, it only supports only primitive messagepack representation like bool, char, sbyte, byte, short, int, long, ushort, uint, ulong, float, double, DateTime, string, byte[], ICollection, IDictionary.

[neuecc]

This is due to the symmetry of serialization and deserialization of Resolver.
If type is object, can serialize but can not deserialize.
MsgPack-Cli's object deserialization is deserialized to MessagePackObject(it is like JObject of JSON.NET, XElement of LINQ to XML).
But MessagePack-CSharp deserialize to primitive of C#, primitive of MsgPack spec.

Avoidance can be avoided by preparing an asymmetric Resolver that can only be serialized.
If you can fill it with your requirement, I will prepare ... ....

[neuecc]

Sorry, I reconsidered based on your opinion.
ContractlessStandardResolver should can serialize everything.

I've implemented new ContractlessStandardResolver.
Maybe the result is your expected.

var p = new Person
{
    Name = "John",
    Addresses = new[]
    {
        new Address { Street = "St." },
        new Address { Street = "Ave." }
    }
};

var result = MessagePack.MessagePackSerializer.Serialize(p, MessagePack.Resolvers.ContractlessStandardResolver.Instance);

// {"Name":"John","Addresses":[{"Street":"St."},{"Street":"Ave."}]}
var jsonDump = MessagePackSerializer.ToJson(result);

var p2 = MessagePack.MessagePackSerializer.Deserialize<Person>(result, MessagePack.Resolvers.ContractlessStandardResolver.Instance);
p2.Name.Is("John");

var addresses = p2.Addresses as IList; // object[]
var d1 = addresses[0] as IDictionary; // Dictionary<object, object>
var d2 = addresses[1] as IDictionary;
(d1["Street"] as string).Is("St.");
(d2["Street"] as string).Is("Ave.");

[neuecc]

I've released 1.3.1 it contains this fix.
Please try it.

[avtc]

Great!

How about little bit more complex object:

    public class Model
    {
        public string Name { get; set; }
        public List<string> Items { get; set; }
        public long Id { get; set; }
        public List<Model> Subs { get; set; }
    }

object[] arr = new object[] { new Model() { Items = new List<string>() {"a", "b"} }, 5 };

var serialized = MessagePackSerializer.Serialize(arr, ContractlessStandardResolver.Instance);
var deserialized = MessagePackSerializer.Deserialize(serialized, ContractlessStandardResolver.Instance);

deserialized[0]["Items"] - is of type object[] instead of List<string>, is it expected behavior?

[avtc]

Sample variant of restoring dto from dictionary, recursive, uses reflection.
It works for proposed Model.
Properties that deserialized to object[] are converted to generic Lists.

a lot of code there

    public static class DictionaryToObjectDeserializer
    {
        public static object ToObject(this IDictionary<object, object> source, Type destType)
        {
            if (destType.IsAssignableFrom(typeof(IDictionary<object, object>)))
                return source;
            object obj = Activator.CreateInstance(destType);
            // fields
            var allFields = GetAllFields(destType);
            foreach(var field in allFields)
            {
                if (source.TryGetValue(field.Name, out object value))
                {
                    // Fix nullables...
                    Type fieldType = Nullable.GetUnderlyingType(field.FieldType) ?? field.FieldType;

                    value = RecursiveConvert(field, value, fieldType);
                    try
                    {
                        field.SetValue(obj, value);
                    }
                    catch
                    {
                        Console.WriteLine($"{field.DeclaringType}.{field.Name} failed to set value {value} of type {value.GetType()}");
                    }
                }
            }
            // properties
            var allProps = GetAllProperties(destType);
            foreach (var prop in allProps)
            {
                if (prop.CanWrite && prop.GetIndexParameters().Length == 0 && source.TryGetValue(prop.Name, out object value))
                {
                    // Fix nullables...
                    Type propType = Nullable.GetUnderlyingType(prop.PropertyType) ?? prop.PropertyType;

                    value = RecursiveConvert(prop, value, propType);
                    try
                    {
                        prop.SetValue(obj, value);
                    }
                    catch
                    {
                        Console.WriteLine($"{prop.DeclaringType}.{prop.Name} failed to set value {value} of type {value.GetType()}");
                    }
                }
            }

            return obj;
        }

        private static object RecursiveConvert(MemberInfo prop, object value, Type propType)
        {
            if (value != null)
            {
                Type valueType = value.GetType();
                if (valueType == typeof(Dictionary<object, object>) && propType != valueType)
                {
                    // recursively resolve dictionary to typed objects
                    value = (value as IDictionary<object, object>).ToObject(propType);
                }
                else if (valueType == typeof(object[]) && propType != valueType)
                {
                    // restore list
                    var genericArguments = propType.GetGenericArguments();
                    if (propType.GetTypeInfo().IsGenericType
                        && typeof(IEnumerable).IsAssignableFrom(propType)
                        && genericArguments.Length == 1)
                    {
                        var listType = typeof(List<>);
                        var genericArgumentType = genericArguments[0];
                        var constructedListType = listType.MakeGenericType(genericArgumentType);
                        var valueAsList = (IList)Activator.CreateInstance(constructedListType);
                        foreach (var item in value as object[])
                        {
                            var convertedItem = RecursiveConvert(prop, item, genericArgumentType);
                            valueAsList.Add(convertedItem);
                        }
                        value = valueAsList;
                    }
                    else
                    {
                        // ??
                        Console.WriteLine($"{prop.DeclaringType}.{prop.Name} failed to set value {value} of type {value.GetType()}");
                    }
                }
                else if (propType.GetTypeInfo().IsEnum)
                {
                    //pass as is
                }
                else if (propType == typeof(Guid) && valueType == typeof(string))
                { 
                    if (Guid.TryParse(value.ToString(), out Guid valueGuid))
                    {
                        value = valueGuid;
                    }
                }
                else
                {
                    value = Convert.ChangeType(value, propType);
                }
            }

            return value;
        }

        public static IEnumerable<FieldInfo> GetAllFields(Type t)
        {
            if (t == null)
                return Enumerable.Empty<FieldInfo>();

            BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.DeclaredOnly;
            return t.GetFields(flags).Concat(GetAllFields(t.GetTypeInfo().BaseType));
        }

        public static IEnumerable<PropertyInfo> GetAllProperties(Type t)
        {
            if (t == null)
                return Enumerable.Empty<PropertyInfo>();

            BindingFlags flags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.DeclaredOnly;
            return t.GetProperties(flags).Concat(GetAllProperties(t.GetTypeInfo().BaseType));
        }

    }

Sample usage:
var model = deserialized[0].ToObject(typeof(Model));

*added fix for enums and Guid

[moozzyk]

@neuecc - Thanks for fixing this quickly. I will be playing with the new version today.

[neuecc]

@avtc

deserialized[0]["Items"] - is of type object[] instead of List, is it expected behavior?

Yes, it is expected behavior.
Beacuse serialized byte[] lose C# type information, the binary information is only messagepack-array.
So MessagePack-CSharp's deserializer deserialize from messagepack-array to object[].

[avtc]

I think ContractlessStandardResolver - should save c# type information /maybe name with namespace (LZ4 should be able to pack it well)/, to allow deserializer to handle restoring to same type.

[neuecc]

should save c# type information

No, it is different problem.
save type information is like BinaryFormatter or NetDataContractSerializer, it will provide another resolver. https://github.com/neuecc/MessagePack-CSharp/issues/12

[avtc]

I am working on it

[aniltkurahatti]

Hi,
I am Getting the Below error while deserializing the complex sorted List in c#.

System.ArgumentException: An entry with the same key already exists. at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource) at System.Collections.Generic.SortedList2.Add(TKey key, TValue value)
at MessagePack.Formatters.SortedListFormatter2.Add(SortedList2 collection, Int32 index, TKey key, TValue value)
at MessagePack.Formatters.DictionaryFormatterBase5.Deserialize(Byte[] bytes, Int32 offset, IFormatterResolver formatterResolver, Int32& readSize) at MessagePack.Formatters.Cgi_Nds_Domain_EventCalendarFormatter1.Deserialize(Byte[] , Int32 , IFormatterResolver , Int32& ) at MessagePack.MessagePackSerializer.Deserialize[T](Byte[] bytes, IFormatterResolver resolver) at MessagePack.MessagePackSerializer.Deserialize[T](Byte[] bytes)

For the below value and code I have used#

Class Created.

[MessagePackObject]
    public class EventCalendar
    {
        [Key(0)]

        public SortedList<DateTime,SortedList<int,SortedList<int,SortedSet<int>>>> _calendar { get; set; }

    }

Input value

var compressedValue=""H4sIAAAAAAAEAJvYdO1/FJPVh26GNuaJurwTdcUn6ipO1NWeqGs6UZcRixgTFjFmLGIsWMRYW5lRBdhakBVxYtHChUWMG4sYTzNcDOKhZp4WZHleLHr4sIgBAG/6BkIQAQAA""

Method

var _memByteData = JsonConvert.DeserializeObject<byte[]>(compressedValue); using (var inStream = new MemoryStream(_memByteData)) using (var bigStream = new GZipStream(inStream,CompressionMode.Decompress)) using (var bigStreamOut = new MemoryStream()) { bigStream.CopyTo(bigStreamOut); outPut = bigStreamOut.ToArray(); } var cal = MessagePackSerializer.Deserialize<EventCalendar>(outPut);

Note: I have used GZipStream and Newtonsoft Json as well

THis stopping our activity. Kindly provide your input