Messagepack-csharp: Unity & Messagepack v2.x Error on some Android devices.

[odysoftware]

Bug description

Just built a Android version with the new message pack v2.x on an Android device (iOS is working fine - both using IL2CPP). As soon as I call this line

MessagePackSerializer.Deserialize<CharInfo>(_bytes);

It crashes with the following error:

Plugins: Couldn't open advapi32, error: dlopen failed: library "advapi32" not found

Repro steps

Tested on an Android 6.0 and 8.0 device with different API Levels 23 and 27 crashed, 26 was fine - so it is also not related to any specific Android API Level or Android Version...

Expected behavior

Should not crash.

Actual behavior

Program crashes.

• Version used: Unity 2019.2.16f1 with MessagePack v2.0.323 (unity package)
• Runtime: Android + IL2CPP (API compatibility level .NET Stadard 2.0)

[AArnott]

Can you please include the full callstack from the exception?

[odysoftware]

That is all information I can get when debugging with Android device connected to unity. This is a warning message from unity and right after that message the android device is saying "App is closed" - no more information given. I guess I need to check if I can get more info via adb log from android device itself - ill check that.

[AArnott]

Ya, anything else you can give would be great. Messagepack has no direct reference to advapi32 so it's probably something else that's failing.

[odysoftware]

So with Android and symbols the best I can get is:

Native StackTrace:

Thread 0 (crashed)
0   libunity.so                        0x00000000c78b4b5a CrashReporting::Android::SignalHandler(int, siginfo*, void*) 
1   libunity.so                        0x00000000c718e60e NativeRuntimeException::signal_handler(int, siginfo*, void*) 
2   libc.so                            0x00000000e7472af4 <system symbols missing> 
3   libil2cpp.so                       0x00000000c0fa2a90 Unsafe_ReadUnaligned_TisInt64_t7A386C2FF7B0280A0F516992401DDFCF0FF7B436_mCF953FD610E15812D1063A3C6501EB7BC720AE2B_gshared 
4   libil2cpp.so                       0x00000000c38275d4 SequenceReaderExtensions_TryRead_TisInt64_t7A386C2FF7B0280A0F516992401DDFCF0FF7B436_mE7C293597ACEE090A9913E99B2EABBC0507CAE61_gshared_inline 
5   libil2cpp.so                       0x00000000c3822764 SequenceReaderExtensions_TryReadReverseEndianness_m86440CA009B364528CBFBE9C1E9831D7E9D43DC0 
6   libil2cpp.so                       0x00000000c38220c0 SequenceReaderExtensions_TryReadBigEndian_m84CF325B0F9ED1C4BC4D63D22C7E56F9E035E94E 
7   libil2cpp.so                       0x00000000bff1f970 MessagePackReader_ReadDateTime_m555B60F62286B8A8E326471FDA31B585E954F0ED 
8   libil2cpp.so                       0x00000000bff1e020 MessagePackReader_ReadDateTime_m06C1CE1BC72B6C5845E1321506DE1C50ED322CD9 
9   libil2cpp.so                       0x00000000c0ce4cdc DateTimeFormatter_Deserialize_m435870F49A04B1E7E28EF07B64A4126408470E8C

It is deserialising stuff, but it appears to be the date time ! Interesting is that when looking for advapi32 errors in google, I can find an recent issue with unity:

https://issuetracker.unity3d.com/issues/unable-to-load-dll-advapi32-dot-dll-when-calling-datetime-dot-now

I have to check but I am using datetime.now too and there I don't have problems... very strange...

[odysoftware]

I can confirm using DateTime.Now does not cause any problems here... - very strange... so it must be related to how message pack is using it somehow... -> Unsafe_ReadUnaligned_TisInt64 ?

[AArnott]

Thanks. This is helpful. Where did you get your Unsafe.dll? I wonder if you're shipping one with your Android app that is compiled for Windows.

[neuecc]

Stack trace is mentioned Unsafe.ReadUnaligned<long> was crashed.
I think Unsafe.dll is managed dll, it is no dependent platform.

[AArnott]

@neuecc Are you saying that advapi32.dll is unrelated?

[neuecc]

@AArnott
I guess it's a Unity IL2CPP VM issue.
In that case, the details are a black box,
so you have to create a reproduction procedure and report a bug to Unity.

[neuecc]

Unsafe.dll is not usually used in Unity, so it's no wonder bugs exist in the IL2CPP VM.
However, if this always happens, Details and workarounds must be sought.

However, I and several companies have already tried v2 on Android,
I have not received a similar report.
Therefore, we cannot assert that there is a problem.

[odysoftware]

Well unsafe is checked in unity for Android and since it’s working on a few devices and on iOS it seems to be fine overal. Maybe you can reproduce it with some datetime in messagepack classes because that is where it crashes but it definitely a combination from messagepack and unity. The crash happens in deserialize and i have now moved back to v1.8 - all fine again- because many had this crash with several different Android device new and old. 😥

[odysoftware]

I can try to look what exactly in my class using message pack is causing this, that should help for reproduction.

Maybe it’s related to this same unity issue https://issuetracker.unity3d.com/issues/unable-to-load-dll-advapi32-dot-dll-when-calling-datetime-dot-now

[odysoftware]

What I can see is that it is only happening for the datetime type...

e.g.

[Key(0)]
DateTime lastVisited {get; private set;}

public Test(DateTime _lastVisit)
                {
                    LastVisit = _lastVisit;
                }

So as soon as message pack is trying to deserialise the datetime in unity via this unsafe.dll - that is the point where it is fatal crashing on some devices. Removing this DateTime prevents it from crashing.

Maybe this helps in reproduction...?

From MessagePackGenerated.cs (that where it seems to crash...)

 __lastVisit__ = formatterResolver.GetFormatterWithVerify<global::System.DateTime>().Deserialize(ref reader, options);

[neuecc]

Since DateTime.Now is not used, I don't think it has a direct effect, but I think it is a similar problem.
Unity says Fixed in 2019.3, and you're trying in 2019.2,
Could you check in 2019.3?

[odysoftware]

Well its not a small project, but I can try.

What is still interesting though is, that if you say several companies have already v2 in use - don't they use any datetime fields in messagepack - because if they would have them they should also experience those crashes on a lot of android devices...

[odysoftware]

Well while surprisingly the upgrade to v2019.3.0f4 went very smooth - the error is still the same - so I guess we need a workaround or a fix for this datetime deserialisation? I just wonder that no other is reporting this, am I the only one with datetime in message pack v2 or with a few android devices where it crashes?

Thread 0 (crashed)
0   libil2cpp.so                       0x00000000c3eb8ee4 Unsafe_ReadUnaligned_TisInt64_t7A386C2FF7B0280A0F516992401DDFCF0FF7B436_mCF953FD610E15812D1063A3C6501EB7BC720AE2B_gshared 
1   libil2cpp.so                       0x00000000c75bdd2c SequenceReaderExtensions_TryRead_TisInt64_t7A386C2FF7B0280A0F516992401DDFCF0FF7B436_mE7C293597ACEE090A9913E99B2EABBC0507CAE61_gshared_inline 
2   libil2cpp.so                       0x00000000c75ba138 SequenceReaderExtensions_TryReadReverseEndianness_m86440CA009B364528CBFBE9C1E9831D7E9D43DC0 
3   libil2cpp.so                       0x00000000c75b9ca8 SequenceReaderExtensions_TryReadBigEndian_m84CF325B0F9ED1C4BC4D63D22C7E56F9E035E94E 
4   libil2cpp.so                       0x00000000c3d1705c MessagePackReader_ReadDateTime_m555B60F62286B8A8E326471FDA31B585E954F0ED 
5   libil2cpp.so                       0x00000000c3d161d4 MessagePackReader_ReadDateTime_m06C1CE1BC72B6C5845E1321506DE1C50ED322CD9 
6   libil2cpp.so                       0x00000000c4dd81b0 DateTimeFormatter_Deserialize_m435870F49A04B1E7E28EF07B64A4126408470E8C 
7   libil2cpp.so                       0x00000000c5310dec ActionFormatter_Deserialize_m1EFDD2815D686214B3DB7CF396716282F18B95BD 
8   libil2cpp.so                       0x00000000c6402f34 ListFormatter_1_Deserialize_mFBFE782477C4EFBA078A5B8CC5FF848482BA5DBD_gshared 
9   libil2cpp.so                       0x00000000c53351e8 CharInfoFormatter_Deserialize_m38C56A9000B191F97DE4C84D705A2C24170993E9 
10  libil2cpp.so                       0x00000000c45ff888 MessagePackSerializer_Deserialize_TisRuntimeObject_m8F20CC9966B8B0CBB100919FCB5475F96BC7CC22_gshared 

My current device for reproduction is a Samsung-T580 tablet with Android OS 8.1/API-27 but many more reported this... -> one Android 9 and one Android 6.0 device. A working device for example is a Huawei mobile phones (Huawei P10 lite)...

[AArnott]

So is it unrelated to advapi32.dll then?
Given a crash is in the Unsafe assembly, we should file a bug against dotnet/runtime as well as they probably need to fix this (and we might workaround it in the meantime).

[odysoftware]

Well I am not sure if it is unrelated because before the crash it is reporting this warning ⚠️ in unity’s debug log.

[odysoftware]

Okay now I can provide even more information. The Android devices where it does not crash are the ARM64 devices!

I just did a test on an Android Huawei phone, where it was working fine by compiling it with unity to ARMv7 ! After that it is also crashing on this device (!).

So message pack v2 DateTime deserialisation is working on Android ARM 64 devices (as well as iOS devices) but not on Android ARM v7 devices.

I was checking the message pack source code - sadly I have no experiences on this low level - but the error must rely in this unsafe read ulong (int64) stuff (as seen in stack trace) on non 64 bit android devices. But you should be able to reproduce it now by deserializing a Message pack datetime on any android device which is deployed with unity (.net Standard 2.0 and IL2CPP) and with ARMv7 only.

Hope you can fix it as I would love to switch back to v2.0.

Here is the source code from .NET Core about this Unsafe.ReadUnaligned (Line 34 where it crashes on ARMv7/Android)
https://source.dot.net/#System.Memory/System/Buffers/SequenceReaderExtensions.Binary.cs
So either a .net core or il2cpp issue...

[odysoftware]

Just asking if there is anything planned in order to fix this on your side or how we should proceed? Otherwise I guess we have to stay on v1.80 or we have to remove datetime as a property and switch it to a long and do manual conversions.

Thanks in advance for your time.

[AArnott]

I don't think we have any idea yet what we would need to change to workaround this. But I still think it's worthwhile to file a bug against dotnet/runtime as this feels like a bug there.

[odysoftware]

I see but well then I think people need to be aware that using the datetime on ARMv7 devices with message pack v2 and unity will lead to problems. I'll move back then to 1.80 for the client and wait for future releases/fixes.

[Durnev]

I have a little update. I was able to reproduce just for double:

double data = 0;
var bytes = MessagePackSerializer.Serialize(data);
MessagePackSerializer.Deserialize<double>(bytes);

Сrash is stable reproduced for Unity ARMv7 builds. It is NOT depends on advapi32.dll

Crash stacktrace is:

SequenceReaderExtensions_TryRead_TisInt64
SequenceReaderExtensions_TryRead_TisInt64
SequenceReaderExtensions_TryReadBigEndian
MessagePackReader_ReadDouble

Can you help me with repro project for this problem which i can send to the Unity? I can't extract using of Unsafe_ReadUnaligned_TisInt64 without any MessagaPack stuff.

[AArnott]

@Durnev, are you saying that you want a repro that doesn't include messagepack if you can get it? I don't have any ARMv7 devices handy to try to develop a repro, sadly.
At least, as far as I know. I have an Android phone. Does that count?

[odysoftware]

Yeah it is easily reproducable, but I also have no idea on how to extract the Unsafe_ReadUnaligned_TisInt64 for the il2cpp compiler - so that we can send unity an example... Maybe its enough to tell them that Unsafe_ReadUnaligned_TisInt64 is crashing ARMv7 builds?

[Durnev]

@AArnott

At least, as far as I know. I have an Android phone. Does that count?

Yes, all you need is Unity and ARMv7 build. You can setup it in Player settings for Android:

But, on the other hand i was able to create repro only with Unsafe_ReadUnaligned_TisInt64.

const long startValue = 0;
var longBytes = BitConverter.GetBytes(startValue); 
var span = new ReadOnlySpan<byte>(longBytes);
var longValue = Unsafe.ReadUnaligned<long>(ref MemoryMarshal.GetReference(span));
Debug.Log(longValue == 0);

I use long because SequenceReaderExtensions.TryReadBigEndian uses it for double.

public static unsafe bool TryReadBigEndian(ref this SequenceReader<byte> reader, out double value)
{
    if (TryReadBigEndian(ref reader, out long longValue))
    {
        value = *(double*)&longValue;
        return true;
    }

    value = default;
    return false;
}

And for this repro crash does not reproduce! So now i'm not sure that the problem is in Unsafe.ReadUnaligned<long>.

Also, i made a few other repro examples, where i serialize diferent types like long, byte, int and deserialize it like double.

int data = 15;
var bytes = MessagePackSerializer.Serialize(data);
var newValue = MessagePackSerializer.Deserialize<double>(bytes);
Debug.Log("New value is " + newValue);

And crash does not reproduce! It reproduce only when i serialize double and deserialize double. So maybe problem is in combo of Serialize and Deserialize methods, but i'm not sure.

I'll continue my research and will keep you informed.

[Durnev]

I can’t even reproduce it for this code. I changed some access modifiers for clases and methods here.

const double startValue = 42;

var span  = new Span<byte>(new byte[8]);
MessagePackWriter.WriteBigEndian(startValue, span);
var bytes = span.ToArray();

var reader = new SequenceReader<byte>(bytes);
reader.TryReadBigEndian(out double result);
Debug.Log("Result is " + result);

Any ideas?

[neuecc]

maybe it is memory alignment issue.

[Durnev]

I found out that crash appears if we call TryRead on SequenceReader to read byte with MP code.

const double startValue = 42;
var span = new Span<byte>(new byte[9])
{
    [0] = MessagePackCode.Float64
};
MessagePackWriter.WriteBigEndian(startValue, span.Slice(1));
var bytes = span.ToArray();

var reader = new SequenceReader<byte>(bytes);
reader.TryRead(out _); // HERE 
reader.TryReadBigEndian(out double result);

I replaced double with long and it also crashed. I think problem is in SequenceReader.

It can be memory alignment issue, but my experience is not enough to continue research.

Also, i have a problems with debugging code in SequenceReader.

[odysoftware]

Wow great research here :) and finally we can hope for a fix sooner or later in this regard, so people can use 2.0 with unity. I still wonder why there are not a lot more people having issues with messaepack 2.0 and unity - because ARMv7 is still heavily used and this seems to be a general problems with datetime and doubles...

Anyway I hope someone can catch it now!

[neuecc]

Currently supporting 64bit architecture is must on upload to Google Play Store.
Most of me and my customers have already not support ARMv7.
Therefore, the priority was not high.
The solution is to replace this with managed code.

[Durnev]

@neuecc We develop the game with 700к daily active users on Android. I prepare statistic about CPU architectures distribution in it:

17-18% of ARMv7 it's also quite a lot.

Also, i sent repro project to Unity. I'll keep you informed about it.

[odysoftware]

I totally agree ARMv7, even if it outdated, is still in heavy use, as mentioned by Durnev. Yes you need 64bit support, but if you want higher reach there is no way around to also support ARMv7.

In my eyes at the current stage, Message Pack 2.0 is absolutely not usable with unity, due to this issue. Well its no problem for me to stay on old version... but still I think this needs to get fixed or otherwise people need to be aware that Messagepack 2.0 is not working on ARMv7 devices with unity.

[neuecc]

thanks for the details.
I've added PR, I don't confirm yet but maybe fixed.

[AArnott]

@odysoftware can you confirm with the build in #795 PR that it fixes your issue?

Also, can you link to a bug tracking this in Unity if you have one?

[odysoftware]

Hey there, that sounds great (!), I did not have time to try it out yet - have to do migration again. Maybe I can do it later today, then I can give you feedback if it is working.

[odysoftware]

So far everything seems fine, now starting tests on Android !

There was a mistake though in SequenceReaderExtensions.cs when switching unity to Android ->

#if UNITY_ANDROID
    private static unsafe bool TryReadMultisegment(ref SequenceReader<byte> reader, out long value)
        {
            Debug.Assert(reader.UnreadSpan.Length < sizeof(T), "reader.UnreadSpan.Length < sizeof(T)");
.......

This gives an error, you need to change it to:

Debug.Assert(reader.UnreadSpan.Length < sizeof(long), "reader.UnreadSpan.Length < sizeof(T)");

then it works.

[odysoftware]

It didn't help for all problems yet because the datetime is not calling new method so I still have this crash - it may only work for doubles:

Thread 0 (crashed)
0   libil2cpp.so                       0x00000000a06074b0 SequenceReaderExtensions_BitConverterToInt64_m388EBB0EE5EB8A4CBC5A72606CCFBBF07A6D81A2 (MessagePack3.cpp:35063)
1   libil2cpp.so                       0x00000000a0607dfc SequenceReaderExtensions_TryReadReverseEndianness_m86440CA009B364528CBFBE9C1E9831D7E9D43DC0 (MessagePack3.cpp:36934)
2   libil2cpp.so                       0x00000000a0d41f6c MessagePackReader_ReadDateTime_m555B60F62286B8A8E326471FDA31B585E954F0ED (MessagePack2.cpp:23866)
3   libil2cpp.so                       0x00000000a0d41db8 MessagePackReader_ReadDateTime_m06C1CE1BC72B6C5845E1321506DE1C50ED322CD9 (MessagePack2.cpp:23773)
4   libil2cpp.so                       0x00000000a07a82b0 DateTimeFormatter_Deserialize_m435870F49A04B1E7E28EF07B64A4126408470E8C (MessagePack.cpp:18200) 

So we need this also for the other types?

[Durnev]

Hello.
I have answer from Unity support team about our issue.

ARMv7 requires for all reads to be aligned.

In SequenceReaderExtensions.cs​ on line 43, there is this call made Unsafe.ReadUnaligned<T>(ref MemoryMarshal.GetReference(span));​. It gets there from TryReadBigEndian() -> TryReadReverseEndianness() -> TryRead()​.

IL2CPP does the right thing here as it stops the memory being accessed from an unaligned read, while when using mono, that is ignored and the program returns garbage values.

I haven't found a ReadAligned method which would allow the user to read in an aligned way from the memory. The best solution would be implementing ReadAligned by yourself.

So rewrite this code was right solution by @neuecc. We can wait until ARMv7 will be deprecated, but i think it will be not near future.

[neuecc]

solved by #798