Messagepack-csharp: DateTime Kind is UTC but it should be unspecified

[SergeBerwert]

I am having the following issue.
I have an object with a DateTime property. The DateTime is set with no specified TimeZone. The “kind” property is set to unspecified. I am serializing the object like that:
LZ4MessagePackSerializer.Serialize(obj)
After deserializing the DateTime.Kind property is set to UTC. The time did not change in any way.
Deserializing like that:
LZ4MessagePackSerializer.Deserialize<Type>(stream)

[neuecc]

In ReadMe

DateTime is serialized to new MessagePack extension spec proposal, it serialize/deserialize UTC and loses Kind info. If you useNativeDateTimeResolver serialized native DateTime binary format and it can keep Kind info but cannot communicate other platforms.

[neuecc]

now, timestamp format was merged to msgpack spec document.
https://github.com/msgpack/msgpack/blob/master/spec.md#formats-timestamp

[MikaelUmaN]

Hi.

I understand this is how the spec is written and NativeDateTimeResolver works well for C# to C# applications.

But I really need advice on how to get this to work in a cross-platform manner. All our data is sent with datetime timestamps and when I deserialize on the client side suddenly I have shifted the days by 1 day.

I need to support clients that are not .NET based. On server side, I can have a value that is set as:
{1988-04-27 07:00:00} with Kind=Unspecified. But on client side, I get 1988-04-26 00:00:00. In general we allow the user to specify the timezone they want, with the default case of everything being represented in the timezone the contract (whose prices we transmit) belong in.

Is there some workaround that's cross platform we can consider?

[neuecc]

I don't know cross platform TimeZone format except text format.
Currently, it is better to convert to string.

[MikaelUmaN]

But there must be some way I can customize to make this work.

I don't even need timezone info. I just want to send my DateTime with Kind=Unspecified and when I receive it I want the same number of ticks to be present.

Currently, when I send a DateTime with Kind=Unspecified and ticks=636873129000000000, then on the client side I will get back Kind=UTC and ticks=636873093000000000.

The difference in time can be seen from this python snippet:

>>> import pandas as pd
>>> pd.Timestamp((636873129000000000-621355968000000000)/10,unit='us')
Timestamp('2019-03-04 16:15:00')
>>> pd.Timestamp((636873093000000000-621355968000000000)/10,unit='us')
Timestamp('2019-03-04 15:15:00')

Why does the formatter remove 1 hour arbitrarily? It would be fine if it just treated Kind=Unspecified as being in UTC.

If there is some good reason for this, then can you point in the direction of making a custom DateTime formatter that basically just leaves the ticks unchanged? Any help greatly appreciated. Sorry if I am misunderstanding something.

Edit: I found the line in MessagePackBinary:

public static int WriteDateTime(ref byte[] bytes, int offset, DateTime dateTime)
        {
            dateTime = dateTime.ToUniversalTime();

I think this line should only be performed if the kind is "local", not unspecified.

Edit 2: Ok, I just tried creating a custom Formatter here: https://github.com/MikaelUmaN/MessagePack-CSharp/commit/e81b03f790bbabde821961ef36f03a57074aabe4 which seems to work.

Based on the documentation of ToUniversalTime behaviour: https://docs.microsoft.com/en-us/dotnet/api/system.datetime.touniversaltime?view=netframework-4.7.2 I can see why the current code is as it is. DateTimeKind.Unspecified seems like a broken part of .NET Framework...

Unless there is any interest in a real pull request I will just make a custom formatter in my own project. Thanks.

[aleh4d]

We also have the similar situation, time changes after restoring the object,
Is the issue fixed now?

[AArnott]

Why does the formatter remove 1 hour arbitrarily?

It doesn't remove 1 hour arbitrarily. What happens is the writer tries to convert to UTC:

https://github.com/neuecc/MessagePack-CSharp/blob/0242103ebbd0123fc4072357b04da78cd78e0976/src/MessagePack/MessagePackBinary.cs#L2246-L2248

When the value being written is either local or UTC, it's fine. But when it is Unspecified, .NET's behavior is to assume that it is not what's being converted to. So ToLocalTime and ToUniversalTime both will change the "Unspecified" value.

It would be fine if it just treated Kind=Unspecified as being in UTC.

I think that might be more reasonable as well. I'll send out a PR and see what @neuecc thinks.

[aleh4d]

Best would be to use DateTime.ToBinary() internally befrore saving, and FromBinary() on restore, to avoid the time zone question.
Or treat Kind.Unspecified as Local, as in fact that works in C#,
just not serializing Unspecified, makes MessagePack disabled, not capable to serialize simple case with new DateTime value, used by default

[Haihg]

This doesn't only happen with un-specified kind. In my case, the date time kind is local and immediately after de-serialized it becomes utc and ticks are modified. Any explanation why this is happening is excuses as this is clearly mal-functioning (properties changed after de-serializing).

Also DateTime type in C# doesn't have a time zone, so when converting to UTC time it relies on the local system's timezone, which only the developer knows if they want that to happen or not. With this happening, the same data serialized on 1 system and de-serialized on another system with a different time zone will require more work for developer to do conversion between local/utc times. In some cases it requires a lot of work. In fact, this conversion has caused conflict with default date time in C# (which has kind = local) and it changed the actual data behind Datetime, e.g. Ticks. It is also conflict with date time data types in other languages, because in most languages a Datetime values would not include information about timezone either. It should be the job of the developer to decide whether to handle UTC time or not.

[AArnott]

Any explanation why this is happening is excuses

You're right, @Haihg. You're paying us adequately for this otherwise useful library and we are shirking our duties. We're just making excuses now and you should fire us. 👨‍⚖ 🤦‍♂

[AArnott]

@aleh4d, as @neuecc already mentioned you can use NativeDateTimeResolver in order to retain DateTimeKind information if that's what you want.

The messagepack spec itself has a native representation for datetime but it requires representation as UTC (which makes sense, since all handling of datetime should be UTC until the moment it is presented to the end user, for oh so many reasons).

[Haihg]

You're right, @Haihg. You're paying us adequately for this otherwise useful library and we are shirking our duties. We're just making excuses now and you should fire us. 👨‍⚖ 🤦‍♂

Think usage! You can have very good features hidden under the library, but if the default settings for .Net doesn't work correctly on .Net (needs to customise) then it's not easy to use. This is personal experience and I have never blamed you for this. Also the project is open source so I'm commenting for the community, to make it better, NOT meant to target at blaming you!

[AArnott]

Thanks, @Haihg. I agree with your sentiment generally for a .NET library. But when the library implements a protocol that is broader than .NET, where dozens of other messagepack libraries exist across many platforms, our first duty in the library is interop under the protocol the library is named after. Datetime is one of these primitives that exist both in .NET and messagepack and they are unfortunately defined differently. We follow the messagepack spec by default and that does cause a bit of a rift for .NET. To prioritize .NET over messagepack by default is an option of course, but in my opinion the interop sacrifice is worse.

[Haihg]

Thanks, @Haihg. I agree with your sentiment generally for a .NET library. But when the library implements a protocol that is broader than .NET, where dozens of other messagepack libraries exist across many platforms, our first duty in the library is interop under the protocol the library is named after. Datetime is one of these primitives that exist both in .NET and messagepack and they are unfortunately defined differently. We follow the messagepack spec by default and that does cause a bit of a rift for .NET. To prioritize .NET over messagepack by default is an option of course, but in my opinion the interop sacrifice is worse.

Understood. I was under the impression date time default Kind was local but it actually depends on how the date time value is initiated (Local for DateTime.Now, Utc for DateTime.UtcNow, or Unspecified for new DateTime()). Issue is in the .Net DateTime first thing, could have been fixed in MessagePack binary specs though, but becomes impossible now that it's been widely used.