Newtonsoft.json: Json.NET interprets and modifies ISO dates when deserializing to JObject

[FrancoisBeaune]

We're using Json.NET 8.0.2 in a C# 4.5 project, and we love it.

However this behavior bit us pretty hard:

``` c#
var obj = JsonConvert.DeserializeObject("{ 'x': '2016-03-31T07:02:00+07:00' }");

Here, in our time zone, `ob["x"].Value<string>()` returns `"2016-03-31T02:02:00+02:00"`.

The problem disappears if we specify how to parse dates:

``` c#
var obj = JsonConvert.DeserializeObject<JObject>("{ 'x': '2016-03-31T07:02:00+07:00' }",
    new JsonSerializerSettings() { DateParseHandling = DateParseHandling.None });

But I find it _dangerously surprising_ that by default Json.NET interprets strings as dates (let alone the performance issues this inevitably brings with it).

Is this the expected behavior? If so, is there a way to disable this behavior globally?

[FrancoisBeaune]

Seems like JsonConvert.DefaultSettings would allow to globally override this behavior:
http://james.newtonking.com/archive/2013/05/08/json-net-5-0-release-5-defaultsettings-and-extension-data

I would still be interested in hearing your opinion on this!

[antonovicha]

I suppose this is for convenience. We in our project do use this behavior because that way dates are human readable. On other hand MS Dates ( /Date(1224043200000)/ ) are not.

[FrancoisBeaune]

Deserializing a _string_ to a _string_ should not arbitrarily change the string. That really sounds like a bug to me, possibly a design bug. I'm not expecting breaking changes to be introduced to fix this, of course; I'm just interested in hearing the auhor's opinion on this. We found a workaround and that's good enough for us right now.

[antonovicha]

Way to represent a Date in Json is quite debatable because there is none in Json standard.
Here is SO question regarding this topic: http://stackoverflow.com/questions/10286204/the-right-json-date-format

Could be that this is why author choose 'use ISO8601 string as Date' behavior.

[FrancoisBeaune]

@antonovicha That's completely beyond the point. I'm just deserializing a string into a string, I do _not_ expect any changes to the string.

[FrancoisBeaune]

I'm hitting this problem again, with a different scenario.

Given a string s with the value ['2016-05-10T13:51:20Z', '2016-05-10T13:51:20+00:00'], I need to check that it is indeed a valid JSON array, and I then need to retrieve the _unmodified_ value of the individual items. Unfortunately, setting default settings does not appear to have any effect.

Here is the code:

``` c#
JsonConvert.DefaultSettings = () => new JsonSerializerSettings
{
DateParseHandling = DateParseHandling.None
};

var s = "['2016-05-10T13:51:20Z', '2016-05-10T13:51:20+00:00']";

var array = JArray.Parse(s);
foreach (var item in array)
{
var itemValue = item.Value();
Console.WriteLine(itemValue);
}

This prints

05/10/2016 13:51:20
05/10/2016 13:51:20

instead of

2016-05-10T13:51:20Z
2016-05-10T13:51:20+00:00
```

[tangdf]

``` c#
var s = "['2016-05-10T13:51:20Z', '2016-05-10T13:51:20+00:00']";
using (JsonReader jsonReader = new JsonTextReader(new StringReader(s))) {

    jsonReader.DateParseHandling = DateParseHandling.None;

   var array = JArray.Load(jsonReader);

    foreach (var item in array) {
        var itemValue = item.Value<string>();
           Console.WriteLine(itemValue);
      }

}

This prints

2016-05-10T13:51:20Z
2016-05-10T13:51:20+00:00
```

[jetuser]

They've commented on this before and I still disagree with their assessment of "by design" for this scenario:

865

[JamesNK]

Is this the expected behavior?

Yes it is. Use DateParseHandling if you want to change it.

[FrancoisBeaune]

@JamesNK The point is that DateParseHandling has no effect on JToken.Parse(), only on JsonConvert.DeserializeObject().

[JamesNK]

Set it on JsonTextReader that you pass to JToken.Load

[FrancoisBeaune]

Alright, thanks.

[thomaslevesque]

Is this the expected behavior?

Yes it is. Use DateParseHandling if you want to change it.

@JamesNK could you explain the reasons behind this design? It seems pretty strange to me that a string value should arbitrarily be interpreted as a date just because it looks like a date...

[JamesNK]

Because JSON doesn't have a date format.

[thomaslevesque]

Because JSON doesn't have a date format.

Maybe I'm missing something, but I don't understand how this answers my question... or maybe my question was unclear. I'll try to clarify.

It doesn't matter which date format is used; the problem is that JSON.NET tries to parse the string as a date when we're just asking for the raw string. If I have something like this:

[
    {
        "id": 123,
        "name": "foo",
        "comment": "blah blah"
    },
    {
        "id": 456,
        "name": "bar",
        "comment": "2016-05-10T13:51:20Z"
    },
]

The comment field is just a string, and isn't supposed to be interpreted as a date, even though it happens to contain something that looks like a date in the second element. When I do this:

var array = JArray.Parse(json); // with the JSON sample above
string value = array[1]["comment"].Value<string>();

I expect value to be exactly "2016-05-10T13:51:20Z", not the result of parsing it as a date and reformatting it in whatever happens to be the current culture. Right now, this code gives me "05/10/2016 13:51:20" (on a machine with the French culture). The string shouldn't be parsed as a date unless we explicitly request a date (e.g. with .Value<DateTime>()).

So what I'm asking is, why is the default value for DateParseHandling DateTime, rather than None?

[thomaslevesque]

Whether a date is a string or not is determined when loading the JObject, not when converting it to a strongly typed object.

(I just saw this message in the other discussion; GitHub seems to be having trouble delivering notifications)

This might be the root cause of the problem. Since there is no standard date format in JSON, JSON.NET shouldn't even _try_ to determine the type when loading the JObject... it should just assume it's a string, and leave the decision to treat it as a date to higher layers that actually know what it's supposed to be.

Anyway, I realize that's not something that can be changed easily, as it would likely affect many users. But is there a way to get the DateParseHandling.None behavior when using JToken.Parse? i.e. without reverting to JToken.ReadFrom and setting DateParseHandling on the reader?

[JamesNK]

i.e. without reverting to JToken.ReadFrom and setting DateParseHandling on the reader?

No. If you want to customize that setting then use ReadFrom.

So what I'm asking is, why is the default value for DateParseHandling DateTime, rather than None?

Because I judged most people want a date rather than a string. And for people who don't then there is a setting to change it.

[FrancoisBeaune]

While I really like Json.NET and thank you for the great work you're doing on that library, I deeply regret this design decision.

As I and other have been pointing out again and again, here and elsewhere, it really _is_ surprising that what is supposed to be an opaque string gets mutated based on the culture.

Having to sprinkle

JsonConvert.DefaultSettings = () => new JsonSerializerSettings
{
    DateParseHandling = DateParseHandling.None
};

at the beginning of every project using Json.NET is annoying and extremely error-prone. Moreover there are cases that even this setting doesn't handle, requiring deep convolutions _just to get a string out of a string_...

[FrancoisBeaune]

Moreover, this _got_ to have performance implications, because for every string property of a JSON object, you need to find out if it's parsable as a date, even when one just want to get strings back!

[JamesNK]

Sometimes you can't please everyone and in this case you're the person that isn't pleased.

Performance isn't an issue. A couple of simple checks on the string length, and certain chars exist at certain indexes eliminates 99% of non-ISO8601 strings instantly.

You can keep discussing this among yourselves but I like what it does, I have no plans to change it, and I would do it again if give the chance.

And if you don't like the default it is literally one line to change: DateParseHandling.None

[FrancoisBeaune]

Fair enough. Regardless of this particular issue, thanks for the hard work and the high quality library.

[mjamesTX]

But if JSON.NET chooses to parse ISO8601 strings as DateTimes (which is fair given JSON has no date time format), shouldn't it also ToString them into the same format? That could eliminate the issue, making the fact that it was technically parsed into a DateTime in between irrelevant. I have a hard time accepting that this test should not pass:

JObject obj = JObject.Parse("{'time':'2016-08-01T03:04:05Z'}");
Assert.AreEqual("2016-08-01T03:04:05Z", obj.Value<string>("time")); // This will fail, regardless of the value of JsonConvert.DefaultSettings

[sehrgut]

Because JSON doesn't have a date format.

That's the point. JSON _doesn't_ have a date format. You claim this is a JSON parser, but when you force-enable incompatible extensions to the standard, you no longer have a JSON parser. You have a NewtonsoftObjectNotation parser.

[ekevoo]

but when you force-enable incompatible extensions to the standard, you no longer have a JSON parser. You have a NewtonsoftObjectNotation parser.

While I do agree with your point, it made me very happy to find out that it permits comments even though the standard format explicitly disallows them. An obvious-to-use strict mode would go a long way for people who need strict JSON conformity.

[Cobaltikus]

I have been experiencing the same problem: a string being changed just because it happens to look like a date. This could easily be resolved by adding a new method in Json.NET to get the raw string value of a given property. As far as performance goes, it may not have a noticeable impact on deserializing a single object, but put it in a loop and now every action matters. The larger your dataset, the larger the performance impact. I hope to see a GetRawStringValue method in the future. In the meantime thank you for pointing out the workaround.

[sehrgut]

@ekevoo An obvious-to-use non-strict mode would be the way to go, since many times this library is called from within compiled assemblies. I can't modify every third-party library I use just because I've upgraded the version of Newtonsoft.Json attached to a project, to make them all opt-in to strict mode.

The users who want non-standard behaviour are the ones who have the responsibility to opt-in.

[ekevoo]

@sehrgut That little horse has left the barn. This is a decision to make at
library inception time, not to a years-old library that is literally THE
most popular library of the runtime (if NuGet is to be believed).

Besides, it's not like strict mode is ever default. Look at the defunct
Perl (use strict;) or lively Javascript ('use strict'; in es5).

[sehrgut]

The date parsing is a new feature. This operated correctly for most of the
time it was BECOMING such a widely-used library, and was introduced as a
breaking change very recently. Search the closed issues.

On Friday, November 4, 2016, Ekevoo [email protected] wrote:

@sehrgut That little horse has left the barn. This is a decision to make at
library inception time, not to a years-old library that is literally THE
most popular library of the runtime (if NuGet is to be believed).

Besides, it's not like strict mode is ever default. Look at the defunct
Perl (use strict;) or lively Javascript ('use strict'; in es5).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JamesNK/Newtonsoft.Json/issues/862#issuecomment-258570809,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABIFLIZPgTIibEQDWQwSRt0BP_CtGA1Oks5q672rgaJpZM4H7thy
.

[sehrgut]

As well, Perl and JavaScript strict pragmas are subsets of the standard.
This bad date parsing behaviour is not even a superset, it's a deviation
from correct parsing.

On Friday, November 4, 2016, Ekevoo [email protected] wrote:

@sehrgut That little horse has left the barn. This is a decision to make at
library inception time, not to a years-old library that is literally THE
most popular library of the runtime (if NuGet is to be believed).

Besides, it's not like strict mode is ever default. Look at the defunct
Perl (use strict;) or lively Javascript ('use strict'; in es5).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/JamesNK/Newtonsoft.Json/issues/862#issuecomment-258570809,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABIFLIZPgTIibEQDWQwSRt0BP_CtGA1Oks5q672rgaJpZM4H7thy
.

[JamesNK]

Json.NET has parsed ISO formatted strings as dates for over 4 years, which is when it switched to writing dates as ISO by default, and it it has parsed MS date style strings as dates since day one.

[PeterDowdy]

Oh man I am very much not pleased. I just lost a day to this. I even specified the data as a string to try to pin down where it was going wrong. This library will parse a date time string, alter it to the local timezone, and then _output it as a string_ if a valid DateTime string shows up in a string property. wat

[fmannhardt]

This is horrible. I lost a few hours on this, too. Too much magic.

[carlbm]

Have to say that I disagree with this design decision too. Many hours lost today.

[mwinslow]

I strongly feel that the current default behavior is wrong. The defult should be DateParseHandling = DateParseHandling.None. It is really unwelcome that the data being read is being "fixed" underneath the covers by default.

[antonovicha]

@mwinslow I agree with you that since json do not have DateTime format it is not a good idea to magically convert string to DateTime. However changing current behavior to not convert will introduce breaking change for existing code base that already relay on this fact (like for example bunch of our applications).

[sehrgut]

Then do a major version increment. Breaking code that depends on bad behaviour should not outweigh implementing correct behaviour.

[mwinslow]

have to agree with @sehrgut , but I do want to state overall I think this a great library. Thank you!

[jimmyheaddon]

This is downright painful, especially when you're writing libraries where you cannot second-guess what users will pass you, and equally, you cannot expect them to understand this behaviour! We're having to fudge public libraries to work around this - it just doesn't make sense to parse in one format and not permit ToString() to do the same by default

[StefanBertels]

Ok, this is an old issue and it is closed. But I again stumbled on it.

I already knew DateParseHandling option and its timezone and culture problems , but nevertheless I didn't expect that

"2017-01-02T03:04:05+01:00"

would be parsed to wrong date (2017-02-01 ...)

This is bad because ISO 8601 is a valid, unambiguous format in Germany and ISO 8601 should be correctly parsed worldwide / with arbitrary CurrentCulture.

DateTimeOffset.Parse and DateTime.Parse work correct out of the box for ISO format -- without changing or specifiying some culture.

[sammck]

It does seem that if you are going to interpret some strings in a special way and turn them into non-strings, then at the very least you should also provide a way to add escape codes to a string that would allow you to encode strings that would otherwise match your special pattern. As it stands it is not possible to serialize an object, then unserialize the result, and get back the same thing that you started with.

[loop-evgeny]

I just wasted half a day of debugging on this and will likely waste much more to figure out the impact of changing the DateParseHandling default throughout our code. Years later, this design bug is still wasting people's time.

Oh, and JObject.Parse doesn't provide any way to set DateParseHandling, so I had to disassemble it and copy its contents into my own method only to change DateParseHandling.

[wasabii]

I also hit this problem. Took my developers over a week to discover the issue. We are parsing JSON from external sources, modifying it algorithmically, and then doing other things with it.

One of the free text fields going through, randomly, had it's value changed. A very data dependent bug. It was of course fixed by turning off the magic date stuff, but holy hell it wasted a lot of time trying to pin down.

[louiskruger4c]

It seems like nobody wants this "feature", so it is surprising to me that this has still not been fixed. In my case it even causes loss of precision. My string value is "2018-04-23T09:16:41.991Z", but after JObject.Parse mangles the string it ends up being "23/04/2018 09:16:41".

Please, please fix!

[thomaslevesque]

Given the amount of feedback, I think this issue should be reopened. Not a single commenter thinks the current behavior is a good thing. There should at least be an easy way to turn it off globally.

[BowserKingKoopa]

I just smacked into this too. It's nasty and totally unexpected.

[TritonNet]

Vote for not to do the magical conversion if the target type is a string. Why someone would expect when your property is a string, the serializer elevate a json value that looks like a DateTime to a DateTime object and then converts it back to a string? it's strange.

PS: It makes sense to me the default behaviour of converting to a DateTime object when the target is object/JToken/JValue/JObject (something, not a string; Which allows holding a DateTime object)

[0xced]

I also got bit by this today. 🙁

You can keep discussing this among yourselves but I like what it does, I have no plans to change it, and I would do it again if give the chance.

@JamesNK If you still have no plans to change it, would you then consider @mjamesTX's proposal (ToString them into the same format) to at least mitigate this issue?

[mwinslow]

One of the excuses for not changing this behavior is because it will break existing code. Isn't this exactly what semantic versioning is for? Why not change the default behavior (to not manipulate data implicitly) for when a new major version is released (that would be 12 I think)? The existing code can be migrated to the new major version when it is fixed to handle the more "sane" new default behavior. I think this would make a lot of people happy.

[thomaslevesque]

One of the excuses for not changing this behavior is because it will break existing code.

Although it would technically be a breaking change, I doubt anyone actually relies on the current behavior. Most people are unaware of it, and those who are don't like it.

[antonovicha]

I doubt anyone actually relies on the current behavior.

Don't doubt. Numerous of our applications are relaying on that behavior.

[mwinslow]

@antonovicha I'm not doubting that, somehow further back in this discussion that was mentioned. What I am advocating is that semantic versioning is used to ensure the existing applications still work, while people that would love to have the new default behavior can adopt a new major version (i.e., 12.*). This would allow for safely migrating the applications.

[Cobaltikus]

There's no need to change the current behavior. Just add a new property or function to get the original raw text string for any given value. Best of both worlds. Everyone is happy. If you want the date object, you got it already. If you want the original raw text value, use the new property or function. It will become standard practice to use the new stuff and people will stop getting surprised.

[thomaslevesque]

Don't doubt. Numerous of our applications are relaying on that behavior.

I'd be curious to see the code. I can't think of any sensible way to take advantage of it.

[mwinslow]

@Cobaltikus Regarding your comment about "no need to change the current behavior"
If you look at the beginning examples regarding this issue you will note examples that show that the original string has already been manipulated once read through the library. With the current default behavior there is no way to get the original string value once you have loaded the data through the Newtonsoft.Json library. You don't have access to the raw string. Automatically parsing what is a string as a datetime just because it appears to be a date is a flawed design decision.

People are still commenting on this sneaky behavior two years after the issue was closed because the chosen default is not a sensible default. It really isn't. I guess since it works for the maintainers of this library this isn't going to change, but I think there will continue to be a steady stream of people that comment on this issue because when you get sideswiped by it you are tearing your hair out trying to understand why the data is being mangled unexpectedly. It is really brutal when it happens.

[Cobaltikus]

@mwinslow, Sorry for the confusion. I understood all of that already. What I am talking about is adding a brand new feature. There absolutely is a way. During the initial load, save the original raw text string. Leave everything else the same. When you want the string value, access it through this new property that does not exist today. If I was a maintainer of this library I could do it myself. Does that make more sense? The suggestion to add the new property or function was intended for the maintainers of this code, but I failed to make that clear.

[Cobaltikus]

Here's a better explanation. JObject.Value() returns the unexpected that we all dislike. JObject.RawString, or JProperty.RawString, or JToken.RawString, etc could be the new property that returns the expected value. But this RawString property doesn't exist. So I'm suggesting that a library maintainer should add it. Once this is in place users of this library could use it, and it would become standard practice to use it, and this problem would go away without having to change existing code for those who like the current behavior. A new property with the expected result would eliminate the need to change the existing behavior for the existing functions.

[loop-evgeny]

... and it should then use that RawString property when serializing back to JSON, so that the string survives the roundtrip.

[TheDuQe]

(Update : See Workaround below)
Well, spent years with this great library to face suddenly this issue. It could be a "by design" behavior if I wouldn't lose following information :

Before parse :
value=2018-03-21T16:18:17.3725571+01:00 <-- Customer time zone

JsonConvert.DefaultSettings =  () => new JsonSerializerSettings{DateParseHandling = DateParseHandling.None};
message.payload = JObject.Parse(json);

After parse :
value=2018-03-22T02:18:17.3725571+11:00 <-- My time zone

Result :
Values are equal but I have definitely lost one information : my customer's timezone which is important to my boss.

# WORKAROUND : Parse(json) is not the right method, use DeserializeObject(json) instead

Before parse :
value=2018-03-21T16:18:17.3725571+01:00 <-- Customer time zone

JObject obj = JsonConvert.DeserializeObject(json) as JObject;
message.payload = obj["Payload"] as JObject;

After parse :
value=2018-03-21T16:18:17.3725571+01:00 <-- Customer time zone

[jhellemann]

Thanks, @RoiArthur! You saved my day :)
In my case all DateTimes are in UTC and it's really important to stay in UTC and not local time. So I also had to add:

JObject obj = JsonConvert.DeserializeObject(line, new JsonSerializerSettings
{                        
      DateTimeZoneHandling = DateTimeZoneHandling.Utc
}) as JObject;

[jhbertra]

Yup, I just hit this "feature". Boo 👎

Thanks to the community for providing workarounds though! Too bad the default implementation is something that needs to be worked around in the first place... oh well.

[bdcoder2]

I have LOST days to this "feature" -- overall, I like the library, but really? !! To "just decide" to convert a string to a date on whim is not a "design feature", it has been nightmare -- for me at least !

[benbenwilde]

@JamesNK would you consider taking a poll and changing the default DateParseHandling setting based on the results in the next major version release? Pretty much everyone uses this library, so they should have a say??

[TheDuQe]

@JamesNK Parsing string as date using JObject.Parse(json) should result in a DateTimeOffset in order to keep the full message content. Parse the string as a DateTime lead to lose important data (Time zone).

[mavericken]

I only lost about an hour figuring out this was why my code was breaking. Maybe you guys aren't seeing the beauty of this feature. A minefield of little gotchas goes a long way for job security. We wouldn't want new developers coming in and showing us up, now would we?

[SailSouthwest]

Possibly the worst design decision I've come across -- ever -- in an otherwise great library.

[jhbertra]

Serialization should be pure - this is not pure. This is implicit environment reading. This feature means that, by default, a test case could pass or fail depending on the timezone of the host machine.

[mazjindeel]

This is not a feature its a bug

[hickford]

For comparison, ServiceStack.JSON.parse returns a string "2016-03-31T07:02:00+07:00"

(ServiceStack.JSON.parse("{ 'x': '2016-03-31T07:02:00+07:00' }") as Dictionary<string, object>)["x"]

[ckpearson]

Just as an additional hat in the ring, this has just bitten us and taken a day or two to find what was going on.

The behaviour of a string being interpreted without being asked is pretty counterintuitive, and any "spooky action at a distance" in programming really should be avoided...

That said, a few settings changes to tweak this and ensure outbound dates get serialised the way we need them to isn't too onerous.

[bdcoder2]

Again -- this COULD and SHOULD be resolved by introducing a simple global setting, ie:

NewtonSoft.Json.Do_not_screw_with_dates = True

[markzielinski]

I just came across this as well, lost a few hours of time due to it. I've never seen such arrogance.

[TylerBrinkley]

It seems the majority of cases here where this causes pain is when the object is deserialized and then re-serialized and the user is expecting the strings to be identical when the value is not modified. Is there any way the JValue that is created upon deserialization could store the original string value and if the date value is not changed then it would serialize the JValue with the original string as opposed to re-serializing the date value?

[thomaslevesque]

@TylerBrinkley actually roundtrip deserialization/reserialization works fine. It's only when you try to get the value as a string from the JObject that you get the issue.

https://gist.github.com/thomaslevesque/e832247047791fcfbda488ae3f2df60c

[jsolman]

I strongly believe this is a bug and not a feature. @thomaslevesque the roundtrip deserialization/reserialization does not work properly either when there are fracitonal seconds in the time. It loses the fractional seconds.

[bdcoder2]

THIS IS A BUG -- period. Add the necessary code to leave dates alone by adding a configuration parameter or property.

[thomaslevesque]

@thomaslevesque the roundtrip deserialization/reserialization does not work properly either when there are fracitonal seconds in the time. It loses the fractional seconds.

Are you sure? The gist I posted above works fine with fractional seconds as well

[jsolman]

are you sure? The gist I posted above works fine with fractional seconds as well

Your gist may work; however, if you set the JObject value for the key retrieved to the string it gives you, and then re-serialize it will have lost the fractional seconds.

jObject["TheDate"] = jObject["TheDate"].Value<string>();

I guess that isn’t what you are calling round trip though as I guess you mean round trip without modifying the JObject key with the value retrieved from the key.

In my opinion it is still a deserialize and reserialize problem — since after copying the JObject value into a new JObject’s value it will not have the same value as the original after reserialization.

[jsolman]

Why didn’t they at least make the default string serialization format when converting to a string be something like DateFormatHandling.IsoDateFormat?

[B4rT45]

This is a great library but unfortunately you can't call such behaviour a feature - it's clearly a bug, however you look at it.

You pass an int and get an int, you pass a bool and get a bool, you pass a string and you might get a string, or a date that matches the passed string or not, or has a different time or represents a completely different calendar day altogether ...
Surely, this can't be right.

Please note, manipulating and/or changing the processed data and/or data types is clearly not a responsibility of a serialisation library, at least not by default.

It should be up for the consumer to opt-in for the under-the-hood-magic conversion, i.e. the DateParseHandling.None should be the default and user should explicitly specify a different one if they want to have any auto conversions.

I think one could get away with calling such behaviour a feature, instead of a bug, if it was on the opt-in basis, or at least if it was handling dates properly which is very, very complicated and is a VERY good a reason in itself that it should not be done by default, if at all (even Microsoft's .NET DateTime implementation is pretty much useless, see nodatime).

Found today only by chance by having some unit tests that started failing and because there were quite complicated (I know they should not be, but I inherited them) I wasted a few hours to find this little nugget as this was the last place I would look for a solution for the problem.
It was also very strange that my log was telling me I'm having and passing a string to serialise it but on the receiving I would accept certain types only (including string that I passed), and DateTime was not one of them, but since logs said I passed, lets say a Dictionary of why the other end can't deserialise it as Dictionary?
Now I know, because it once of the elements comes back as

More importantly now I'm also worried that because of loosing precision/changing the dates (we run stuff globally) we can have some issues somewhere in the system (which must be VERY precise with times and dates) and those issues are just lingering somewhere waiting to manifests themselves at some point in future when it's too late, we are just not aware of them yet.
I/we will have to go through the whole codebase to make sure other devs didn't make an assumption that they will get whatever they put it (_because why would they make such a strange assumption?_)

If it was "by design", then like others here, I think that this was a poor decision as it's not intuitive and that's what it should be.

Although there are some nice suggestions above how this could be solved (temporally) in the current version (in preparation for the breaking change) I hope it will be changed with in the next major (breaking) version.

Thanks for your great work.

[dancrn]

i too have been bitten by this. honestly, if you want to parse strings as dates (which imo isn't a good move) then at least do it correctly;

JObject.Parse("{\"date\":\"0001-01-01T00:00:00+00:00\"}").ToString();
//returns "{\n  \"date\": \"0001-01-01T00:00:00-00:01\"\n}"

where's the extra minute coming from? :upside_down_face:

[smasherprog]

I just encountered this bug as well. Took me for ever to figure out that there was actually a bug in this library.
This library should NOT DESTROY DATA!
This default setting to destroy data is really confusing.. Why is destroying data a good design decision?

[dlumeida]

This behavior is very frustrating. After some time trying to figure it out I could solve my problem using

services.AddMvc().AddJsonOptions(opt =>
{
    opt.SerializerSettings.DateParseHandling = Newtonsoft.Json.DateParseHandling.None;
});

But I think it should be set as a default behavior since it causes lots of troubles when working with different timezones, nodatime, etc. Another pain is when working with <input type="datetime-local" step="1"> in Google Chrome.

Ex: If the field is set to "2018-11-05 10:20:59" Chrome serialize it as "2018-11-05T10:20:59" and Json.Net parse it as DateTime. If the field is set to "2018-11-05 10:20:00" Chrome serialize it as "2018-11-05T10:20" and Json.Net parse it as String.

[mikeeheler]

I wonder how much time collectively has been wasted on people running into exactly this issue, as I've just wasted some of mine this morning.

// User entered in a timestamp from somewhere for their own reasons
string input = "{\"user_comment\": \"2018-07-26T19:45:36-07:00\"}";
JObject parsed = JObject.Parse(input);

Console.WriteLIne(parsed["user_comment"].Value<string>());
// "07/26/2018 19:45:36"

Console.WriteLine(parsed["user_comment"].Value<DateTime>().Kind);
// "Local" ... if you say so

No timezone info in the result, no timezone transformation (the machine this ran on was at UTC-8), no indication that the field was meant to be a date. Arbitrary user input that means that if this were in an array of similar objects, the same field in different objects would be represented differently. This user just happened to put a ISO timestamp in the comment.

It just ain't right. The sad part is there is probably a ton of code out there right now which depends on this broken behavior.

To work around this and use JObject.Parse I need to set a global somewhere? How is that meant to be compatible with unit testing or concurrency? There's no variant of JObject.Parse that takes a settings object that can override the broken default.

[smasherprog]

Your preaching to the choir here mike. It was a bug that people continually run into which will not be fixed unfortunately.

[RoadTrain]

Funny thing is that this bug helped me find a bug in my own code which was incorrectly handling negative timezones (i.e. Json.NET modified the date representation in a json file, which triggered the bug in my code).

Still, I find this design choice very surprising, and the overall handling of the issue looks unprofessional.

[thomas-darling]

@JamesNK
I've just wasted a few hours of my life because of this incredibly stupid default behavior.
I don't care about the reasoning that went into this - it's wrong, and should be fixed.

@TylerBrinkley

It seems the majority of cases here where this causes pain is when the object is deserialized and then re-serialized and the user is expecting the strings to be identical when the value is not modified.

Exactly! The fact that this is not the default behavior is absolutely insane.

[dancrn]

look at all the times this bug has been mentioned in other project's issues! some significant mentions are Microsoft (twice! Azure, and SignalR!) and Stripe.

but hey, it's a feature, not a bug 🙃

[TheMightyPope]

I think enough is said here about this bug, and it seems there are no solution in sight, so I'll try the DateParseHandling = DateParseHandling.None global approach. Great to just notice this after building a big project and now having to redo a bunch of things, re-test, etc.. As a side note, what I found was that even the conversion was wrong, the library threw away half an hour for some reason (...T00:30:00) became 12:00:00 AM in the local representation. We only noticed because of the lost 30 mins.

[rotanov]

Bit me hard too.

[springy76]

Sometimes you can't please everyone and in this case you're the person that isn't pleased.

Boy, there is more than one here who isn't pleased.

IT IS A BUG!

I can see in the debugger that the JToken still is aware of the original value but when requested to deserialize to a [DataMember] attributed string property using ToObject<T> the string is rebuilt from some interpreted data. And if that alone is not buggy enough it also uses an undefined string format for this which is NOT the configured DefaultDateFormatString yyyy'-'MM'-'dd'T'HH':'mm':'ss.FFFFFFFK but something ugly different using "/" and totally unsortable ymd ordering.

[sehrgut]

Sometimes you can't please everyone and in this case you're the person that isn't pleased.

The thing is, you _can_ please everyone! By making your parser perform according to spec. That's _literally_ the definition of pleasing everyone in this case.

If you want to make a wrapper class/subclass that auto-coerces datatypes, then do that _too_, if you must.

[mehrbat]

I also believe it is a bug and there must be a way to stop this conversion. How could it possibly convert without knowing the exact datetime format? Day and Month could be confused.

I have date/time and datetime offset in my input json as text, so no top level settings can sort it out. I would prefer an attribute on the properties of the target [c#] class to convert the values the way I want per property, like these ones: https://www.newtonsoft.com/json/help/html/SerializationAttributes.htm

[mcintyre321]

It's the default that keeps on giving. I just spent an hour trying to figure out why my serialization code was mangling my datetimes, when in fact it was that the Visual Studio debugger Json Viewer is parsing string properties as DateTimes and ToStringing() them. Wonder what library it uses under the hood...

[codermrrob]

Just burnt me too. Great library with just one horrible default behavior setting. I really feel that this should be something you have to turn on rather than turn off. In my scenario we deal with a lot of different date formats and typically need to re-format them into something else. Having this awesome library change the format of the date to something unexpected is the only not so awesome feature in it.
Its made worse even because I explicitly request the property value as string and still it gets read as a date in the incorrect format.

[bdcoder2]

Did I miss @JamesNK's announcement on April 1st?

"As of version 12.1, date/time values will NO LONGER be internally interpreted and instead the original value will be retained".

[Cobaltikus]

Where is that announcement? Was it possibly an April Fool's Day joke? If so, by JamesMK, or by bdcoder2?

[bartelink]

I think we need to get the real JamesMK on this issue!

[bdcoder2]

It was meant as a joke -- the same way this bug is being treated as !

[milosloub]

@JamesNK It's a trap when develper use external library based on:
JToken.Parse(someString)
and there is no way to get rid of this NON-STANDARD behaviour. Finally this leads to full database of corrupted data and you can't do anything about this.

This is not acceptable!!!

[PathToLife]

This burnt me as well, was trying to cast as (string)value

Who the hell checks serialization of different strings to make sure they aren't objects.
Serialization of string should give string.

TLDR; to whoopdydo a server using Newtonsoft Json. Just save a DateTime Json string as a troll user.

[SteveSmith16384]

Add me to the list of user's who've just wasted a few hours wondering why my code was failing to parse what I thought was a string, and finally getting to the bottom of it. Now just waiting for the code to break again when a future person decides to send a different date format and Json.Net stops thinking that it's a date.

I'm sure it's not the first time any of us have had their time wasted by code that tries to be too clever and ends up causing more problems.

[paulpce]

Add me as another one, thankfully I'm lucky it only took me half an hour questioning the bahaviour of my app before ending up at this thread. Disappointed.

[markzielinski]

This library is obsolete now anyway, .NET Core 3.0 has built in support for JSON with System.Text.Json which is 1.5 - 2x faster and has more efficient memory use as well.

See this link for details:
https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-apis/

[thomaslevesque]

This library is obsolete now anyway, .NET Core 3.0 has built in support for JSON with System.Text.Json which is 1.5 - 2x faster and has more efficient memory use as well.

I wouldn't say it's obsolete... Sure, System.Text.Json is faster and more efficient, but it's not nearly as feature-rich as JSON.NET. In many situations System.Text.Json will be enough, but in more advanced scenarios, you might need the fine control offered by JSON.NET

[PathToLife]

Well it seems like the author, who graciously made public this code, .
(Therefore I'm saying this respectfully and thankfully.)

Has made their decision, and seems to not want to change it. (because a twitter poll he did about about changing this resulted in 50/50). I Say Fair enough~!
But if he refuses a pull req.. changing it. You could fork this and note him being the original creator.

I feel like a guilty freeloader who's despised by the creator for bumping this issue.
But it has definitely cost many new developers time over the years.

With System.Text.Json coming along, hopefully no more new developers will be burnt by this feature.

TLDR; Don't sound too disrespectful, this is a generous opensource project.
If you have the skills to fix it. Fork, make pull req & promote here.
Use System.text.Json when your project can support it in a few months/years

[jhandby]

I'm a long time Json.NET user, and I am very grateful for its existence, and I love the work you do. Except this one design decision, which is baffling. I've spent hours trying to figure out why my string containing a date was being mysteriously modified on serialization / deserialization, only to discover that Json.NET was _unilaterally deciding that it should inspect the contents of my strings and deciding that it should be changing them to other strings_.

This is absolutely sucky behaviour.

[jhandby]

Well it seems like the author, who graciously made public this code, .
(Therefore I'm saying this respectfully and thankfully.)

and I felt very much like that, but then I read his replies above, and I've got to say I'm quite disappointed by the tone in which he chose to respond to people reporting a bug caused by a breaking change he decided to make.

As much as I value his work thus far, it feels like this is one of those evolutionary moments when you realise that half the modern internet is completely dependent on one guy's library, and whatever design decisions that guy makes, and what mood he wakes up in from day to day. However much one appreciates that guy's contribution, that's not really sustainable.

[nicky1038]

First of all, many thanks to the author for such a great library.

As for the issue, what has not been said yet: a design decision is good if it is intuitive how to use a feature. If so many people are sure something works one way but in real it works another way, something is definitely wrong.

I can also suggest another measure of this decision's correctness, other than "a lot of people use date strings parsing" (and how many do not?!). "Total time people spent to sort out how to set up serialization/deserialization and what goes wrong is huge and could be greatly decreased by changing the default setting".

[aurelien-boubennec]

It took me 2 hours to figure out newtonsoft was altering string data without consent...

[aurelien-boubennec]

"Feature request: ditch Newtonsoft.Json in favor of System.Text.Json IdentityModel/IdentityModel#286"

LOL

[dsaul]

Yep, this one has messed me up on a number of occasions.

[glyngholm]

Just got bit by this.....

[dotnetprofessional]

The below code saved my project. This issue was leading to very unexpected behavior (aka bugs!) and its not at all intuitive that this issue exists, at a minimum it could honor UTC format and make the datas Utc not Local. Anyway this code fixed my issue, the global setting isn't good for two reasons, 1) you dont want to apply to all aspects and 2) it doesn't work with an existing JObject ie .ToObject.

from: @FrancoisBeaune

The problem disappears if we specify how to parse dates:

var obj = JsonConvert.DeserializeObject<JObject>("{ 'x': '2016-03-31T07:02:00+07:00' }",
    new JsonSerializerSettings() { DateParseHandling = DateParseHandling.None });

I used this on just the section that contained dates as string which was a Dictionary.

Thanks!

[danrien]

Another fix workaround that worked well for using JSON.Net in what should be a generic caching library was to convert DateTime types to their binary representation using the ToBinary() method on DateTime within a customer converter. This encodes all of the DateTime information in a long, far away from the prying eyes of JSON.Net's DateTime handling logic. Since the serialized value was hidden from consumers of the library, this was sufficient for my needs, but this behavior was definitely unexpected.

[dvick]

Making this default to deserializing as a DateTime by default is absolutely a terrible design choice. Broke a JsonConverter for me when used with JsonPatchDocument, which doesn't take JsonSerializerSettings (which it definitely should). Had to set the DateParseHandling property to None on the JsonReader in the JsonConverter for the class containing the property with the broken JsonConverter.

[gdalsnes]

The worst thing here is not that the default is DateParseHandling.DateTime but that when using converters and the string is already converted to DateTime, there is no way to overrule the parsing because no way to get original string. Setting DateParseHandling.None globally may not always be easy\possible\wanted, and it may be difficult to access individual settings as they will be burred inside a library etc.
I was actually thinking about forking the library because of this, the problem is still 3rd party libraries...no easy way to get them to use the forked version. I guess we just need to wait until all has migrated to System.Text.Json, but that will take many years.

[TritonNet]

@gdalsnes Just noted your idea to get the right behaviour with forking. I've done the same and you could get the third party libraries to use the forked version with AssemblyRedirection in the config file.

[B4rT45]

we started using System.Text.Json for all new .NET core projects despite having have quite mature "framework" (libraries/packages) that are are written for/with use of newtonsoft - so far we were able to achieve everything that was done in the old projects/libraries/packages with no issues and given that there will be a lot of new features/improvements in STJ starting with .net 5 I think _we_ can safely say .... bye-bye newton and your "by-design feature that changes users' data" (where serialise(x) = y and deserialise(y) = z) :D

[sungam3r]

with .net 5 I think we can safely say .... bye-bye newton

100% agree

[codermrrob]

You can switch if you use only 10% of the library...

Get Outlook for Androidhttps://aka.ms/ghei36

---

From: Ivan Maximov [e-mail]
Sent: Sunday, November 1, 2020 4:49:26 PM
To: JamesNK/Newtonsoft.Json [e-mail]
Cc: codermrrob [e-mail]; Comment [e-mail]
Subject: Re: [JamesNK/Newtonsoft.Json] Json.NET interprets and modifies ISO dates when deserializing to JObject (#862)

with .net 5 I think we can safely say .... bye-bye newton

100% agree

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/JamesNK/Newtonsoft.Json/issues/862#issuecomment-720099983, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AE42QAN54A3TILPAMKKEBMLSNVYPNANCNFSM4B7O3BZA.

[B4rT45]

You can switch if you use only 10% of the library...

I don't where the 10% is coming from (I guess the % would refer to the current state of STJ, not the future versions?) but I know the switch worked for us like a treat :)