Runtime: System.Text.Json deserializing of object[bool] does not produce boolean

[radek7210]

I understand that the System.Text.Json is still in development. However, I would like to point out that the new deserializer produces very different results than the previous one.

Scenario:

public class MyRequest
{
  public string ParamName { get; set; }
  public object ParamValue { get; set; }
}

Controller method:

[HttpPost]
public ActionResult<string> Post([FromBody] MyRequest myRequest)
{
  return Content($"Type of '{myRequest.ParamName}' is {myRequest.ParamValue.GetType()}; value is {myRequest.ParamValue}");
}

Posting this json:

{
  "ParamName": "Bool param",
  "ParamValue": false
}

In .net core 2.1, the false value deserializes as boolean:
Type of 'Bool param' is System.Boolean; value is False

However, in .net core 3.0 preview 6, the false value deserializes as System.Text.Json.JsonElement:
Type of 'Bool param' is System.Text.Json.JsonElement; value is False

Will there be any chance to make the new deserializer work the same as in 2.1?

Note: we declare the ParamValue as object, as in the real app the values are of several different types, and so far the deserializer handled all them for us without issues. In 3.0, all this functionality is broken.

Thanks.

[steveharter]

Current functionality treats any object parameter as JsonElement when deserializing. The reason is that we don't know what CLR type to create, and decided as part of the design that the deserializer shouldn't "guess".

For example, a JSON string could be a DateTime but the deserializer doesn't attempt to inspect. For a "True" or "False" in JSON that is fairly unambiguous to deserialize to a Boolean, but we don't since we don't want to special case String or Number, we don't want to make an exception for True or False.

With the upcoming preview 7, it is possible to write a custom converter for object that changes that behavior. Here's a sample converter:

public class ObjectBoolConverter : JsonConverter<object>
{
    public override object Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.True)
        {
            return true;
        }

        if (reader.TokenType == JsonTokenType.False)
        {
            return false;
        }

        // Forward to the JsonElement converter
        var converter = options.GetConverter(typeof(JsonElement)) as JsonConverter<JsonElement>;
        if (converter != null)
        {
            return converter.Read(ref reader, type, options);
        }

        throw new JsonException();

        // or for best performance, copy-paste the code from that converter:
        //using (JsonDocument document = JsonDocument.ParseValue(ref reader))
        //{
        //    return document.RootElement.Clone();
        //}
    }

    public override void Write(Utf8JsonWriter writer, object value, JsonSerializerOptions options)
    {
        throw new InvalidOperationException("Directly writing object not supported");
    }
}

Used like

var options = new JsonSerializerOptions();
options.Converters.Add(new ObjectConverter());

object boolObj = JsonSerializer.Parse<object>("true", options);
bool b = (bool)boolObj;
Debug.Assert(b == true);

object elemObj = JsonSerializer.Parse<object>(@"{}", options);
Debug.Assert(elemObj is JsonElement);

[radek7210]

sounds good, I'll use the custom converter. Thanks.

[John0King]

@steveharter Json and Javascript don't have many type, but boolean is known one.
System.Text.Json should map json known'd type to clr type directly without custom converter

json | clr
|---|----|
boolean| bool
number| double
string | string
null | null
undefined| null

[steveharter]

@John0King I provided the rationale previously. Automatically mapping number to double won't work for large decimals.

What are the scenarios for using system.object instead of the strong type (double, string, bool)?

[Mike-E-angelo]

FWIW, using an object/dynamic property for message deserialization is a security consideration and should be handled with care:

https://www.alphabot.com/security/blog/2017/net/How-to-configure-Json.NET-to-create-a-vulnerable-web-API.html

[John0King]

@steveharter
you are right, In many scenarios , handle JsonElement is much easier than an unknow/dynamic object type . it's just different than Json.Net that will break many people

[GSPP]

Maybe it's better to throw instead of deserializing as JsonElement. That way the decision can still be made in the future on how to deserialize to type object. Now, we're locked in to JsonElement.

I have personally wanted to deserialize to object a few times in the past with JSON.NET. I always wanted the "obvious" deserialization to bool etc.

If I want to deserialize to JsonElement then I can make the property a JsonElement.

[nick2893]

@steveharter
So, regarding dictionary, what is this supposed to look like? The following throws an exception.

        if (reader.TokenType == System.Text.Json.JsonTokenType.StartObject) 
        { 
            // https://github.com/dotnet/corefx/issues/39953 
            // https://github.com/dotnet/corefx/issues/38713 
            var conv = options.GetConverter(typeof(Dictionary<string, object>)) as System.Text.Json.Serialization.JsonConverter<Dictionary<string, object>>; 
            if (conv != null) 
            { 
                return conv.Read(ref reader, type, options); 
            } 
            throw new System.Text.Json.JsonException(); 
        }

Regarding the design choices;
What can be used to represent an Object other than dictionary?
What can be used to represent an Array other than array?

I can appreciate abstraction, but somebody has the make the final implementation decision. The current design completely ignores people who happened to stumble upon a Json file as a job requirement, and most likely need to use Json and System.Text.Json once and never again. For those folks, something that reasonably easily generates dictionarys and arrays and strings (Json files are one big string, right?) with little effort (such as System.Web.Script.Serialization.JavaScriptSerializer), and some (any?) reasonable default deserialization decisions/assumptions would make life so much easier.

[gmurray81]

I don't really like this design. You are making the most common use cases for a parser an impenetrable mess to serve the general correctness. To serve the 2% edge cases, you are making this yet another unusable .NET Json API for the 98% use cases. You have an options bucket, use it to OPT into the generally correct, but unhelpful behavior, not have it be the default.

[steveharter]

@gmurray81 can you explain your scenario for using System.Object (instead of just a bool) and what other values it may hold other than a bool?

Since JsonElement would still be used for non-bool types (e.g. objects, arrays, numbers) how would having a special case for bool make the code easier to consume?

[scalablecory]

In most cases when I've seen this, it has been an anti-pattern. object is rarely what people want, and have seen far too many model.Foo.ToString() from people who didn't realize they shouldn't be using object.

[scalablecory]

The JSON->POCO auto-generator in VS creates object members when it sees null, so this behavior breaks anyone migrating from such a thing.

But in most cases, this is again people just letting the default give them a bad model when they should be fixing what it gives them.

[John0King]

suggest to add a new strategy ：PreferRawObject:true to use default object mapping

json | clr
|---|----|
boolean| bool
number| double
string | string
null | null
undefined| null
Date(string with TimezoneInfo) | DateTimeOffset
Date(string without TimezoneInfo )| DateTime

[steveharter]

suggest to add a new strategy ：PreferRawObject:true to use default object mapping

Yes some global option is doable.

However, it is actually fairly easy to create a custom converter to handle the cases you show above. Earlier I provided the sample for bool and here's a sample for bool, double, string, DateTimeOffset and DateTime. I will get the sample added to https://github.com/dotnet/runtime/blob/3e4a06c0e90e65c0ad514d8e2a9f93cb584d775a/src/libraries/System.Text.Json/tests/Serialization/CustomConverterTests.Object.cs#L267

private class SystemObjectNewtonsoftCompatibleConverter : JsonConverter<object>
{
    public override object Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.True)
        {
            return true;
        }

        if (reader.TokenType == JsonTokenType.False)
        {
            return false;
        }

        if (reader.TokenType == JsonTokenType.Number)
        {
            if (reader.TryGetInt64(out long l))
            {
                return l;
            }

            return reader.GetDouble();
        }

        if (reader.TokenType == JsonTokenType.String)
        {
            if (reader.TryGetDateTime(out DateTime datetime))
            {
                return datetime;
            }

            return reader.GetString();
        }

        // Use JsonElement as fallback.
        // Newtonsoft uses JArray or JObject.
        using (JsonDocument document = JsonDocument.ParseValue(ref reader))
        {
            return document.RootElement.Clone();
        }
    }

    public override void Write(Utf8JsonWriter writer, object value, JsonSerializerOptions options)
    {
        throw new InvalidOperationException("Should not get here.");
    }
}

[steveharter]

See the SystemObjectNewtonsoftCompatibleConverter sample class in https://github.com/dotnet/runtime/blob/7eea339df0dab9feb1a9b7bf6be66ddcb9924dc9/src/libraries/System.Text.Json/tests/Serialization/CustomConverterTests.Object.cs#L267 for semantics similar to Json.NET (except for objects and arrays).

[ahsonkhan]

Is JsonElement now a new type for POCO?

It's a struct that holds the JSON itself and provides ways for you to walk the document object model (DOM), programmatically.
https://docs.microsoft.com/en-us/dotnet/api/system.text.json.jsonelement?view=netcore-3.1

https://docs.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-how-to?view=netcore-3.1#use-jsondocument-for-access-to-data

I am trying to deserialize a response with a value-list as object.
expected values types are dynamic and i made a List property in my model.

You can cast the object model to JsonElement and then get the values you need.

So:
C# object model = JsonSerializer.Deserialize<object>("[\"str\", \"2020-02-22T05:50:46.3605858+00:00\", 25.5]"); if (model is JsonElement element) { string str = element[0].GetString(); // returns "str" DateTimeOffset date = element[1].GetDateTimeOffset(); double num = element[2].GetDouble(); //returns 25.5 }

[Zetanova]

@ahsonkhan Thx for your answere, but it is not answering it.
Because there is no generic option to cast/read the value from JsonElement,
the consumer of the POCO need to know about the JsonElement Type
and so it would be a new POCO type.
This would be somehow against the naming of Plain-Old-CLR-Object.

[johnwc]

For those looking to convert JSON object to Hashtable, feel free to use my example. It should cover all JSON types. But, I have not tested it thoroughly, only for what we are needing it for. I have not used the write either, as we are only reading.

var options = new JsonSerializerOptions();
options.Converters.Add(new JsonHashtableConverter());
var obj = JsonSerializer.Deserialize<Hashtable>(object, options);


public class JsonHashtableConverter : JsonConverterFactory
{
    private static JsonConverter<Hashtable> _valueConverter = null;

    public override bool CanConvert(Type typeToConvert)
    {
        return typeToConvert == typeof(Hashtable);
    }

    public override JsonConverter CreateConverter(Type type, JsonSerializerOptions options)
    {
        return _valueConverter ?? (_valueConverter = new HashtableConverterInner(options));
    }

    private class HashtableConverterInner : JsonConverter<Hashtable>
    {
        private JsonSerializerOptions _options;
        private JsonConverter<Hashtable> _valueConverter = null;

        JsonConverter<Hashtable> converter
        {
            get
            {
                return _valueConverter ?? (_valueConverter = (JsonConverter<Hashtable>)_options.GetConverter(typeof(Hashtable)));
            }
        }

        public HashtableConverterInner(JsonSerializerOptions options)
        {
            _options = options;
        }

        public override Hashtable Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        {
            if (reader.TokenType != JsonTokenType.StartObject)
            {
                throw new JsonException();
            }

            Hashtable hashtable = new Hashtable();

            while (reader.Read())
            {
                if (reader.TokenType == JsonTokenType.EndObject)
                {
                    return hashtable;
                }

                // Get the key.
                if (reader.TokenType != JsonTokenType.PropertyName)
                {
                    throw new JsonException();
                }

                string propertyName = reader.GetString();
                reader.Read();

                hashtable[propertyName] = getValue(ref reader, options);
            }
            return hashtable;
        }

        private object getValue(ref Utf8JsonReader reader, JsonSerializerOptions options)
        {
            switch (reader.TokenType)
            {
                case JsonTokenType.String:
                    return reader.GetString();
                case JsonTokenType.False:
                    return false;
                case JsonTokenType.True:
                    return true;
                case JsonTokenType.Null:
                    return null;
                case JsonTokenType.Number:
                    if (reader.TryGetInt64(out long _long))
                        return _long;
                    else if (reader.TryGetDecimal(out decimal _dec))
                        return _dec;
                    throw new JsonException($"Unhandled Number value");
                case JsonTokenType.StartObject:
                    return JsonSerializer.Deserialize<Hashtable>(ref reader, options);
                case JsonTokenType.StartArray:
                    List<object> array = new List<object>();
                    while (reader.Read() &&
                        reader.TokenType != JsonTokenType.EndArray)
                    {
                        array.Add(getValue(ref reader, options));
                    }
                    return array.ToArray();
            }
            throw new JsonException($"Unhandled TokenType {reader.TokenType}");
        }

        public override void Write(Utf8JsonWriter writer, Hashtable hashtable, JsonSerializerOptions options)
        {
            writer.WriteStartObject();

            foreach (KeyValuePair<string, object> kvp in hashtable)
            {
                writer.WritePropertyName(kvp.Key);

                if (converter != null &&
                    kvp.Value is Hashtable)
                {
                    converter.Write(writer, (Hashtable)kvp.Value, options);
                }
                else
                {
                    JsonSerializer.Serialize(writer, kvp.Value, options);
                }
            }

            writer.WriteEndObject();
        }
    }
}

[thinkards]

Edit: Added in write ability for json objects being deserialized (returned) from the controller actions as well. I omitted the extension methods for isIntegerType etc... for brevity. I agree with @gmurray81 that I expected this to be a fairly common use case to serialize/deserialize basic primitives in an object property. It was a 2 hour detour for me today to solve this problem, and I'm not even sure if I solved it the right way (it works, but I'm open to feedback on better ways). Multiply that by all the developers out there who have and will continue to stumble through this problem, coming up with their own clunky workarounds.

For others wanting to do something similar...

I needed an object property that accepts bool, number, or string for binding through ASP.NET core controller actions:

    public class MyConverter : JsonConverter<object>
    {
        public override object Read(ref Utf8JsonReader reader, Type type, JsonSerializerOptions options)
        {
            long longValue = 0;
            Decimal decimalValue = 0.0M;

            if (reader.TokenType == JsonTokenType.Number)
            {
                if (reader.TryGetInt64(out longValue))
                {
                    return longValue;
                }
                else
                {
                    if (reader.TryGetDecimal(out decimalValue))
                    {
                        return decimalValue;
                    }
                }
            }
            else if (reader.TokenType == JsonTokenType.True)
            {
                return true;
            }
            else if (reader.TokenType == JsonTokenType.False)
            {
                return false;
            }
            else if (reader.TokenType == JsonTokenType.String)
            {
                return reader.GetString();
            }
            throw new JsonException($"Invalid type {reader.TokenType}");

        }

            public override void Write(Utf8JsonWriter writer, object value, JsonSerializerOptions options)
            {
                if (value.IsString())
                {
                    writer.WriteStringValue((string)value);
                }
                else if (value.IsFloating())
                {
                    writer.WriteNumberValue((decimal)value);
                }
                else if (value.IsInteger())
                {
                    writer.WriteNumberValue((long)value);
                }
                else if (value.IsBoolean())
                {
                    writer.WriteBooleanValue((bool)value);
                }
                else
                {
                    throw new InvalidOperationException("Directly writing object not supported");
                }
            }

            public override bool CanConvert(Type typeToConvert)
            {
                return typeToConvert.IsObjectType() || typeToConvert.IsIntegerType() || typeToConvert.IsFloatingType() || typeToConvert.IsBooleanType() || typeToConvert.IsStringType();
            }
    }

Which I applied to my model for ASP.NET core binding:

    public class MyModel
    {
        ...
        [JsonConverter(typeof(MyConverter))]
        public object Value { get; set; }
        ...
    }

And used in my Controller Action:

        public async Task Create([FromBody]MyModel model)
        {
            await DoWork(model);
        }