Newtonsoft.json: How to create a custom converter that then calls the default serialization behavior?

[bytenik]

I am writing a custom converter for a type, let's say type X.

In my converter's read overload, I need to determine the type of entity at which I am looking. If the current entity is a string, I need to perform some custom behavior to convert the string to an X. If the current entity is an object, I need to use the ordinary serializer behavior (i.e. what the serializer would do if my converter was not registered) in order to deserialize that object and pass it along.

Is there a way to do this? If I call serializer.Deserialize I will end up in infinite recursion calling my converter. If I use a bool flag in the converter to turn off my converter before calling deserialize, I have the issue that I cannot have an X nested with in an X. It also isn't thread safe if multiple threads are using the serializer at the same time.

[JamesNK]

Ask questions on stackoverflow

[bytenik]

You mean like this question on StackOverflow I posted two years ago on the same topic? :smile:

http://stackoverflow.com/questions/16085805/recursively-call-jsonserializer-in-a-jsonconverter

[jordisan]

Try with
JObject.FromObject(value).WriteTo(writer);

[loop-evgeny]

Try with
JObject.FromObject(value).WriteTo(writer);

This does the serialization, but it doesn't honour the serialization settings configured on the JsonSerializer. If I pass in the serializer (JObject.FromObject(value, serializer).WriteTo(writer)) then I get a StackOverflowException as it calls back into the converter. I fixed it by creating a new JsonSerializer inside the converter with all the same settings except that converter. Awkward!

[logiclrd]

There is actually a legitimate issue here, because it doesn't seem that currently there is a way to make a custom converter _based on_ the default conversion that is registered with a [JsonConverter] attribute on the type. Even a brand new JsonSerializer for the inner conversion doesn't work, because it sees the [JsonConverter] attribute and implicitly suffers from re-entrance.

What would be really useful in this regard is a "default" implementation of the JsonConverter abstract class, so that you can then derive from it and use base.WriteObject in your implementation.

Is Newtonsoft.Json officially dead, though, now that System.Text.Json exists? :-(

[bytenik]

For some reason @JamesNK has shown no interest in this problem over the years. Its an incredibly frustrating problem.

[nyircsak]

Yep, I met with the same issue. :(

[kevinarthurackerman]

I know this thread is forever old, but if anyone is still trying to figure this out here was my solution. In my case I needed to check the type of the message payload before serializing or deserializing to enforce a whitelist of allowed types, and if valid continue with all other settings identical.

The basic workflow is inspect the type, remove the custom converter, serialize/deserialize the data, then add the converter back. For writing the JSON checking the type was simple because you have the in memory object. For reading it was more complex because you need to inspect the JSON graph to locate the type.

Note that this implementation is not multi-threading safe.

_jsonSerializer = new JsonSerializer
{
    PreserveReferencesHandling = PreserveReferencesHandling.Objects,
    ReferenceLoopHandling = ReferenceLoopHandling.Serialize,
    TypeNameHandling = TypeNameHandling.Auto,
    DefaultValueHandling = DefaultValueHandling.IgnoreAndPopulate,
    NullValueHandling = NullValueHandling.Ignore
};

_jsonSerializer.Converters.Add(new MessageConverter(allowedMessageTypes));

private class Message
{
    public object? Data { get; set; }
    public Dictionary<string,string>? Meta { get; set; }
}

private class MessageConverter : JsonConverter
{
    private readonly HashSet<Type> _allowedMessageTypes;

    public MessageConverter(IEnumerable<Type> allowedMessageTypes)
    {
        _allowedMessageTypes = allowedMessageTypes.ToHashSet();
    }

    public override bool CanConvert(Type objectType) => objectType == typeof(Message);

    public override bool CanRead => true;

    public override bool CanWrite => true;

    public override object? ReadJson(JsonReader reader, Type objectType, object? existingValue, JsonSerializer serializer)
    {
        var jObject = serializer.Deserialize<JObject>(reader);

        jObject.TryGetValue(nameof(Message.Data), StringComparison.OrdinalIgnoreCase, out var dataToken);

        if (dataToken == null) throw new InvalidOperationException($"{nameof(Message)} payload missing {nameof(Message.Data)} property");

        var dataObject = dataToken as JObject;

        if (dataObject == null) throw new InvalidOperationException($"{nameof(Message)} {nameof(Message.Data)} property was not an object type");

        dataObject.TryGetValue("$type", out var typeToken);

        if (dataToken == null) throw new InvalidOperationException($"{nameof(Message)} {nameof(Message.Data)} property was missing $type property");

        var typeString = typeToken.Value<string>();

        var type = Type.GetType(typeString, throwOnError: false, ignoreCase: true);

        if (type == null || !_allowedMessageTypes.Contains(type))
            throw new InvalidOperationException($"{nameof(Message)}.{nameof(Message.Data)} is not an allowed type");

        var thisIndex = serializer.Converters.IndexOf(this);
        serializer.Converters.RemoveAt(thisIndex);

        var result = jObject.ToObject(objectType, serializer);

        serializer.Converters.Insert(thisIndex, this);

        return result;
    }

    public override void WriteJson(JsonWriter writer, object? value, JsonSerializer serializer)
    {
        if (value == null)
        {
            writer.WriteNull();
            return;
        }

        var message = value as Message;

        if (message!.Data != null && !_allowedMessageTypes.Contains(message.Data.GetType()))
            throw new InvalidOperationException($"{nameof(Message)}.{nameof(Message.Data)} is not an allowed type");

        var thisIndex = serializer.Converters.IndexOf(this);
        serializer.Converters.RemoveAt(thisIndex);

        serializer.Serialize(writer, message);

        serializer.Converters.Insert(thisIndex, this);
    }
}

[Plastiquewind]

@kevinarthurackerman you saved my day! Thanks a lot for the snippet.
Btw, probably it could have been improved with the generic JsonConverter<T> version.

[logiclrd]

Note that this approach will probably not do the right thing for a type that contains fields of the same type.

class TreeNode
{
  public TreeNode[] Children;
  ...
}

Only the first TreeNode encountered while digging into the object graph will get processed by the custom converter, because it will be inactive for the "actual" processing of everything from that point down.

This is obviously only a concern for this specific scenario, but is worth keeping in mind, as it illustrates that the work-around does not actually fully achieve the desired behaviour in a generalized sense.

[kevinarthurackerman]

@Plastiquewind glad I could help! I'm modifying my implementation to use JsonConverter.

@logiclrd that is a limitation to my approach, which in my use case was not a problem. A more robust solution might look like the code below (note: this has not been extensively tested). The key points are having a thread local flag to indicate if the converter should be skipped, and then turning the flag off after each call to CanRead or CanWrite so that it is only ever skipped once each time it is set. It's a bit of a hack on a hack, but I think it would achieve OP's original goal with support for thread safety and nesting.

private class MessageConverter : JsonConverter<Message>, IDisposable
{
    private readonly HashSet<Type> _allowedMessageTypes;

    private readonly ThreadLocal<bool> _skip = new ThreadLocal<bool>(() => false);

    public MessageConverter(IEnumerable<Type> allowedMessageTypes)
    {
        _allowedMessageTypes = allowedMessageTypes.ToHashSet();
    }

    public override bool CanRead => !_skip.Value || (_skip.Value = false);

    public override bool CanWrite => !_skip.Value || (_skip.Value = false);

    public override Message ReadJson(JsonReader reader, Type objectType, Message? existingValue, bool hasExistingValue, JsonSerializer serializer)
    {
        var jObject = serializer.Deserialize<JObject>(reader);

        jObject!.TryGetValue(nameof(Message.Data), StringComparison.OrdinalIgnoreCase, out var dataToken);

        if (dataToken == null) throw new InvalidOperationException($"{nameof(Message)} payload missing {nameof(Message.Data)} property");

        var dataObject = dataToken as JObject;

        if (dataObject == null) throw new InvalidOperationException($"{nameof(Message)} {nameof(Message.Data)} property was not an object type");

        dataObject.TryGetValue("$type", out var typeToken);

        if (typeToken == null) throw new InvalidOperationException($"{nameof(Message)} {nameof(Message.Data)} property was missing $type property");

        var typeString = typeToken.Value<string>();

        var type = Type.GetType(typeString, throwOnError: false, ignoreCase: true);

        if (type == null || !_allowedMessageTypes.Contains(type))
            throw new InvalidOperationException($"{nameof(Message)}.{nameof(Message.Data)} is not an allowed type");

        _skip.Value = true;

        var result = jObject.ToObject(objectType, serializer) as Message;

        return result;
    }

    public override void WriteJson(JsonWriter writer, Message? value, JsonSerializer serializer)
    {
        if (value == null)
        {
            writer.WriteNull();
            return;
        }

        if (value!.Data != null && !_allowedMessageTypes.Contains(value.Data.GetType()))
            throw new InvalidOperationException($"{nameof(Message)}.{nameof(Message.Data)} is not an allowed type");

        var thisIndex = serializer.Converters.IndexOf(this);
        serializer.Converters.RemoveAt(thisIndex);

        serializer.Serialize(writer, value);

        serializer.Converters.Insert(thisIndex, this);
    }

    public void Dispose()
    {
        _skip.Dispose();
    }
}

Edited to add disposing