Newtonsoft.json: The public setter of a property that is a List<T> is not called during deserialization.

[JohnWashburn]

I thought the setter of a property was called (it the setter is public) as part of the Populate() call within JSON.net. There must be more to it than this naive belief.

I was working on an implementation of a BealeCipher AKA a book cipher. This involves managing and combining 4 lists; one list for each of the cipher elements. My plan was to store the WIP (work in progress) as a JSON serialization of the main object; the cipher engine.

I eventually got the serialization and de-serialization to work, but only after writing a customized converter. I wrote the customization only because the default behavior of v6.08 would serialize the public List<T> properties defined within the Engine object as expected, but would not de-serialize those same properties. My investigation surprised me when it showed me that the public setter of the property was not called as part of de-serialization of the JSON into an object.

I have re-implement the code in a familiar programmer motif of Orders comprised of zero or more OrderLines.

The core part of Order object is it extends List<OrderLine> and then also makes the upcast from an Order object to a List<OrderLine> object via a property in Order named: OrderLines.

Here is n excerpt from the class definition for Order:

[JsonObject(MemberSerialization.OptOut)]
public class Order : List<OrderLine>, ICloneable
{
  public enum PaymentType
  {
    Cash, Check, Visa, MasterCard, AmEx, Diner, Other = 99
  }

  #region Members

  [JsonProperty]
  public List<OrderLine> OrderLines
  {
    get
    {
      // return a copy to the requestor not a reference to this list
      // return new List<OrderLine>(this);
      return this.ToList();
    }
    set
    {
      Clear();
      if (null != value)
      {
        AddRange(value);
      }
    }
  }
}

Which serialized into:

{
  "$id": "1",
  "$type": "JsonDotNetDefectDemo01.Order, JsonDotNetDefectDemo01",
  "OrderLines": {
    "$id": "2",
    "$type": "System.Collections.Generic.List`1[[JsonDotNetDefectDemo01.OrderLine, JsonDotNetDefectDemo01]], mscorlib",
    "$values": [
      {
        "$id": "3",
        "$type": "JsonDotNetDefectDemo01.OrderLine, JsonDotNetDefectDemo01",
        "PartOrdered": {
          "$id": "4",
          "$type": "JsonDotNetDefectDemo01.Item, JsonDotNetDefectDemo01",
          "PartNumber": 9234,
          "PartName": "Widget",
          "PartDescription": "Non-Existent part used for illustrative purposes only",
          "UnitPrice": 12.34,
          "PackageQuantity": 1
        },
        "Quantity": 5
      },
      {
        "$id": "5",
        "$type": "JsonDotNetDefectDemo01.OrderLine, JsonDotNetDefectDemo01",
        "PartOrdered": {
          "$id": "6",
          "$type": "JsonDotNetDefectDemo01.Item, JsonDotNetDefectDemo01",
          "PartNumber": 87424,
          "PartName": "Sham Wow",
          "PartDescription": "Indescribable",
          "UnitPrice": 19.95,
          "PackageQuantity": 12
        },
        "Quantity": 4
      },
      {
        "$id": "7",
        "$type": "JsonDotNetDefectDemo01.OrderLine, JsonDotNetDefectDemo01",
        "PartOrdered": {
          "$id": "8",
          "$type": "JsonDotNetDefectDemo01.Item, JsonDotNetDefectDemo01",
          "PartNumber": 9234,
          "PartName": "Widget",
          "PartDescription": "Non-Existent part used for illustrative purposes only",
          "UnitPrice": 12.34,
          "PackageQuantity": 1
        },
        "Quantity": 9
      }
    ]
  },
  "Buyer": {
    "$id": "9",
    "$type": "JsonDotNetDefectDemo01.Customer, JsonDotNetDefectDemo01",
    "FirstName": "John",
    "LastName": "Washburn",
    "VoicePhone": "262-555-6739",
    "CellPhone": "414-555-0734",
    "FaxPhone": ""
  },
  "ShippingAddress": {
    "$id": "10",
    "$type": "JsonDotNetDefectDemo01.Address, JsonDotNetDefectDemo01",
    "Street1": "N128W128795 Highland Road",
    "Street2": "",
    "City": "Germantown",
    "StateProvince": "WI",
    "PostalCode": "53022"
  },
  "BillingAddress": {
    "$ref": "10"
  },
  "PaymentMethod": 4,
  "PurchaseDate": "2015-03-15T01:03:31.7922884-05:00",
  "Capacity": 4,
  "Count": 3
}

But when the above JSON is read, the Order created and inflated by JSON.net has an empty list as the value of the property: OrderLines, instead of the 3 order lines found in the JSON text.

I have a full VisualStudio 2012 solution with that demonstrates this behavior. The solution has two projects; an implementation of the objects above (Order, OrderLine, Customer, Item, etc.) and a pair NUnit tests; One that serializes an Order to disk and then reads the JSON into a second instance of Order and a second that demonstrate the extension and Fascade design patter for the property work as expected. The test of the serialization round trip fails because the number of order list do not agree (de-serialized 0, but expected 3 OrderLines).

Is it possible to post this solution to you? Or is the information above complete enough?

[IanYates]

My suspicion is that JSON.Net is checking the property, seeing it as being not null, so it's populating the returned list from that property. Because you're returning a new list, that list is getting the order lines added to it but that returned list is never referenced.

As a test, could you modify your getter for order lines so it's like

    {
      if (this._stashedCopy == null) { this._stashedCopy = this.toList();}
      return this._stashedCopy;
    }

Then examine the contents of _stashedCopy after deserialization.

This could be avoided by NOT returning a .ToList() transformed copy of this from your getter. The alternative is to not have your Order object descend from List in the first place, which would force you in to the pit of success by just having a plain old list in a field of your order, exposed via a property.

[IanYates]

Note: you'll need to declare some private field called _stashedCopy in your class for that test to work :)

[JohnWashburn]

RE: _The alternative is to not have your Order object descend from List in the first place, which would force you in to the pit of success by just having a plain old list in a field of your Order Lines, exposed via a property._

True, but my objects (from the Beale Cipher) are lists with 2-5 extra properties/fields. Descending from List makes sense. Your proposal is I have a property within Order which is a List and have Order implement the IList interface using this internal List. This, works, but is a lot of work to get:

  myOrder.Add(new OrderLine());
  myOrder.RemoveAt(2);
  var selectedOrderLine = myOrder[3];

to work. The descent, 'Order : List' works fine in code until I serialize or de-serialize an Order object.

This is because because the elements of the parent, 'List', are not serialized.

My initial approach was to expose the parent object of Order (i.e. List) as a property of Order in order to get the OrderLines to serialize. Which worked splendidly for serialization. It is the de-serialization that failed, but it might be as you say.

The JSON.Net code probed the property, Order.OrderLine, just prior to populating the property by calling the getter and, upon getting a non-NULL result, the JSON.Net codes thinks its work is done and, thus, there is no need to call the setter.

I have a work around, but I have used JSON.Net for 4 years (and in some very interesting ways) and this is the _first_ time its behavior has _surprised_ me. Moreover, the surprise came in something I did not think was an edge or corner case.

Your suggestion of a stashed copy of a List field as a signal to the JSON de-serialization engine though might work. Let me get back to you on that.

Thank you for your suggestion on this.

[JohnWashburn]

RE: the use of .ToList()

It is a cheap way to create a shallow clone of a list.

In general, I do not like to return to a requestor a reference to the collection object (List, Dictionary, etc.) which my object manages (or interfices to the managed object). I provide a copy so that when they call Clear() on their copy of the List<OrderLine> my list of OrderLine objects is unaffected.

The more I contemplate this with regards to my Beal Cipher, the more I realize I should return a deeply cloned list not a shallow clone. Thank you for bringing that to my attention.
The merits of deep/shallow cloning of collections is way off topic though, and further discussion on that matter (cloning techniques) is closed on this thread. The front lines of that religious war can be found at:
http://www.jusfortechies.com/java/core-java/deepcopy_and_shallowcopy.php
and
http://java-questions.com/Cloning-interview-questions.html

[JamesNK]

Use ObjectCreationHandling.Replace

[IanYates]

Thanks @JamesNK - as usual, there's always an elegant way to force the library to do what you need. @JohnWashburn - thanks for the links - they look interesting so I'll add them to my reading list. I tend to deal with a lot of immutable objects (using the immutableObjectGraph library - I've even got JSON.Net working with it via contract resolvers and custom contracts) so I avoid the shallow vs deep issue for the most part.

[JohnWashburn]

I am using ObjectCreationHandling = ObjectCreationHandling.Replace for the serialization engine. It has become my default setting for the last several years. Here are the serializer settings I am using:

  if (null == jasonSerializer)
    return;

  jasonSerializer.MaxDepth = 32;

  jasonSerializer.NullValueHandling = NullValueHandling.Include;
  jasonSerializer.MissingMemberHandling = MissingMemberHandling.Ignore;

  jasonSerializer.DateFormatHandling = DateFormatHandling.IsoDateFormat;
  jasonSerializer.DateParseHandling = DateParseHandling.DateTimeOffset;

  jasonSerializer.ObjectCreationHandling = ObjectCreationHandling.Replace; // ObjectCreationHandling.Auto; // 
  jasonSerializer.Formatting = Formatting.Indented; // Formatting.None; //

  jasonSerializer.ReferenceLoopHandling = ReferenceLoopHandling.Serialize;
  jasonSerializer.PreserveReferencesHandling = PreserveReferencesHandling.All;
  jasonSerializer.TypeNameHandling = TypeNameHandling.All; // TypeNameHandling.None; // 

  jasonSerializer.TypeNameAssemblyFormat = FormatterAssemblyStyle.Simple; // FormatterAssemblyStyle.Full; // 
}

I will experiment with the settings and let you know if the issue becomes resolved.

[JohnWashburn]

I solved the issue. I had the public property to be {get; private set;} but failed to annotate the property with [JsonProperty] This behavior is documented ere: https://json.codeplex.com/discussions/222774

I am sorry to have bothered you on this.