Azure-functions-durable-extension: Allow configuration of JsonSerializerSettings in DurableOrchestrationContext and DurableActivityContext

[SimonLuckenuik]

This will enable for complex entities to be passed around (for example: polymorphism/discriminator converter).
Thanks!

[cgillum]

Thanks for the suggestion. I'm wondering what the best way to expose this is (it would likely need to be different for Functions users vs. WebJobs users). Which JsonSerializerSettings do you think would me most useful in your case?

[SimonLuckenuik]

I was thinking an overload when we get the input for deserialization and for serialization, another overload when we provide the value at workflow/activity start.

[andylholloway]

Hi
Is there an ETA for this? I currently have a couple of enum types that I want serialise as strings but there doesn't seem to be a way to configure the JsonSerializerSettings and the default behaviour is to serialise the enum value, e.g.

public enum EmailType {
        Work,
        Personal,
 }

serialises by default to:

{
....
    "EmailAddress": "[email protected]",
    "EmailType": 0
....
}

I would prefer it to be:

{
....
    "EmailAddress": "[email protected]",
    "EmailType": "Work"
....
}

I can get the enums to serialise as strings, but I'm required to decorate the enum type with [JsonConverter(typeof(StringEnumConverter))]. I would prefer not to have to do this and setup the StringEnumConverter globally via the JsonSerializerSettings.

Thanks.

[cgillum]

No ETA for this currently. It needs to be prioritized against other work items.

Can you elaborate on why this is important for your scenario? These JSON payloads are internal to the durable extension and generally are not expected to be surfaced to application code. I understand Simon's scenario because of the impact on polymorphism. It's less clear to me the benefit of changing the default enum serialization format.

[andylholloway]

Sure no problem.

When I make a request to the Orchestration's statusQueryGetUri the response contains the input data which was sent to the Orchestrator. If I do not decorate the enum types with [JsonConverter(typeof(StringEnumConverter))], they will be serialised to the default value, in my case an int.

example:
This snippet is from the statusQueryGetUri response. The Emails[] contains an Email type which has an EmailType property, the value for this property comes from the EmailType enum I mentioned in my previous post.

{
    "instanceId": "adf0bac590e8467e8cdde755801d956c",
    "runtimeStatus": "Completed",
    "input": {
        "$type": "Common.Models.Payload, Common.Models",
        "Entities": [
            {
                "$type": "Common.Models.Identity, Common,Models",
                "Id": "e64498a7-6be2-46f8-9fd5-655a46e89d43",
                "Emails": [
                    {
                        "$type": "Common.Models.Email, Common.Models",
                        "EmailAddress": "[email protected]",
                        "EmailType": 0,
                        "Primary": true
                    }
                ]
            }
......

When I decorate the EmailType enum with [JsonConverter(typeof(StringEnumConverter))]
I get the desired result:

.....
"Emails": [
                    {
                        "$type": "Common.Models.Email, Common.Models",
                        "EmailAddress": "[email protected]",
                        "EmailType": "Work",
                        "Primary": true
                    }
                ]
.....

If the JsonSerializerSettings were configurable then the StringEnumConverter could be set globally e.g.
MessageSettings.Converters.Add(new StringEnumConverter());

This isn't a major problem but its fairly common to have access to the JsonSerializerSettings (WebAPI, MVC as an example) and I'm sure there are other reasons for needing access to the JsonSerializerSettings such as setting how to handle default values, providing custom converters, etc.

Thanks

[SimonLuckenuik]

Using the integer value instead of string for an enum that is serialized/deserialized can cause hard to track problems, if the order of the enum members is changed and no specific value is assigned to each enum member

Example:

public enum EmailType {
        Work, // Value zero assigned by compiler
        Personal, // Value one assigned by compiler
 }

versus

public enum EmailType {
        Personal, // Value zero assigned by compiler
        Work, // Value one assigned by compiler
 }

in order to work around that potential issue, user must thing of assigning values

public enum EmailType {
        Personal = 0, 
        Work = 1,
 }

or have the serialization using a string by default and make sure users are not facing that issue.

[cgillum]

@andylholloway Ah, I hadn't thought about it from the REST API perspective - that's definitely a valid scenario for a feature like this. Thanks for clarifying.

@SimonLuckenuik that makes sense. Like you suggested, we could change the default enum serialization to use strings by default. That would be a breaking change, however, so it could only come out as part of a major version release.

[SimonLuckenuik]

First step of providing a way to customize the serialization settings would not be a breaking change, only forcing the new serialization settings to everyone would be.

Customization would be a good short term compromise for the developers, it would be up to them to force those settings if they need them and deal with the breaking change at the application level instead of the platform level.

[Costo]

One area where this is problematic is when using a JObject with DateTimeOffset instances. DateTimeOffset instances are serialized to ISO8601 strings, but by default JSON.Net will deserialize them to DateTime, therefore loosing the offset part.

One way to avoid this issue is to specify DateParseHandling = DateParseHandling.DateTimeOffset in JsonSeralizerSettings

[cgillum]

A few additional thoughts on how this could be done:

1. All methods which take object as a serializeable input could also take serialization settings as an optional parameter.
2. This could be configured globally as an injected service in IWebJobsStartup.

The first option is more flexible but clutters that API surface area quite a bit. The second option is simpler but less flexible.

[SimonLuckenuik]

Number 1 could be awkward to use I think, simply because I will need to provide the serialization settings at both serialization and deserialization time... I have a preference for globally. Having it globally is standard in platforms like ASPNET core with the AddJsonOptions extension.

If flexibility is required, what about having both options (starting with global for short term)? Configuring it globally so that default behaviors like null handling, date parsing is shared everywhere, and allow to override for specific cases (like the polymorphism example I gave initially) as an optional parameter, for special cases where global configuration could be overkill.

[jbeeko]

What about being able to specify this as part of the OrchestrationClientAttribute? This would be similar to what was done with the CosmosDB DocumentClient where custom serializer settings can be provided when creating the client.

As regards motivation, not being able to specify customer serializer settings is problematic when

1. We have defined customer serialization for the JSON accepted and emitted fin a REST client
2. When using some F# types with durable functions for example F# tupples don't seem to work as activity function parameters out of the box.

[cgillum]

One thing I'm considering for the 2.0 release (which relates to this issue) is allowing complete customization of serialization - i.e. not even limiting to Newtonsoft.Json. For example, there may be certain workloads that would benefit from binary serialization. It would be a bigger work item for sure, but it would also give you sufficient flexibility to provide deeper customization, even for JSON serialization.

[SimonLuckenuik]

I like it! I think that mixed with the extensibility to use a different storage could be very interesting and woulld add a lot of flexibility.

[gdodd1977]

What is the latest on this? Currently you can't put an abstract class on a context, which can really be limiting. It would be nice to have some more control over serialization and deserialization.

[cjaferre]

complete customization of serialization

This is definitely a goal, but I would stick with Json settings for now. It'll be quicker to implement and serve the majority of cases.

[oising]

Came here for this -- we're passing messages as DTOs but they're decorated with jsonignore in places but I wasn't expecting them to get stripped off between the orchestration and its activities... lol. I was hoping to swap out json.net with datacontractserializer