Azure-functions-durable-extension: Serialization of DateTimeOffset loses Offset

[ADringer]

Hi,

I've been having an issue with my code and have narrowed it down to the Activity serialization of an object containing a DateTimeOffset.

In my orchestration i have an object such as:

var payrollGroup = new PayrollGroup {
   NextOccurence = new DateTimeOffset(2018, 24, 7, 0, 0, 0, new TimeSpan(1, 0, 0))
};

And then I call an activity passing this object:
await context.CallActivityAsync("CreatePayrollProcessing_CompletePayrollGroup", payrollGroup);

In this activity I did a log of the field when it started and got the following:

payroll group 1 has next occurrence at 7/23/2018 11:00:00 PM +00:00

So it looks like DateTimeOffset is being serialized to UTC and so losing the offset. Is it possible to keep the offset as is?

I'm using v1.5.0 of the Microsoft.Azure.WebJobs.Extensions.DurableTask.

Thanks

[cgillum]

Internally we're using Newtonsoft.Json to do all the serialization. Does this version have any known issues with DateTimeOffset?

[ADringer]

Not that I can find. I did a unit test that serializes my PayrollGroup object with Newtonsoft.Json v11.0.2 and deserialized it back and it kept the offset so out the box it's fine. I couldn't see it in the code, have any additional serialization settings been applied as part of JsonSerializerSettings ?

[ADringer]

Just trying to narrow it down, my Function app is located in UK West. Would these servers have their timezone set to UTC (but no daylight savings)? Just wondering if Newtonsoft.Json is deserializing to local machine time.

[ADringer]

So I've changed the code from:

public static async Task RunOrchestrator(
            [OrchestrationTrigger] DurableOrchestrationContext context, ILogger log)
{
    var payrollGroup = new PayrollGroup {
       NextOccurence = new DateTimeOffset(2018, 24, 7, 0, 0, 0, new TimeSpan(1, 0, 0))
    };
    await context.CallActivityAsync("CreatePayrollProcessing_CompletePayrollGroup", payrollGroup);
}

[FunctionName("CreatePayrollProcessing_CompletePayrollGroup")]
public static Task CompletePayrollGroup([ActivityTrigger] PayrollGroup payrollGroup, ILogger log)
{
    log.LogInformation($"payroll group {payrollGroup.Id} has next occurrence {payrollGroup.Frequency.NextOccurrence}");
}

To:

public static async Task RunOrchestrator(
            [OrchestrationTrigger] DurableOrchestrationContext context, ILogger log)
{
    var payrollGroup = new PayrollGroup {
       NextOccurence = new DateTimeOffset(2018, 24, 7, 0, 0, 0, new TimeSpan(1, 0, 0))
    };
    await context.CallActivityAsync("CreatePayrollProcessing_CompletePayrollGroup", JsonConvert.SerializeObject(payrollGroup));
}

[FunctionName("CreatePayrollProcessing_CompletePayrollGroup")]
public static Task CompletePayrollGroup([ActivityTrigger] string payrollGroupJson, ILogger log)
 {
        var payrollGroup = JsonConvert.DeserializeObject<PayrollGroup>(payrollGroupJson);
        log.LogInformation($"payroll group {payrollGroup.Id} has next occurrence {payrollGroup.Frequency.NextOccurrence}");
}

And it's now working as expected. So by de/serializing the object myself with JsonCovert keeps the offset, but using the underlying Json serializer serializes it to UTC

[cgillum]

Interesting - now I'm really curious why our own serialization mechanisms can't seem to do this correctly. I'm glad you found a workaround. We'll follow up on this issue to see if we can understand it better.

[Costo]

Hi,

We also have this problem and we managed to find the source of the issue. This happens because DurableOrchestrationContext.GetInput<T>() and DurableActivityContext.GetInput<T>() don't use the same mechanism to deserialize the input.

The Orchestration context deserializes the input directly using MessagePayloadDataConverter.Default.Deserialize<T>(this.serializedInput). In that case, DateTimeOffsets are deserialized correctly because the serializer can determine the correct type (DateTimeOffset vs DateTime) based on the destination property type.

https://github.com/Azure/azure-functions-durable-extension/blob/434f905c8eb2838fccf5c2139214b0948eb780e8/src/WebJobs.Extensions.DurableTask/DurableOrchestrationContext.cs#L107

The Activity context deserializes the input first to a JToken and then it is converted to the destination type. When deserializing to a JToken, ISO8601 strings are deserialized to DateTime objects by default (and converted to the local time zone), therefore loosing the offset part. Then the DateTimes are converted to DateTimeOffsets when converting to the destination type.

https://github.com/Azure/azure-functions-durable-extension/blob/434f905c8eb2838fccf5c2139214b0948eb780e8/src/WebJobs.Extensions.DurableTask/DurableActivityContext.cs#L84-L89

---

In order to fix this issue, MessageSettings property in MessagePayloadDataConverter needs to be changed to:

internal static readonly JsonSerializerSettings MessageSettings = new JsonSerializerSettings
{
    TypeNameHandling = TypeNameHandling.None,
    DateParseHandling = DateParseHandling.DateTimeOffset,
};

https://github.com/Azure/azure-functions-durable-extension/blob/434f905c8eb2838fccf5c2139214b0948eb780e8/src/WebJobs.Extensions.DurableTask/MessagePayloadDataConverter.cs#L19-L22

[TristanApc]

I also stumbled across this issue.
My input contains a datetimeOffset with the timezone offset of a particular user for the intent of sending personalized message with their own time displayed.

We were wondering why they kept receiving the message with an incorrect time but it was because the Activity isn't deserializing datetimeOffset correctly.

[Costo]

Internally we're using Newtonsoft.Json to do all the serialization. Does this version have any known issues with DateTimeOffset?

@cgillum In case my previous comment was not clear enough: By default, Newtonsoft "prefers" DateTime over DateTimeOffset when deserializing JSON to a JToken. This happens in DurableActivityContext.

[TristanApc]

@cgillum If this can help to investigate this bug, I created a reproduction case with this repository. It's running the latest version of Microsoft.Azure.WebJobs.Extensions.DurableTask (v1.8.3).

[ConnorMcMahon]

Not sure we can fix this without it being a breaking change.

You may be able to control this yourself in Durable v2.1 or greater by injecting your own IMessageSerializerSettingsFactory. We plan on documenting this as a part of #1208.

[drew-forsberg]

@ConnorMcMahon Any update on documentation for injecting our own IMessageSerializerSettingsFactory? Even a code snippet here would help. Thx

[ConnorMcMahon]

Anthony put a quick example on this stack overflow answer. It looks like right now this is not properly being used in Activity serialization, just in orchestration input/output serialization. Hoping to fix this for 2.2.0 release.

[amdeel]

Fixed in https://github.com/Azure/azure-functions-durable-extension/pull/1440