Confluent-kafka-dotnet: Error when serializing DateTime using AVRO

[keithazzopardi]

I am using Avro serialization to send a custom object as a message to the broker. One of my properties is DateTime and I am having a serialization error
"System.Int64 required to write against Long schema but found System.DateTime in field timestamp"

This is my message:

public class SomeEvent: ISpecificRecord {
  ....
public DateTime Timestamp { get; set; }
}

Schema defined as per Avro 1.8

{
   ""name"":""timestamp"",
    ""type"": 
         {
              ""type"" : ""long"", ""logicalType"" : ""timestamp-millis""
         }
}

Am I doing something wrong?

[mhowlett]

logical types aren't supported by the .net avro implementation. see also #855.

huge demand for this - noted.

[timjroberts]

I have an open PR for this: https://github.com/confluentinc/avro/pull/14

We couldn't adopt Avro without it.

[mhowlett]

At a quick glance @timjroberts your code looks great. THANK YOU! I should be able to get to reviewing/merging this soon after we get 1.0 released.

[raducrisan1]

I have an open PR for this: confluentinc/avro#14

We couldn't adopt Avro without it.

Hi,

How did you build it for .NET core?

[FurcyPin]

This issue is solved and can be closed.

I spent hours running in circles around this error, and now that I finally got it figured out I'll share a quick summary of what I learned.

• A long time ago, Confluent forked Apache.Avro and made Confluent.Apache.Avro which you were supposed to use within the Confluent Platform.
• That fork is now long dead (last commit January 2018) and you are now supposed to use the original Apache.Avro (sadly Confluent did not update the readme of that fork to tell users that they should not use it anymore)
• The PR mentioned above was made on Confluent's fork, which is dead, so it was never merged and never will.
• However it ultimately lead to this PR on the original Apache repo which was merged in February 2020.

Conclusion

I managed to make an minimal example work with logical types: sending Avro messages to Confluent Kafka with .NET
and reading them back with PySpark. For the .NET part I used the following dependencies :

<PackageReference Include="Apache.Avro" Version="1.10.0" />
<PackageReference Include="Confluent.Kafka" Version="1.5.2" />    
<PackageReference Include="Confluent.SchemaRegistry" Version="1.5.2" />
<PackageReference Include="Confluent.SchemaRegistry.Serdes.Avro" Version="1.5.2" />
<PackageReference Include="Microsoft.Avro.Core" Version="0.1.0" />

Warning : there is a nuget named Confluent.SchemaRegistry.Serdes which is dead since April 2019. It has been replaced with Confluent.SchemaRegistry.Serdes.Avro. It would be nice if Confluent updated that dead nuget to mark it as deprecated.

[mhowlett]

sorry for the confusion - have updated the old packages on nuget.org marking them as depreciated.

note that you don't need to reference the Apache.Avro package explicitly (as per the examples in the examples folder) - this is a dependency of Confluent.SchemaRegistry.Serdes.Avro and will be included automatically. Also note that Microsoft.Avro.Core isn't relevant and not used by any Confluent libraries. Finally, you don't need to reference Confluent.SchemaRegistry - it's a dependency of Confluent.SchemaRegistry.Serdes.Avro.