Confluent-kafka-dotnet: Is there support for multiple schemas per topic when using avro?

[Quickstream]

I'm trying to use avro schema registry with multiple schema types per topic, as discussed in this blog post: https://www.confluent.io/blog/put-several-event-types-kafka-topic/

It mentions setting key.subject.name.strategy and value.subject.name.strategy to one of 3 options:
io.confluent.kafka.serializers.subject.TopicNameStrategy
io.confluent.kafka.serializers.subject.RecordNameStrategy
io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

I'd like to set it to TopicRecordNameStrategy, but I can't figure out how to do this with the .net client. Is it possible?

[mhowlett]

no, we haven't added this functionality yet. it won't make it into 1.0, but we'll add it after.

[jblackburn21]

PR #585 had covered this, but it is outdated from the 1.0 api changes. I have a fork that I've been using to get around it https://github.com/jblackburn21/confluent-kafka-dotnet/tree/avro-subject-name-strategy

[mhowlett]

also, IIRC we have more than one customer asking for this, and it's a quick win, so I expect it will get done fairly soon after the 1.0 API is finalized.

[WSt89]

Can you tell when multiple Avro schemas per topic will be supported by your dotnet client?

[paddypawprints]

@mhowlett - I'm looking into this at the moment and have something that basically works. Is the approach below anything like what you are planning? If not, can you tell me your design approach? I'd ideally like not to stray too far from the beaten path.

Thanks.

I'm using custom serializers based on SchemaRegistry.serdes and using the schema subject in the message to discriminate the type for deserialization. For the producer and consumer, don't set a serializer or deserializer when building the producer/consumer - produce and consume binary.

Internally I have a registry of subject, version and C# Type.

For serialization, use the type to look up the subject and version and find the target schema. Then create the right serializer and produce the binary message.

For deserialization, get the schema id from the message, look up the subject and version from the schema registry, and use these to look up the target type for the deserializer.

[terryvrooman]

@paddypawprints I'm trying to do this exact thing, while waiting for the official implementation.

Would you have any example code to share?

[codyrehm]

We also have a strong interest in this. The original posting is asking specifically about the inclusion of the new configuration values for the schema registry client. I am also interested in any plans to allow support for multiple deserializers for a single topic. Are there any plans on the horizon to allow a consumer to register multiple deserializers?

[mhowlett]

Not sure exactly what your requirements are, but you could easily write a IDeserializer<object> and return different types based on the data, or just parse byte[] data inline in your consume loop. At some point you need to do a switch on type, and we decided a standard framework for this was not worth including in the library. Feel free to demonstrate an API you'd like to see if not the above.

[codyrehm]

Thanks for the suggestion! I'll look into it and do some thought on an API.

[paddypawprints]

@mhowlett - thanks for the feedback.

Here is where I'm up to. I'm checking internally whether I can put the code up on GitHub. Until that happens if you are interested mail me at patrick.[email protected] for details

1. Extended to Avro specific serializers to work with POCO classes rather than classes generated via avrogen. Also removed the tight coupling between the schema namespace and names and C# classes which will make it easier for developers with their own classes to integrate.
2. Wrote custom serializer that takes the schema registry subject and version as an input. The subject and version are unrelated to topic and are specified by the developer.
3. Created consumer and producer that serialize/deserialize outside of the dotnet client.

It isn't production code by any means but seems to work ok in testing so far.

[mickdelaney]

@mhowlett

Am i right in interpreting the position of the confluent-kafka-dotnet team as

"dont use the schema registry in .net if you have multiple types per topic"

if i have misunderstood sorry, but i was hoping we could use the schema registry and get all the schema evolutionary benefits etc.

[mhowlett]

you can't at the moment. we will add support in the future, quite likely early Q3.

[mickdelaney]

@mhowlett
I've got a example running here:
https://github.com/mickdelaney/kafka-spike/

basically the SerDe stuff is here:
https://github.com/mickdelaney/kafka-spike/blob/master/src/Core/Kafka/AvroSubjectNameStrategyDeserializer.cs#L77

https://github.com/mickdelaney/kafka-spike/blob/master/src/Core/Kafka/SubjectNameSerializer.cs#L66

Using:
https://github.com/mickdelaney/kafka-spike/blob/master/src/Core/Kafka/SubjectNameFactory.cs
https://github.com/mickdelaney/kafka-spike/blob/master/src/Core/Kafka/SubjectNameSchemaCache.cs

And Usage Wise Its a Switch On Object.
https://github.com/mickdelaney/kafka-spike/blob/master/src/Core/Domain/Users/UserConsumer.cs#L51

[mickdelaney]

@mhowlett we’ve gone with Avro Unions per topic for now.

[SeniSeattle]

@mhowlett I noticed you said this feature will be available "more likely early Q3". has this been released already? if not, do you have a new timeline? My team also needs this feature for a project we are working on.

[mhowlett]

sorry, it's not done yet. commonly requested, it's coming.

[JohnTicehurst]

Hi,
I've also been watching this space, can you forecast a date ?

[mhowlett]

additional +1 noted

[dsrkeddi]

which version of confluent will support multi Avro schema to one topic? is there any date planned.

[aoffpo]

Watching this as well.

[martelskiy]

weNeedThisFeature

[mhowlett]

it'll be in 1.3 for sure. i hope to get it merged to master this week.

[mhowlett]

just merged #1070 to master. expect to do a 1.3-PRE1 release that includes this later this week based on librdkafka 1.2.1, also due later this week.

[JarrodJ83]

@mhowlett thanks for all you great work on this! This was one gap I was hoping to see filled before my company got moving forward with our event driven architecture journey.

I was wondering, will there be any follow up to this on the consumer side to provide something similar to the AvroSubjectNameStrategyDeserializer @mickdelaney built? Something to allow the consumer to deserialize using Avro based on the SubjectNameStrategy? I didn't see any code or issues pointing to this. If I am missing it could please point me in the right direction? Maybe it is easy/possible to do with everything that is currently int he library and I am just not seeing how to piece it together. Any info would be much appreciated!

[martelskiy]

@mhowlett is it an option to implement SpecificDeserializer with a constructor that will take for example HashSet<Avro.Schema>. So it will be the collection of the reader's schemas. Then you could just do something like this:
var readerSchema = _readerSchemas.FirstOrDefault(schema => schema.Name == writerSchema.Name);

It will allow for deserializing multiple event types per topic. Deserializer will return an object type. Then you can perform simple type check
object is %yourType%

[mhowlett]

it's an option (i think it wouldn't be too hard, you can always write a custom serializer). I'd prefer an API with strong type checking though. I do want to make a producer without type parameters, and here's the reason you might want that on the consumer. You could then perhaps have TryConsume<K,V> methods (and C# can infer the K,V, so they don't need to be specified which is nice). anyway, that's not going to be done imminently. making a custom deserializer as you suggest sounds like a good idea.

[niksgupta36]

additional +1 noted

My team is also looking for this update. Please let us know when this is gonna be released. Thank You!

[m1nkeh]

is there documentation on this feature now that 1.3 is released?

[Pabi171]

Hi All, I have gone through this post but not able to figure out - how to create a avro schema from a C# class? - Any help on this.

[paddypawprints]

Hi Pabi,

I did some work on this - see here https://github.com/paddypawprints/schemagen

It uses Roslyn to parse c# and generate an avro schema (or protocol). If you build it the program has a —help option.

It is parked here and I am trying to clean it up to contribute it to the avro project. I’d appreciate any comments you have if this work is interesting to you.

Sent from my iPad

On Dec 23, 2019, at 1:56 AM, Pabi171 [e-mail] wrote:



Hi All, I have gone through this post but not able to figure out - how to create a avro schema from a C# class? - Any help on this

comments You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/confluentinc/confluent-kafka-dotnet/issues/746?email_source=notifications&email_token=AFCP6V62MNZNU5EYGL5C55LQ2CDOHA5CNFSM4GRDFJ5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHQXZXQ#issuecomment-568425694, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFCP6V7FEJXWEZPIQLIMGFTQ2CDOHANCNFSM4GRDFJ5A.o

[mhowlett]

thanks for your efforts @paddypawprints - there are various other similar efforts to make a better avro implementation about as well.

i want to note here that protobuf and json support is coming to SR, and i expect to be adding .net client support for this this quarter. These formats are a slight mismatch for the use case, but are close enough and have the benefit of having first class support in .net.

[vaclavpfeifer]

Any update for strategy support for C# client