Schema-registry: Add support for spark structured streaming

[ghost]

There is no easy way to integrate the schema-registry to a spark structured streaming application using the available componenents.
Simply adding the parameter schema.registry.url with value.deserializer as io.confluent.kafka.serializers.KafkaAvroDeserialize has no effect on the available kafka sink (https://spark.apache.org/docs/2.3.0/structured-streaming-kafka-integration.html).
Are you planning to support the integration between the three?

[OneCricketeer]

Your SO question was answered: https://stackoverflow.com/questions/48882723/integrating-spark-structured-streaming-with-the-kafka-schema-registry

Also I'll be marking as a duplicate. https://stackoverflow.com/questions/40705926/reading-avro-messages-from-kafka-with-spark-2-0-2-structured-streaming

[ghost]

Thanks @cricket007 ! It's the "workaround" I implemented too. Still it's not as obvious as one would do with a parameter "schema.registry.url" in the kafka source.

[OneCricketeer]

Personally, I feel that should be an issue for the Spark community to integrate

Or even databricks/spark-avro

[davidahern]

hi,
i raised this ticket recently on main Spark project once i saw they had better Avro integration and this is their response
https://issues.apache.org/jira/browse/SPARK-26314
they feel it's an issue for the vendor, Confluent in this case, to integrate

Last time i worked with this was a year ago and i encountered the issues the other people have described
I hope that either Confluent can provide better integration or work this the Spark community to do this
Maybe they can look to Phoenix Spark for inspiration? This integration is pretty seamless.

[OneCricketeer]

I think the community has since been using

https://github.com/AbsaOSS/ABRiS

I'm not speaking for anyone, but Spark accepts pull requests, and I'm not sure Confluent is designating resource(s) for maintaining a section of the Spark code, or non-Confluent products.

For example, Flink, Nifi, Streamsets are other Apache projects that work with the Confluent Schema Registry, and I believe those were contributed by third-party developers

My point here is that Spark integration should be put there, not within this repo

[davidahern]

Yes. I mentioned that project in my ticket on the Spark Jira. It looks like it was developed in the meantime since I looked at this over a year ago. Since I saw the Avro announcement for Spark 2.4, I thought to raise this and hopefully save some ppl pain in the future if they can get this integrated. I saw your comment and you make a good argument for them to include this kind of code. Let's hope they do and this works out of the box for anyone who may have to consume Confluent with Spark Structured Streaming in the future.

[OneCricketeer]

For anyone else that stumbles across this, there is more sample code here - https://github.com/xebia-france/spark-structured-streaming-blog/blob/master/src/main/scala/AvroConsumer.scala