Ksql: Allow the specification of the schema-registry subject when creating avro tables/stream

[dguy]

Currently when we create tables and streams we infer the avro subject from the topic name. This works great if everything has been published via the confluent stack, however, there are cases where this might not be the case and people want to share a schema across multiple topics. It would be nice to add an extra value to the WITH clause such that an operator can set the subject to a different value

[noelalonso]

Hello,
When do you expect that this bug will be solved? I have a problem creating streams with Avro.

Error:

Unable to verify if the Avro schema for topic test-topic is compatible with KSQL.
Reason: Avro schema for message values on topic test-topic does not exist in the Schema Registry.
Subject: test-topic-value

If I get subjects from schema registry, test-topic-value subject doesn't exist, so I think I have to set the schema-registry subject when creating avro tables/stream.

[darkwings]

This would be a really interesting feature, that will open up many possibilities, especially if we consider that (as per the excellent https://github.com/confluentinc/schema-registry/pull/680), schema registry already supports the publishing of different AVRO objects on a topic.

What I would like to do in KSQL CLI is something like the following:

CREATE STREAM xxx AS SELECT ... WITH (KAFKA_TOPIC='topic1', VALUE_FORMAT='AVRO', SCHEMA='topic1-com.mypackage.MyMessage")

which creates a stream that is filtered on the specific type of message, even when the topic hosts a plethora of different types of objects.

Ok, I perfectly know that a possible answer could be "you can create as many topics as the objects you want to handle", but there are cases in which it is important to publish different events (related to the same entity, that is, events that share the same partition key) in a topic, in order to keep the ordering of events.

[darkwings]

Actually, this is a duplicate of #1267. Up to you to close it or link it to the original one.

[renatowu]

I don't think this is a duplicate of #1267 as that one is about multiple schemas per topic and here I understand that we want a way to reuse an already existing AVRO schema when creating a new KSQL stream. But this issue may be a duplicate of #2427 where I have proposed something similar to what @darkwings proposed above (a SCHEMA argument on the WITH clause to set which AVRO schema should be used).

For me it would be great to reuse AVRO schemas on KSQL Streams. Or at least there should be a way to add documentation on the auto-generated AVRO schemas create by KSQL.

[big-andy-coates]

related to https://github.com/confluentinc/ksql/issues/3634