Schema-registry: Docs: clarify semantics of schema ids?

[miguno]

Edit: Found the answer in the docs at Design. I confused _versions_ of a subject (1, 2, 3, 4) with the _ids_ of schemas.

[miguno]

And the background for this question: Some of our users have expressed interest in tracing the "history" (think: git commit log) of schemas registered with a subject.

[ewencp]

@miguno Hopefully most people don't need to worry much about the guarantees provided for ordering of IDs. But given that you had to dig around the docs a bit to find the info you wanted, if you have a suggestion for how to improve the docs, especially if we should do something like define version ID vs. schema ID more clearly in the intro documentation, we'd like to hear it!

[miguno]

Yeah, I agree that most people don't need to worry about the guarantees. I think I got lured into wondering about such guarantees because the wording in the docs _suggest_ there are some, although possibly not explicitly stated -- and hence I wondered whether there should be A) clearer wording if there are indeed such guarantees or B) a clarification that there are in fact none (like: "Currently, schema IDs are monotonously increasing numbers, but this is an implementation detail that may change at any point in future releases, so don't count on it!").

I'm happy to send a PR with a suggestion to improve the docs as long as you could let me know whether you envision case A (yep, there are some guarantees) or B (nope, there are none).

[In our case, for example, schema IDs look like random alphanumeric chars similar to git commit hashes, and we clearly describe that there's no "method to this madness", at least none that we guarantee by contract. ;-)]

[ewencp]

@miguno I think one potential problem in trying to use schema IDs for reasoning about ordering is that a schema could be used in different subjects. So we might register schema A for topic 1 and get A -> schema ID 1. Register schema B for topic 2 and get B -> schema ID 2. Then, if we register schema A for topic 2, we'll reuse the same schema ID and the schema IDs for the versions for topic 2 will not be monotonically increasing.

I think this reuse means there aren't any useful guarantees based solely on schema IDs. @granders could probably verify this reasoning is correct -- he's more familiar with that code than I am.

I still think a) clarifying the different namespaces (schema ID, version ID), b) explaining explicitly the guarantees, or lack thereof, and c) maybe even explaining an example like the above to show why trying to use schema ID ordering can lead to errors would be useful.

[miguno]

Interestingly I interpreted the docs a bit differently for the use case you described (A -> 1, B -> 2). So I'm adding to the confusion. ;-)

Since the API refs of /subjects/foo/versions and /subjects/foo/version/<versionId> say (a bit ambigously) _version_ of the returned schema, I thought that e.g. [1,2,3,4] was not a list of schema IDs -- but instead sth like an increasing series of "commit numbers" (which could possibly even be a range with start and end index, if it would increase by +1 always). So querying for /schemas/foo/versions/2 would not return the schema with id 2 (B in your example) but rather the second schema that was registered with the subject, regardless of that schema's id.

But if I understand you correctly, [1,2,3,4] would actually be a list of schema IDs, and in practice would look rather like [62, 78, 100] or even [62, 5, 100], and the position/index in this list would tell the user that, say, 5 was the second schema ever registered with the subject.

I agree with your suggestion in the last paragraph. I'm happy to draft a PR once it's clear what the intended behavior / (lack of) guarantees are.

[granders]

Ah, sorry for the confusion, we should definitely update the docs to clarify these concepts.

• subject - a subject is a string with an associated "schema history". This history is durable and immutable. Each subject has its own schema history.
• version - the index into a given subject's "schema history". Version numbers are unique within a subject but not globally unique. Version numbers are indexed from 1 and always increment by 1 - so the first schema registered under subject "kafka-key" would be version 1, and then future schemas under "kafka-key" would have versions 2, 3, 4 etc.
• id - ids are _globally_ unique and monotonically increasing, although there can be gaps between assigned ids. Since the same schema can be registered under different subjects, different "commits" to different subjects of the same schema will have the same schema id, but probably not the same version.

Speaking of "id" and "version", what happens when you register a schema underneath a subject? - there are a couple possibilities, depending on whether the schema has been registered before in the registry, and whether it has been registered underneath the subject.

a) The schema has never been registered: assign a new globally unique schema id; append the schema to the subject's schema history with an incremented version number; return the globally unique schema id

b) The schema _has_ been registered before, but not underneath the given subject: lookup the globally unique id already associated with the schema; append the schema to the subject's schema history with incremented version number; return the preexisting schema id

c) The schema has been registered before under the given subject: lookup the schema id and return it

Below is an example of what the schema history underneath two different subjects might look like

subject: "user-value"

• schema: "aaa", version: 1, id:57
• schema: "aab", version: 2, id: 2
• schema: "aac", version: 3, id: 42

subject: "author-value"

• schema: "aab", version: 1, id: 2
• schema: "zzz", version: 2, id: 32
• schema: "aaa", version: 3, id: 57
• schema: "abc", version: 4, id: 1

Hopefully this clarifies things!

[granders]

Also, note that there's a small error in the api doc for fetching a schema, see #161

[miguno]

Thanks for the clarifications, Geoff!

One follow-up question regarding a corner case and its impact on "latest". Since "latest" is an important feature for Camus integration and thus Kafka->HDFS ingestion, clarifying how schema-registry works in such a scenario might be helpful for users.

c) The schema has been registered before under the given subject: lookup the schema id and return it

What happens if a schema is re-registered again under the same subject foo? An example use case would be to undo a faulty registration of the previously "latest" schema (typos, whatever), or that an application update that introduced the currently "latest" schema will need to be rolled back.

Given the following schema version history for subject foo (and I exclude the impact of schema compatibility checking):

schema A, version: 1, id: 463
schema B, version: 2, id: 1072

Here, /subjects/foo/versions/latest would return version=2 aka schema B.

Now the user realizes that something's broken with B, or that the application update that introduced B will need to be rolled back. At this point we want A to become the "latest" version again.

So the desired effect would be something like this:

schema A, version: 1, id: 463
schema B, version: 2, id: 1072
schema A, version: 3, id: 463

How does this work in the current API? I think the (or rather, my) confusion is due to the fact that POST /subjects/<subject>/versions does two things at the same time: 1) it registers a (known) schema with a subject, and 2) if the posted schema is not known, it will automatically create it and then perform (1) afterwards.

In comparison, our API requires two explicit POST operations for this -- first POST /schemas to explicitly create the schema (and e.g. get a generated schema id in response), and then POST /subjects/<subject>/versions to register the schema with the subject (via the schema's id). This setup handles re-registering a schema (A -> B -> A) quite cleanly.

(PS: I hope this isn't a distracting discussion. If so, feel free to close the ticket.)

[ewencp]

If you try to re-register under the same subject you'll get back the old ID. If you allow re-registering, then some other APIs become more complex and/or more confusing. For example http://confluent.io/docs/current/schema-registry/docs/api.html#post--subjects-(string-%20subject) wouldn't necessarily return a single version ID anymore.

There's definitely a tradeoff here -- making registration/ID lookup a single API call is nice from the client's perspective, including reducing latency. On the other hand, this is hidden from the user anyway and the latency only costs you on the node that actually does the registration. Then again, we also expect that issues like you're talking about to generally be resolved during development, before the registration would ever hit a production server, but obviously things don't always work out that way.

@granders or @junrao might be able to give a more detailed response, and if the lack of rollback-like registrations (aside from hacks like slightly modifying the schema so it doesn't quite match the original) is a problem, we should probably file an issue for that and figure out how to make the API handle that case well.

[FelixGV]

Since avro has first-class support for doc fields, it's not too far-fetched to require a change of schema literal for registering new IDs. If you really want to roll back a schema, then you should probably have some metadata saying why you are doing it, so it seems reasonable to mention that in the doc field.

Alternatively, there could be a concept of read-only schema, meaning that a schema definition is kept for being able to decode old messages, but that it is discontinued for writing new messages. In that case, the concept of asking for the latest schema could be meant to represent the "latest writable schema". This need has been mentioned in the past in AVRO-1124.

[ewencp]

@FelixGV great point! But we'd _definitely_ need to document that as the expected approach to handling rollbacks. Unless you're familiar with Avro, this solution isn't going to be obvious.

On read-only schemas, you could also refer to the action of converting them to read-only as "deprecation" or "invalidation". Adding this feature means that clients would need to check in with the schema registry periodically. Otherwise a long-running producer would continue writing data with a schema that's been made read-only. Not necessarily a problem, but yet another tradeoff.

[miguno]

First, just to be clear: My main intent for this discussion is to clarify the docs of the current code in schema-registry. It's not about asking for a change. When I talk about how our version of schema-registry works, I am only giving another perspective -- I'm not implying one is better or worse than the other. ;-)

Since avro has first-class support for doc fields, it's not too far-fetched to require a change of schema literal for registering new IDs. If you really want to roll back a schema, then you should probably have some metadata saying why you are doing it, so it seems reasonable to mention that in the doc field.

Yeah, and interesting that you mention this example. In our case, we decided to decouple this information / these semantics from the contents of schemas though. Instead, we allow users to provide a "commit message" to go with registrations -- same reasoning as in git. We considered modifying the schema content like you described but decided against it. We even normalize schemas so that we don't create new schemas unnecessarily (think: pretty-print version of a schema vs. the compact version).

[FelixGV]

Relying on properly behaved producers may or may not be achievable,
depending if people are starting fresh new Kafka deployments or not, but an
auditing infrastructure could at least detect badly behaved tenants.

I think at some point there were also talks about doing broker-side
validation that would outright reject bad messages instead of acking them,
but I don't know how feasible that is without sacrificing Kafka's
performance...

-F
On Thu, Apr 2, 2015 at 22:58 Ewen Cheslack-Postava [email protected]
wrote:

@FelixGV https://github.com/FelixGV great point! But we'd _definitely_
need to document that as the expected approach to handling rollbacks.
Unless you're familiar with Avro, this solution isn't going to be obvious.

On read-only schemas, you could also refer to the action of converting
them to read-only as "deprecation" or "invalidation". Adding this feature
means that clients would need to check in with the schema registry
periodically. Otherwise a long-running producer would continue writing data
with a schema that's been made read-only. Not necessarily a problem, but
yet another tradeoff.

—
Reply to this email directly or view it on GitHub
https://github.com/confluentinc/schema-registry/issues/158#issuecomment-89179617
.

[miguno]

There's definitely a tradeoff here -- making registration/ID lookup a single API call is nice from the client's perspective, including reducing latency. On the other hand, this is hidden from the user anyway and the latency only costs you on the node that actually does the registration.

Yes, exactly.

Then again, we also expect that issues like you're talking about to generally be resolved during development, before the registration would ever hit a production server, but obviously things don't always work out that way.

Again, agreed. We expect the very same in theory but realize that in practice, theory and practice are not the same. ;-) We also wanted to be sure that we can actually track that we revert back to _exactly_ the same schema as we used in the past, which is one reason why we don't want to introduce "fake" changes into a schema just for the sake of making it different and thus allow re-registration.

In our case our reasoning behind the current API was somewhat like this. Again, not implying this is any better than what schema-registry is doing. Just summarizing another perspective on the same problem.

IMHO the most important part of the API is the creation of _schemas_ (returning a schema id) and the lookup of schemas (given an id). At this point we are now pretty flexible. We can model "A topic should only ever have one single schema", or "A topic may have evolving schemas over time", or "A topic may contain messages with totally different schemas", etc. In particular, consumers can now decode any messages as long as the messages include a schema id.

But this alone is not enough in practice, at least not for us. Users want to know "Hey, if I read from topic foo, what does the data look like?" (also, we are considering allowing users to do full-text queries on schemas/topics, for instance -- "Show me all schemas that include an IP address"; details up in the air, maybe via Elasticsearch integration or somesuch). Camus, on the other hand, technically requires us to answer the question "What's the latest schema for topic foo, because it will try to convert messages encoded with [A, A, A, B, B, C] to [C, C, C, C, C, C] if C is the latest schema.

So we needed to introduce the notion of _subjects_ to give some order to schemas, model how they relate to subjects aka Kafka topics, and thereby introduce conventions/constraints how we use them in Kafka. One such convention (or assumption) in our case -- and from what I understand also in schema-registry -- is that we expect that messages being sent to a topic will typically be encoded with the same, current schema. "Current" implies order. Still, in practice you may still see data entering a Kafka topic that was encoded with previous schemas (e.g. because of rolling upgrades of producer applications) -- so at least in our case we are not enforcing "one and only one schema may be used to send messages".

For these reasons we decoupled the management of schemas and subjects from how schemas and subjects are mapped to each other. The "mapping" in our case is very much like a git commit history, though we only allow 1 parent per "commit" of course (i.e. strict linear ancestry). This is a way (but not the only way) to model A -> B -> C schema evolution, given a subject, as well as corner cases such as rollbacks A -> B -> C -> A.

Alternatively, there could be a concept of read-only schema, meaning that a schema definition is kept for being able to decode old messages, but that it is discontinued for writing new messages.

Read-only schemas are an interesting idea, thanks for bringing this up! This is not something we have considered yet.

[dossett]

Regarding @ewencp's comment "If you try to re-register under the same subject you'll get back the old ID." Is that documented behavior that can be relied upon?

Thanks!

[cloud-fan]

id - ids are globally unique and monotonically increasing, although there can be gaps between assigned ids. Since the same schema can be registered under different subjects, different "commits" to different subjects of the same schema will have the same schema id, but probably not the same version.

The current API is not very consistent about this. If schema ID can be shared between different subjects, why do we have SchemaRegistryClient#getId(String subject, Schema schema) and SchemaRegistryClient#getBySubjectAndId(String subject, int id)? These APIs make people feel that the schema ID is bound to a subject.

[mageshn]

@cloud-fan the client methods getBySubjectAndId is in there only for performance reasons where AvroConverter need to look up schema by subject and version and also modifies the schema by adding a property for the version. Client implementation is optimized towards how its used in serializers and converters. The real API's are the one on the server side.

[mageshn]

CLosing the issue since there is doc added to talk about schema id allocation. https://docs.confluent.io/current/schema-registry/docs/index.html#schema-id-allocation