Materialize: Support ingesting JSON in Kafka

[benesch]

Right now we can ingest Kafka topics where records are encoded using Avro, but not when the records are encoded in JSON.

[benesch]

This is more complicated than it might seem at first glance, because the Confluent Schema Registry doesn't track schemas for JSON-formatted topics: https://github.com/confluentinc/schema-registry/issues/220.

[benesch]

After chatting with @rjnn and @cuongdo, the plan is to do something quick and dirty for demo purposes. We'll require users to describe the type of their records using JSON schema, inline in the USING SCHEMA ... bit of the data source. To start we're comfortable just chucking JSON records that don't match the schema.

[rjnn]

Probably work failing loudly rather than silently chucking it FWIW (e.g. put in a full row that's entirely NULLs in order to be visibly broken, or something of that sort. Might even be useful to stick JSON streams with an extra boolean column named parse_success or something of that sort.) Flexible on the actual solution, my main point is that anything reasonable that's loud-on-failure would be great.

[benesch]

We need to sort that out in general. Avro records can also be malformatted, and we just chuck those pretty silently right now. (We log a message to the console, where no one is looking.) It's a tricky problem in general; e.g., a parse_success column only works if all the other fields are nullable.

What we chatted about a while ago was a separate errors stream that you could monitor. I think that's probably our best bet, though it's true that it's not as loud as you might hope.

[rjnn]

The error stream sounds like the best solution, although it seems like more work. I was just spitballing on ideas to cut scope and make it loud cheaply.

[ruchirK]

This might be a dumb question but given that we have jsonb support why do we still need schemas for json objects / why not just chuck the whole thing into a jsonb column?

[rjnn]

@benesch, I think Ruchir is right here?

[benesch]

Agreed. Probably nothing to do here but document what you need to do.

[benesch]

Oh, here's another issue that explains why we might yet want JSON schema: https://github.com/MaterializeInc/materialize/issues/1494

Definitely don't need both of these issues though. Leaving this open because I think there's a bit of docs work to do, plus deciding whether it makes sense to have a FORMAT JSON format that just yields one JSON column.

[rjnn]

I'm closing #1494. There's a different way to articulate the-thing-we-might-yet-want, and I've filed #2679 instead.

[awang]

created a separate issue (#3176) for followup documentation, closing this issue as completed.