Materialize: Support sending JSON-encoded data to Kafka sinks

Created on 14 Jan 2020  路  6Comments  路  Source: MaterializeInc/materialize

Right now, we only support Avro-encoded data. This will likely expand the syntax of CREATE SINK.

A-integration C-feature

Most helpful comment

This is fairly high priority, so adding it to public beta.

All 6 comments

@rjnn is there a milestone we should assign to this issue?

This is fairly high priority, so adding it to public beta.

There are a couple things we'll need to tackle in this ticket:

  1. Expanding CREATE SINK to take a format option that indicates either avro or json.
  1. Potentially creating multiple KafkaSinkConnector structs, one for each format type?

  2. Figuring out how to turn Materialize Row objects into JSON objects. Looking around, it seems like there are three options:

    1. Require a JSON schema and emit data alongside the schema (Confluent has JSON schema management "targeted for Q1 2020".)
    2. Emitting JSON data without a schema... we would have to figure it out?
    3. Emitting raw JSON serialized as a string.

Useful references:
https://docs.mongodb.com/kafka-connector/master/kafka-sink-data-formats/

Because the POC that depends on this is delayed, this should be completed by or soon after 0.2.

Any new milestone for this?

Any update on this? Looking forward to it.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

nyancol picture nyancol  路  7Comments

mathieuisabel picture mathieuisabel  路  6Comments

JLDLaughlin picture JLDLaughlin  路  7Comments

johnjmartin picture johnjmartin  路  3Comments

benesch picture benesch  路  5Comments