Materialize: Protobuf+Kafka Format for Upsert key is not yet supported

[macthestack]

What version of Materialize are you using?

materialized v0.6.1 (3247686f8)

How did you install Materialize?

Linux release tarball

What was the issue?

Is the issue reproducible? If so, please provide reproduction instructions.

CREATE SOURCE test FROM
   KAFKA BROKER 'localhost:9092' TOPIC 'tes'
   WITH ( cache = true )
   FORMAT PROTOBUF MESSAGE '.test.Test' USING SCHEMA '{schema data}'
   ENVELOPE UPSERT;

Gives me format for upsert key not yet supported.

What does this actually tell me? Can I not use upsert together with protobuf? The 'key' in my case is a string (utf8 bytes).

This may not be an issue. Is there a slack channel or similar where questions like this are more appropriate?

[benesch]

/cc @ruchirK @wangandi

And yeah, that sounds right. I think we only support Avro UPSERT sinks at the moment.

[awang]

Hi @macthestack! This Github issue is absolutely the right forum. If you're interested though, we do have a Slack channel setup as well: https://materialize.com/s/chat

[macthestack]

Thanks for the answers! Will check out the slack channel.

[wangandi]

If your key is just a utf8 string, then you need to specify that after ENVELOPE UPSERT like this:

CREATE SOURCE test FROM
   KAFKA BROKER 'localhost:9092' TOPIC 'tes'
   WITH ( cache = true )
   FORMAT PROTOBUF MESSAGE '.test.Test' USING SCHEMA '{schema data}'
   ENVELOPE UPSERT FORMAT TEXT;

Without the format specification, we assume that the format of the key is the same as the format of the value.

[wangandi]

FYI: We currently support the following types of keys with envelope upsert:

• plain utf-8 string
• raw bytes
• avro record
  The above three types are also the only three types of values we currently support with envelope upsert.

But if you really desire to use the combination of plain utf-8 key + protobuf value, please tell us. The code for it is already in Materialize; we just haven't turned on the switch due to lack of testing.

[macthestack]

Right now I'm just protoyping so avro works fine but I have several use-cases where are plain utf-8 key + protobuf value is more natural, especially with embedded producers. If you want to turn it on behind experimental I'll be happy to try it out.

[macthestack]

But if you really desire to use the combination of plain utf-8 key + protobuf value, please tell us. The code for it is already in Materialize; we just haven't turned on the switch due to lack of testing.

Coming back to this to check if it's possible to activate a protobuf upsert with utf-8 key combination. Most of the APIs I'm working with already have protobuf support and the developer story is just so much more convenient for us with protobuf. Completely understand if it's not as easy as flipping a switch, in which case I'll just have to wait for it to come out the natural way.

[wangandi]

Protobut value + utf-8 key combination has been turned on as of 051b145e15ced750c11b2bc0b144d4c79561b6bd.

[awang]

Thanks @wangandi ! This is great news, should be update the docs at https://materialize.com/docs/sql/create-source/protobuf-kafka/ with this?

[wangandi]

Yes, we should. And the release notes too.

[macthestack]

Thank you @wangandi . I haven't been able to try this out successfully yet. I'm having issues with parsing the filedescriptor I think. Will get back as soon as I have got it to work.

[macthestack]

I got it to work. Ran into an old issue with the inline schema that I just forgot that I already knew the answer to...

Anyway, thanks for enabling this!