Materialize: Stack Overflow when creating a source with a protobuf schema with a recursively-defined type. (It should error instead).

[PeteJodo]

What version of Materialize are you using?

$ materialized -v
materialized v0.6.1 (3247686f8)

How did you install Materialize?

• [ ] Docker image
• [ ] Linux release tarball
• [ ] APT package
• [ ] macOS release tarball
• [ ] Homebrew tap
• [x] Built from source

What was the issue?

When trying to run CREATE SOURCE, materialized crashes with the following output:

thread '<unknown>' has overflowed its stack
fatal runtime error: stack overflow
fish: './materialized -w 3' terminated by signal SIGABRT (Abort)

Here's the offending SQL (protobuf file attached below):

CREATE SOURCE messages
FROM KAFKA BROKER '<broker omitted>' TOPIC '<topic omitted>'
FORMAT PROTOBUF MESSAGE '.repro.Status' USING SCHEMA FILE 'StatusSchema';

You may notice that I omitted the kafka broker and topic. I tested what would happen if I gave junk values for both (so values pointed to no real broker or topic) and still got the stack overflow.

Also, note that the examples on the documentation site for CREATE SOURCE for Kafka+protobuf are not entirely correct. It's missing FROM and just has USING '[path to schema]' rather than USING SCHEMA FILE '[path to schema]'. Therefore, I'm not sure if there's a mistake with my sql but I'm pretty sure regardless materialized should not crash.

Is the issue reproducible? If so, please provide reproduction instructions.

1. protoc --include_imports --descriptor_set_out=StatusSchema Status.proto (assuming Status.proto is in current dir. Proto file attached below)
2. materialized -w 3
3. psql -h localhost -p 6875 materialize
4. Run the SQL from above

Please attach any applicable log files.

materialized.log
Status.txt (Github only allows attachments of certain file types so just rename this from .txt to .proto

[rjnn]

Thanks for the thorough report @PeteJodo. We'll take a look and triage this.

[umanwizard]

@quodlibetor , could you take a quick look?

[quodlibetor]

Also, just to be clear, you mention stack overflow in your bug report, but the log just has a stack trace, did you get a different log somewhere else or are you referring to the error in the attached log?

[quodlibetor]

Oh, sorry I forgot that the error message you provided inline does in fact refer to a stack overflow, it's just that the log is a separate problem.

[PeteJodo]

That's interested because the query looks entirely unfamiliar to me. I didn't write a query like that at all so idk when that occurred. I was running the grafana docker that's provided separately so maybe that could be what was executing that query??

[quodlibetor]

@PeteJodo ah, sorry not that is on me, I confused a couple different log files.

[benesch]

So in your schema Status is a recursive type. I can 100% believe that Materialize would choke on that. Ideally we'd give you a nice error, but also recursive types are not something we support presently.

[umanwizard]

Here is how we handle recursive types for Avro now: https://github.com/MaterializeInc/materialize/blob/56051a13b745163bacd365b95465a49468d031c4/src/interchange/src/avro.rs#L1033-L1036

We should give a similar error for Protobuf types.

[umanwizard]

@PeteJodo are you able to use a workaround that doesn't involve recursive types? Unfortunately, support for these isn't on the roadmap for Materialize at the moment.

(Issue name changed to reflect this).

[PeteJodo]

Ahh yeah that seems like the main candidate for causing the issue.

I was actually trying out materialize in a work hackathon and so I didn't design the schema nor do I need a short term solution to this. I actually tried using an avro schema reading from a different kafka topic which required me to edit the schema to remove the recursive types. Materialize had surfaced that correctly in that instance. Regardless I couldn't get it working due to how we're serializing avro 😢 and so that led me to try a topic outputting protobufs but missed the recursive type in the schema.

[PeteJodo]

May I ask what the complexity regarding recursive types are to materialize? Like if I was to use the differential dataflow lib and just did everything I needed myself, would I run into a similar set of issues for why they're not supported in materialize?
Just curious, thanks!

[umanwizard]

The issue is with Materialize, not differential-dataflow: differential-dataflow doesn't know about Materialize's type system; it just uses normal Rust objects, and it's fine for the object graph (or its type) to have cycles.

Materialize's type system does not have any notion of backreferences from one node to another. So trying to convert a Protobuf (or Avro for that matter) type with cycles in it to a Materialize SQL type would result in something of infinite size. We could probably fix this in principle, but it would be a moderately large refactor.

[umanwizard]

For example, if you have this Protobuf schema:

message Status {
  Status retweeted_status = 8;
}

We would try to convert it into a type that looks like (in pseudocode):

record Status {
    retweeted_status: record Status {
        retweeted_status: record Status {
            retweeted_status: record Status {
                ...

... ad infinitum