Materialize: Sink: `ERROR panic: not implemented: list types`

[nyancol]

Hi Materialize team,

I am using Materialize v.0.7.2 running inside a Docker container.

What was the issue?

Creating a sink for a view using a column of type list of records crashes Materialize. In the logs, it seems related to the following message (full tracelog at the end of the issue):

ERROR panic: not implemented: list types

Is the issue reproducible? If so, please provide reproduction instructions.

Create source topics

Create two topics: "NESTED_1" and "NESTED_2" and set the value's schema to: nested.avsc.txt

Create Materialize sources

create materialized source nested_1 from kafka broker '<kafkahost>:9092' topic 'NESTED_1'
format avro using confluent schema registry 'http://<kafkahost>:8081' envelope materialize;
create materialized source nested_2 from kafka broker '<kafkahost>:9092' topic 'NESTED_2'
format avro using confluent schema registry 'http://<kafkahost>:8081' envelope materialize;

Populate nested stream

sudo docker-compose exec schema-registry bash
/usr/bin/kafka-avro-console-producer --bootstrap-server kafka:9092 --topic NESTED_1 --property value.schema.file=nested.avsc.txt

NESTED_1

{"array":[{"data":{"id": "0","availabilityIntervals": [{"intervalStart": "A_1", "intervalEnd": "B_1"}]},"time":1,"diff":1}]}
{"com.materialize.cdc.progress":{"lower":[0],"upper":[3],"counts":[{"time": 1, "count": 1}]}}
{"array":[{"data":{"id": "1","availabilityIntervals": [{"intervalStart": "B_1", "intervalEnd": "C_1"}]},"time":4,"diff":1}]}
{"array":[{"data":{"id": "2","availabilityIntervals": [{"intervalStart": "D_1", "intervalEnd": "D_1"}]},"time":4,"diff":1}]}
{"com.materialize.cdc.progress":{"lower":[3],"upper":[5],"counts":[{"time": 4, "count": 2}]}}
{"array":[{"data":{"id": "3","availabilityIntervals": [{"intervalStart": "E_1", "intervalEnd": "F_1"}, {"intervalStart": "F_1", "intervalEnd": "G_1"}]},"time":6,"diff":1}]}
{"com.materialize.cdc.progress":{"lower":[5],"upper":[7],"counts":[{"time": 6, "count": 1}]}}

/usr/bin/kafka-avro-console-producer --bootstrap-server kafka:9092 --topic NESTED_2 property value.schema.file=nested.avsc.txt

NESTED_2

{"array":[{"data":{"id": "0","availabilityIntervals": [{"intervalStart": "A_2", "intervalEnd": "B_2"}]},"time":1,"diff":1}]}
{"com.materialize.cdc.progress":{"lower":[0],"upper":[3],"counts":[{"time": 1, "count": 1}]}}
{"array":[{"data":{"id": "1","availabilityIntervals": [{"intervalStart": "B_2", "intervalEnd": "C_2"}]},"time":4,"diff":1}]}
{"array":[{"data":{"id": "2","availabilityIntervals": [{"intervalStart": "D_2", "intervalEnd": "D_2"}]},"time":4,"diff":1}]}
{"com.materialize.cdc.progress":{"lower":[3],"upper":[5],"counts":[{"time": 4, "count": 2}]}}
{"array":[{"data":{"id": "3","availabilityIntervals": [{"intervalStart": "E_2", "intervalEnd": "F_2"}, {"intervalStart": "F_2", "intervalEnd": "G_2"}]},"time":6,"diff":1}]}
{"com.materialize.cdc.progress":{"lower":[5],"upper":[7],"counts":[{"time": 6, "count": 1}]}}

Create Sink

create materialized view nested_join as
select nested_1.id as id,
    nested_1."availabilityIntervals" as availabilityIntervals_1,
    nested_2."availabilityIntervals" as availabilityIntervals_2
from nested_1 left outer join nested_2
    on nested_1.id = nested_2.id;

The materialized view is properly created and can be queried:

mz> select * from nested_join;
+------+---------------------------+---------------------------+
| id   | availabilityintervals_1   | availabilityintervals_2   |
|------+---------------------------+---------------------------|
| 0    | {"(A_1,B_1)"}             | {"(A_2,B_2)"}             |
| 1    | {"(B_1,C_1)"}             | {"(B_2,C_2)"}             |
| 2    | {"(D_1,D_1)"}             | {"(D_2,D_2)"}             |
+------+---------------------------+---------------------------+
SELECT 3
Time: 0.019s

Although, when creating the sink, Materialize crashes (nested_join_bug.txt):

create sink nested_join_sink
from nested_join into kafka broker '<kafkahost>:9092'
topic 'NESTED_JOIN'
format avro using confluent schema registry 'http://<kafkahost>:8081'
envelope debezium;

Thanks
Yann

[ruf-io]

Thanks for taking the time to document this issue so well Yann. This is what the ideal bug report looks like!

We'll update this issue as we make progress.

[umanwizard]

@cirego Were you still planning on looking at this?

[cirego]

@cirego Were you still planning on looking at this?

I'm looking at #6296 at the moment. Would you like me to look at this as well? This issue isn't assigned to me and didn't show up on my list of things to pay attention to. Happy to look at this if it's urgent.

[awang]

@cirego can you work on this issue concurrently with the #6296 issue?

[cirego]

@cirego can you work on this issue concurrently with the #6296 issue?

Hey @awang, right now it looks like I won't be able to get to this issue for a few weeks. If we're okay waiting until I'm back, I'm happy to look at it then!

[umanwizard]

I'll take this since Chris is out.

[umanwizard]

@nyancol This should be fixed with the latest unstable Materialize; the fix will go to the stable version in v0.8.1.

We believe it's fixed, but please reopen if you run into any issues.