Ksql: Issue with accessing array of struct using "EXPLODE()->" in 0.11.0 ( which works in 0.6.0)

[tophat63]

Issue in 0.11.0 with accessing array of struct using "EXPLODE()->" which works in 0.6.0

1) Setup.
MY_STREAM is an AVRO based stream.
BATCH is constructed as so: BATCH (ARRAY<STRUCT<TIMEBIGINT,BATCHSTRING,MODEINTEGER>>)

2) Steps

2.1) In 0.6.0 works as follows

ksql> SELECT BATCH,
             EXPLODE(BATCH)->TIME AS EXPL_BATCH_TIME
FROM MY_STREAM
EMIT CHANGES;
+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+
|BATCH |EXPL_BATCH_TIME |
+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+
|[{TIME=1598866933, BATCH=START, MODE=0}] |1598866933 |
|[{TIME=1598866933, BATCH=END, MODE=0}] |1598866933 |
|[{TIME=1598879630, BATCH=START, MODE=0}] |1598879630 |
|[{TIME=1598879631, BATCH=END, MODE=0}] |1598879631 |
|[{TIME=1598879888, BATCH=START, MODE=0}] |1598879888 |
|[{TIME=1598879888, BATCH=END, MODE=0}] |1598879888 |
|[{TIME=1598879954, BATCH=START, MODE=0}] |1598879954 |
|[{TIME=1598879955, BATCH=END, MODE=0}] |1598879955 |
|[{TIME=1598880441, BATCH=START, MODE=0}] |1598880441 |
|[{TIME=1598880441, BATCH=END, MODE=0}] |1598880441 |
|[{TIME=1598880453, BATCH=START, MODE=0}] |1598880453 |
|[{TIME=1598880454, BATCH=END, MODE=0}] |1598880454 |
|[{TIME=1598881674, BATCH=START, MODE=0}] |1598881674 |
|[{TIME=1598881674, BATCH=END, MODE=0}] |1598881674

2.2) In 0.11.0, the same query throws an error.

ksql> SELECT BATCH,
            EXPLODE(BATCH)->TIME AS EXPL_BATCH_TIME
FROM MY_STREAM
EMIT CHANGES;
class io.confluent.ksql.execution.expression.tree.DereferenceExpression cannot be cast to class io.confluent.ksql.execution.expression.tree.UnqualifiedColumnReferenceExp (io.confluent.ksql.execution.expression.tree.DereferenceExpression and io.confluent.ksql.execution.expression.tree.UnqualifiedColumnReferenceExp are in unnamed module of loader 'app')

[mingfang]

The section in this article that uses explode is broken due to this error.
https://www.confluent.io/blog/stream-processing-twitter-data-with-ksqldb/

[vcrfxia]

This regression appears to have occurred between ksqlDB 0.9.0 and ksqlDB 0.10.0, so versions 0.10.0, 0.10.1, and 0.11.0 are affected. Thanks for reporting! We'll work on a fix.

[vcrfxia]

In the meantime, the workaround is to split the query into two: one to explode the array, and another to select fields from the struct. The workaround is worse in terms of resources, though, as an extra topic needs to be created, so I believe fixing this issue should be high-priority despite the workaround.

[mingfang]

@vcrfxia do you know when will this fix be released?

[vcrfxia]

@mingfang we track target releases using the "milestone" field of these GitHub issues. Because this one is marked for 0.13.0, that means ksqlDB 0.13.0 is the first release we plan to have the fix in.

The fix will also be in the ksqlDB version included in Confluent Platform 6.0 (to be released shortly).

Do you have a use case that's blocked on this fix? Does the workaround suggested above (of splitting the query into two separate queries) not work for you? Because ksqlDB 0.12.0 has not been released yet it's possible we can squeeze it in there but no promises since it'll be tight. If you could elaborate on your use case and why the workaround isn't acceptable that would be helpful. Thanks.

[mingfang]

@vcrfxia I can use the workaround for now and wait for 0.13.0. Thanks for the update.