Ksql: Support constructing new STRUCTs (i.e., writing nested data)

Created on 13 Nov 2018 · 12Comments · Source: confluentinc/ksql

TL;DR: The goal is to support creating new STRUCTs in your queries.

Background

KSQL already supports reading nested data (JSON, Avro) via the STRUCT type:

Accessing (reading) existing nested data in your queries. For example, you can access a nested field via the myStruct->myField syntax.
"Limited write" support: you can copy nested data -- i.e., a STRUCT -- as-is (unmodified) from an input stream/table to an output/stream table.

However, KSQL does not yet support:

Creating a new STRUCT from the input stream/table (i.e., in a CSAS/CTAS statement).

Example

-- We start with a non-nested stream.
-- The goal is to turn the flat street and zip fields into a nested address STRUCT.
CREATE STREAM orders (item VARCHAR, quantity INTEGER, street VARCHAR, zip INTEGER) WITH (...);

-- The following is not yet supported!
-- Note: The syntax below is for illustrative purposes and does not preclude
-- the design discussion we need to have.
CREATE STREAM nested_orders AS
  SELECT
    item, quantity,
    STRUCT(street, zip) AS address1,
    STRUCT<renamedStreet VARCHAR, renamedZip INTEGER>(street, zip) AS address2,
    STRUCT<VARCHAR, LONG>(street, zip) AS address3
  FROM orders;

data-accessibility enhancement

Source