Clickhouse: Kafka headers

Created on 24 Feb 2020 · 4Comments · Source: ClickHouse/ClickHouse

Hi! We intend to use the Kafka Engine (https://clickhouse.tech/docs/ru/operations/table_engines/kafka/) for storing events in Clickhouse.
Also, some of the necessary information is stored in the Kafka's headers (https://issues.apache.org/jira/browse/KAFKA-4208) and we would like to store them too.
Is there a way to achive this ?

comp-kafka feature

Source

MikeZaharov

👍5

Most helpful comment

Right now it's not supported.

We can add support for that in future version in pair of virtual columns like

_header_names Array(String),
_header_values Array(String)

Would such a solution satisfy your needs?

filimonov on 24 Feb 2020

👍7

All 4 comments

Right now it's not supported.

We can add support for that in future version in pair of virtual columns like

_header_names Array(String),
_header_values Array(String)

Would such a solution satisfy your needs?

filimonov on 24 Feb 2020

👍7

We are willing to have this feature too. Would be very helpful.

nickshatilo on 24 Feb 2020

It's ok for us to use 2 arrays if this approach guarantees the order.
I mean we'll be able to get header's value somehow like this:
arrayElement(header_values, indexOf(header_names, 'header_name')) as header_name

MikeZaharov on 27 Feb 2020

@filimonov It would be great to have it indeed, thanks a lot!