Clickhouse: Kafka Engine failures handling

[arizz96]

Hi to all,
I'm streaming messages from a Kafka cluster to my CH system (version 1.1.54390), but if a malformed message (wrong field type for example) reach the KafkaEngine table, it throws an exception and try to read the message again and again, locking CH ingestion process.

I tried setting these parameters without success:
input_format_allow_errors_num
input_format_allow_errors_ratio
input_format_skip_unknown_fields

How can this be solved?
TY

[Amper]

@arizz96
Hi, I'm facing the same problem.
I'm going to add to Kafka storage engine soon a parameter that allows to specify the topic for messages that could not be processed.

[alexey-milovidov]

The settings input_format_* are for user profiles. For Kafka tables, context with settings is initialized at table creation (at server startup). So, these settings might be applied only if they are set in default profile. Can you please check if they work if you set them in default profile and restart the server?

To solve this issue, we should allow to set them in
https://github.com/yandex/ClickHouse/blob/master/dbms/src/Storages/Kafka/KafkaSettings.h

[luck143]

any update on this ? facing the same issue.
just one single corrupt message(like datatype mismatch) in topic is corrupting whole system and putting everything on complete halt .the only option at this moment seem to be deleting the topic from kafka and recreating the streaming table at kafka .

[abyss7]

First attempt is here #4094

[krasnoperov]

@abyss7 thanks for that attempt.

Now with clickhouse:19.1.6 kafka engine silently drops messages both with kafka_skip_broken_messages = 1 and kafka_skip_broken_messages = 0.

Is there any change to log errors for skipped messages?

[abyss7]

@abyss7 thanks for that attempt.

Now with clickhouse:19.1.6 kafka engine _silently_ drops messages both with kafka_skip_broken_messages = 1 and kafka_skip_broken_messages = 0.

Is there any change to log errors for skipped messages?

Can't confirm your problem at least for 19.5. The kafka_skip_broken_messages=0 forces the CH to repeatedly in an infinite loop try to read the malformed message with a proper parsing error in the logs. While the kafka_skip_broken_messages=1 allows CH to skip the malformed message and mention it in logs:

2019.04.19 15:40:58.177090 [ 32 ] {} <Trace> BlockInputStreamFromRowInputStream: Skipped 1 rows with errors while reading the input stream

If you your problem still persists then please open the new issue with detailed description.

[jywjyw]

Though "kafka_skip_broken_messages=65535" is effective, but if kafka receive a NULL msg, the CH will consume the bad msg repeatly. I found this bug in Ver 20.3.