Confluent-kafka-go: zstd compression

[artemyarulin]

My topics were created with zstd compression enabled like

kafka-topics --create --zookeeper $ZK --config compression.type=zstd --topic test

and while Java consumers with kafka-client v2.1 read those topics just fine without any additional modification current go client stuck and produces errors like Consumer error: Err-76. I've tried to specify compression.type: zstd but it fails with error panic: Failed to create consumer: Invalid value "zstd" for configuration property "compression.codec"

Source/stderr/stdout/broker.logs https://gist.github.com/artemyarulin/8cd5b38b051132173539c41b158101ba

confluent-kafka-go v0.11.6
librdkafka 722687 - 0.11.6
kafka confluentinc/cp-kafka:5.1.0

• [x] confluent-kafka-go and librdkafka version (LibraryVersion()):
• [x] Apache Kafka broker version:
• [x] Client configuration: ConfigMap{...}
• [x] Operating system:
• [x] Provide client logs (with "debug": ".." as necessary)
• [x] Provide broker log excerpts
• [ ] Critical issue

[edenhill]

librdkafka/confluent-kafka-go v.0.11.6 does not support zstd, it was added for the upcoming v1.0.0 release, but there is an issue in v1.0.0 where it can't consume from topics configured with compression.type=zstd. Fix is in https://github.com/edenhill/librdkafka/pull/2183 which most likely will not make the v1.0.0 release (code freeze), but definitely v1.0.1.

The workaround for now is to revert the topic configuration to compression.type=producer (default), but your v0.11.6 consumer will not be able to decompress the messages anyway due to missing zstd support, unless you upgrade librdkafka to v1.0.0-RC7.

[artemyarulin]

Yeah, thanks, I'll wait no prob.

Just wanted to create this issue so that other people can find it as error message is quite vague and doesn't tell what actually is the reason

[edenhill]

zstd compression is available in the v1.0.0 release.

[senthilec566]

@edenhill , I could still see this error on v1.0.0 release . Can you Pls confirm zstd support added in V1.0.0 ?

[artemyarulin]

There is one limitation mentioned in the latest release of librdkafka

Add support for ZSTD compression (KIP-110, @mvavrusa. Caveat: will not currently 
work with topics configured with compression.type=zstd, instead use 
compression.type=producer, see https://github.com/edenhill/librdkafka/pull/2183)

[senthilec566]

@artemyarulin , Sorry bit confused here. Brokers are already using compression.type=zstd . Can you add bit more details to enable zstd support in librdkafka 1.0.0 ?

[artemyarulin]

Well, the way I've understood it

If you've created topic using something like kafka-topics --create --zookeeper $ZK --config compression.type=zstd --topic $name then latest librdkafka wouldn't be able to read it still.

But if you would use kafka-topics --create --zookeeper $ZK --config compression.type=producer --topic $name and then set in a producer config compression.type=zstd then it should work

[senthilec566]

LibRdKafka Producer is failing with Invalid value "zstd" for configuration property "compression.codec" . I've created new issue here : https://github.com/edenhill/librdkafka/issues/2322