Kafka-node: Broker side compression issue

[joewood]

I'm seeing a problem using kafka-node on a 0.10.1.1 cluster with server side compression set to LZW. It looks like the consumer payload is the compressed message (i.e. it's not kicking in decompression before dispatching the callback). The same code works fine on a cluster without this setting. Is this a known issue? This is using the ConsumerGroup client. I've tried playing with attributes to force decompression without luck.

[hyperlink]

I'm not familiar with compression and broker side compression. Is there some documentation about it?

[joewood]

Check the docs here: https://kafka.apache.org/documentation/#brokerconfigs. The property is compression.type.

I can confirm, if this value is set to 'producer' it works fine. If it's set to lz4 the message callback receives the message compressed. I suspect the server uses recursive message sets to do this, as per the protocol doc - so this line would have to check for a message set inside the message set and repeat the process.

[dshush]

Had the same issue, so no plans to support lz4 compression in the future?

[grillorafael]

Having the same issue

[jheirtzler]

Also same issue. Does anyone have a fix for this? Do other compression types like Snappy work?

[Shaidar20]

Having this problem as well. Considering the advantages of LZ4 I can only see it becoming more popular for kafka users. If nobody makes a push request, is this something you at least have on your roadmap? Thanks for this amazing repository! We've enjoyed using it.

[emmaccacca]

same problem

[niahmiah]

Yikes, same problem, 1 year later...

[danielb2]

So how are people handling LZ4 compression ? I switched to kafkajs but I'm seeing some other concerns with that lib

[YuyuZha0]

4 years later...

[danielb2]

4 years later...

If it helps, I stayed with kafkajs and ended up not having issues. Don't bother with this lib