Php-rdkafka: Can I use the chunk function?

[fwang-laralabs]

• PHP version: 7.3
• librdkafka version:
  librdkafka version (runtime) => 0.11.3
  librdkafka version (build) => 0.11.3.255
• php-rdkafka version: how can I know on ubuntu?
• kafka version: kafka_2.12-2.3.0

Can I use the chunk function?
I'm wondering if it's going to be implemented.
If you can use it, I hope you have a friendly example.
My particular message is so big that I need the Chunk feature.

thx.

[nick-zh]

@fwang-laralabs afaik this is not implemented atm, can you maybe quickly elaborate what it does and what you need it for?

[fwang-laralabs]

My specific message size is 1245858 (1.24k).
I need the ability to send messages in pieces.
I'm producing a message but it's not displaying in the consumer console. (Other messages are well consumed.)
It would be convenient to have a chunk-like interface for laravel collections.
(https://laravel.com/docs/5.8/collections#method-chunk)

[fwang-laralabs]

@nick-zh what is atm?

[nick-zh]

atm = at the moment :smiley:

[nick-zh]

@fwang-laralabs ah ok i understand, i might be mistaken, but i think Kafka doesn't have this feature.
So unless i am mistaken, there are mutiple solutions:

• Either allow the message size to be higher (can be configured through message.max.bytes on producer and broker)
• or you chunk it up yourself and add metadata to the headers, so your header could contain a chunk_group_id and a chunk_sequence or something similar to achieve this
  Unless there is a Kafka feature that i have overseen (if so please let me know), i don't think this will be added to the extension, since the need can be covered in the app

[nick-zh]

@fwang-laralabs regarding implementing this in your application, maybe this article helps, it elaborates a bit more how you would do this on the producer / consumer side:
https://medium.com/workday-engineering/large-message-handling-with-kafka-chunking-vs-external-store-33b0fc4ccf14

[fwang-laralabs]

.../opt/kafka/config# vi /opt/kafka/config/server.properties

message.max.bytes=10000000

server.properties 에 수정하고 테스트 해보았습니다.
정상동작하지 않아서, producer 쪽도 수정해 보았습니다.
.../opt/kafka/config# vi /opt/kafka/config/producer.properties
max.request.size=10000000

하지만 여전히 큰 메세지는 동작 하지 않습니다. :(

이에 대한 부분을 찾고 있습니다.

chunking 을 할지, 서버 수정을 통한 큰 메세지 전달이 가능하도록 할지 고민중입니다.

당신의 답변 감사합니다.

[nick-zh]

for the producer you also need to set the setting message.max.bytes (PHP)

[fwang-laralabs]

Can I do this?
$pconf = new Conf();
$pconf->set("message.max.bytes", 10000000 );
$this->producer = new Producer( $pconf );

Let's try it.
Thank you for your kind reply. :)

[nick-zh]

@fwang-laralabs yes, this should work if the broker is set correctly as well :+1:
Let me know if it works

[fwang-laralabs]

Works fine. Thanks to :)