Librdkafka: How do i use parse.key and key.separator properties with RDKafka

[QueryCity]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

How to reproduce

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• [x] librdkafka version (release number or git tag): <REPLACE with e.g., v0.10.5 or a git sha. NOT "latest" or "current">
• [ ] Apache Kafka version: <REPLACE with e.g., 0.10.2.3>
• [ ] librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
• [ ] Operating system: <REPLACE with e.g., Centos 5 (x64)>
• [ ] Provide logs (with debug=.. as necessary) from librdkafka
• [ ] Provide broker log excerpts
• [ ] Critical issue

[QueryCity]

with a console consumer its very easy to set, how can we do the same when using rdkafka client

[rnpridgeon]

I'm not quite sure exactly what you are trying to accomplish here based solely on your comment. But if you are looking for a librdkafka equivalent of the java console consumer you should try out kafkacat.

https://github.com/edenhill/kafkacat

[QueryCity]

If the broker is configured to choose a partition based on the message supplied key.
Then in the message we must supply that key other wise broker rejects the message.
console producer supports supplying the property "parse.key" and "key.separator" .
./kafka-console-producer.sh --broker-list --topic mytopic --property "parse.key=true" --property "key.separator=:"

My Question here is how can i set these two properties when i am not using console producer and i am using RDkafka based c++ client.

I checked in the supported configuration in rdkakfa https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

It does not support any of the config property "parse.key" and "key.separator" .
So how will i set these property in my c++ program to make the same effect as passing
--property "parse.key=true" --property "key.separator=:" to the console producer.

Thanks

[rnpridgeon]

Thanks for clarifying,

Kafkacat, linked previously, includes a flag, -K which you can use to configure your key value delimiter.

https://github.com/edenhill/kafkacat/blob/master/kafkacat.c#L959

[rolandyoung]

You pass the key and the message payload as separate strings to the rdkafka API. (The RdKafka::Producer::produce method has a const std::string * key parameter.)

If you are writing your own C++ client, you will need to write a method to parse the key from the message. If you are using kafkacat, see above. Otherwise, you need to refer to the author of the C++ client you are using.

[rnpridgeon]

It's worth noting these are not client properties for either language. The key.separator and parse.key flags are specific to the console producer(Scala) just as -k is specific to kafkacat(librdkafka derived). You will need to write your own parser for either language if you wish to incorporate the logic into your own code.

[QueryCity]

Thanks a lot rnpridgeon and rolandyoung for your help.

I tried the minimal change, calling "key" supported method in my client of RdKafka::Producer::produce
and it works !!!

Thanks