Kafka-node: Key in KeyedMessage ignored when deciding partition

[michallevin]

From personal testing and looking at the code for baseProducer, it looks like the key in KeyedMessage is not used for determining the partition of each message.

Is this correct? If so, what is the purpose of this key if it is not for determining the partition of a message?

Thanks in advance!

[michallevin]

Can anyone confirm this:

const payload = { topic: 'Test', messages: new KeyedMessage('key_in_message','message1'), key: 'key_in_payload' };

The key_in_payload is used to decide the partition, but this key is ignored afterwards and is not sent on as part of the message.
The key_in_message is ignored when deciding the partition, but IS sent on as part of the message to the Kafka topic.

So essentially both need to be used.

[hyperlink]

I could be wrong on this but I think of the key as metadata about the message. In the kafka world it's optional but it can be used for partitioning-- it's up the implementer of the partitioner.

[krystianity]

Hi @michallevin,
you could achieve deterministic partition production per topic very very easily.
As @hyperlink said it should be up to the producer to deal with this.

Checkout this example to achieve a key-to-partition-determination:

getPartitionForKey(key, partitionCount = 0){

        if(typeof key !== "string"){
            return Promise.reject("key must be a valid string");
        }

        if(partitionCount === 0){
            partitionCount = this.partitionCount;
        }

        return Promise.resolve(murmur(key) % partitionCount);
    }

note that I am using const murmur = require("murmurhash").v3;

[sonihimanshu]

@hyperlink - I faced the same issue, on more analysis I found the key must be a string only but earlier I was using integer value as a key, in that case "hashCode" function was always returning the same hash and due to which Keyed partition was not working for me.

I have created a PR, in which a key can always be converted to string to get the hash code for a given value. please check that out here - https://github.com/SOHU-Co/kafka-node/pull/870

[tcollinsworth]

My testing revealed that the key in keyedMessage is not used.
I had to set the key and partitioner in payload.
However we just discovered that it doesn't appear to be keying the same as murmur2 with partitioner 3.

[tcollinsworth]

Partitioner 0 (default) and partitioner 3 (keyed) are not partitioning the same as Kafka Java.

[pinturic]

I am having the same problem. So should we set the key even in the payload and not only in the keyedMessage ?

[tcollinsworth]

That worked for us.

[pinturic]

for use as well!