Sarama: Support Consumer Rack Awareness

[wmorgan6796]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

| Sarama | Kafka | Go |
|--------|-------|----|
| 1.25.0 | 2.4.0 | 1.13.5 |

Problem Description

According to KIP-392: Allow consumers to fetch from closest replica you can now support Rack Aware consumers to allow the consumers to consume from replicas that are in the same rack as you instead of just consuming from the leader of the partition. It would be great if sarama supported this as it would cut down on Cross AZ traffic saving us a substantial amount of money.

[dnwe]

@wmorgan6796 I did start looking into this and implemented the first part (protocol support)

[dneralla]

@dnwe Any update for the latter part on actually using the preferred replica returned ?

[salsorrentino]

My understanding is all the protocol support is about all you need. By sending client.rack, I believe the consumer will be sent to closest replica by Kafka. You should not have to modify the BalanceStrategies too much, unless you want to omit topics without a replica/leader partition in the clients rack. I was trying to implement my own, but then realized the BalanceStrategyPlan did not have the support that was needed. I was able to the rack data to my balancer using the UserData field of the client config, then hit my roadblock.

[dneralla]

@salsorrentino I think you they need to add support on consumer side.
I just took a stab and sent a pull request https://github.com/Shopify/sarama/pull/1696

@dnwe can you tal ?

[salsorrentino]

@dneralla , @dnwe you are gentlemen and a scholars

Waiting with bated breath for this merge. I have a project that requires zonal isolation. Right now I'm using the confluent driver as it's the only "Go" driver I could find that supports KIP-392 right now. I'd really like to avoid a dependence on a CLib, and In my testing, the Sarama consumers had about 50% more throughput.

[dneralla]

@salsorrentino got busy with other stuff...will push this by eod tomm

[salsorrentino]

Understood.