Sarama: Consider allowing callers to return Message objects for reuse

[shanson7]

Versions

Sarama Version: 1.16.0
Kafka Version: 1.1.0
Go Version: 1.9.2

Problem Description

Not so much a problem, but sarama is the source of most of the memory allocations in our project. This is not too surprising, as we are processing ~30k (small) messages per second.

Here is the top alloc_space after about 4 hours of running.

  227.34GB 22.59% 22.59%   332.51GB 33.04%  github.com/grafana/metrictank/vendor/github.com/eapache/go-xerial-snappy.Decode /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/eapache/go-xerial-snappy/snappy.go
  176.73GB 17.56% 40.16%   176.73GB 17.56%  github.com/grafana/metrictank/vendor/github.com/Shopify/sarama.(*partitionConsumer).parseMessages /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/Shopify/sarama/consumer.go
  134.32GB 13.35% 53.51%   585.07GB 58.14%  github.com/grafana/metrictank/vendor/github.com/Shopify/sarama.(*MessageBlock).decode /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/Shopify/sarama/message_set.go
  105.17GB 10.45% 63.96%   105.17GB 10.45%  github.com/grafana/metrictank/vendor/github.com/golang/snappy.Decode /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/golang/snappy/decode.go
   90.32GB  8.98% 72.93%    90.68GB  9.01%  github.com/grafana/metrictank/vendor/github.com/Shopify/sarama.(*Broker).responseReceiver /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/Shopify/sarama/broker.go
   63.59GB  6.32% 79.25%    63.59GB  6.32%  github.com/grafana/metrictank/vendor/github.com/Shopify/sarama.(*realDecoder).peek /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/Shopify/sarama/real_decoder.go
   38.38GB  3.81% 83.07%   586.37GB 58.27%  github.com/grafana/metrictank/vendor/github.com/Shopify/sarama.(*MessageSet).decode /home/jenkins/workspace/metrictank-build/gopath/src/github.com/grafana/metrictank/vendor/github.com/Shopify/sarama/message_set.go

I believe that with some small tweaks to allow buffer reuse (by allowing consumer clients to return a Message when they are done) a lot of these allocations can be removed.

If go-xerial-snappy also allowed pre-alloc'd buffers to be passed in, sarama could reuse the buffers there as well.

[eapache]

That's a great idea and makes a lot of sense; I'd be happy to take PRs to this effect.

[shanson7]

Excellent. So, I staged and tested such a change using a sarama global channel-based leaky buffer. It was pretty effective, but I was hindered by the fact that a large amount of the memory is actually used by the decoded messageSet/Block and each message actually refers to that larger underlying memory. Bookkeeping around when it is safe to reuse that larger chunk of memory would likely be overkill for this feature. Simply reusing the *ConsumerMessage did reduce allocations by about 10%. I'm not sure overall if it's worth it though, without being able to reclaim the actual []byte blocks for Key/Value.

[kjelle]

Sarama generates high GC pressure without reusing objects and buffers. As @shanson7 describes, perhaps it would be better to reuse the entire messageSet buffer, along side reusing ConsumerMessage, turning to go-xerial-snappy.DecodeInto etc.

[probot-shopify[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the master branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.