Go-micro: gRPC client memory leak

[Wintereise]

Hi team,

We have a service ecosystem built using micro (great toolkit btw!), and are simply trying to consume one service from another.

To do this, we use the micro-client which internally uses gRPC.

The problem we're seeing is that the memory usage seems to grow by the amount of service calls we make. Memory is not being freed, and the consuming service eventually crashes.

Attached, you'll find pprof heap dumps visualizing this issue clearly. They are roughly 1-2 second, at a time the service was under a continuous request load of around 250/s (which continued for 30 seconds). We used https://www.speedscope.app/ to visualize

It is worth noting that however, a lot of these requests failed due to autoscale issues in the downstream service (and hence got a 408 or a 500 response.)

Has anyone seen this before? We've opened this issue at the micro repo to make sure it isn't something to do with how we / the framework tries to use gRPC.

heap-dump.zip

[Wintereise]

Here's an extreme example showing the ballooning more effectively.

api-heap.zip

[RichardLindhout]

On which version are you? Are both services on the same version of go micro?

[Wintereise]

Hi @RichardLindhout

We're on v1.10.0, both services indeed do use the same version.

[asim]

Please upgrade to 1.13.2. there was a bug fix for a cache deadlock issue which would have caused the leaking

[Wintereise]

Please upgrade to 1.13.2. there was a bug fix for a cache deadlock issue which would have caused the leaking

We'll try it out and report back. Thank you!

[Wintereise]

Please upgrade to 1.13.2. there was a bug fix for a cache deadlock issue which would have caused the leaking

Yep, that fixed it.

Thank you!