Sensu-go: Memory consumption issue since upgrade to 5.18.0

[Surian]

Expected Behavior

Sensu backend memory consumption should be stable over time if there is no additional hosts added or config change.

Current Behavior

We upgraded our backends from 5.16.1 to 5.18.0 on March 3th. Before that, the memory consumption of the backends was very stable, always around 575Mb for each backend.
Since the upgrade to 5.18.0, backends starts to consume all the available memory (and then crash when all the memory is consumed, obviously).

There was no configuration change before or after the upgrade.

Your Environment

• Sensu version used (sensuctl, sensu-backend, and/or sensu-agent): backend is on 5.18.0 / agents are in various versions
• Installation method (packages, binaries, docker etc.): compiled from OSS
• Operating System and version (e.g. Ubuntu 14.04): Ubuntu 18.04

[echlebek]

We do typically keep an eye out for memory growth during load testing, but it's possible we missed this somehow. We'll need to do some investigation.

[echlebek]

In my load tests on etcd and postgres for the 5.18.1 release, I did not observe the memory growth scenarios you're seeing.

The performance testing we do doesn't utilize all of Sensu's feature set, so that could be one reason why we are seeing different results.

Any further information you could provide, including heap profiles, would be much appreciated.

[Surian]

@echlebek

Here is 2 heap profile taken from 2 of our backends. The backends were started since 4h30~.
The first one (heap.out.gz) is taken from a backend with a normal memory consumption and the second one (heap-issue.out.gz) is taken from a backend with the issue: the memory was increasing periodically since the service start.
When the heap profile has been taken, the memory consumption was around 611Mb for the 2 backends with no issue at this moment and 1241Mb (and still increasing) for the backend that was showing the issue.

heap.out.gz
heap-issue.out.gz

Hope this can help!
I can provide anything else you should need about this issue, just tell me.

[echlebek]

@Surian thank you, this is crucial information. Could you also provide the agent version(s) you are using? Thanks!

[Surian]

@echlebek We have more than 500 agents and ~50% are older than 5.14.0 (I'm sure we still have one or two 5.1.0 hidden somewhere), the other ~50% is equally distributed between 5.14.x or 5.16.1 or 5.18.0.

[Surian]

@echlebek https://github.com/sensu/sensu-go/pull/3723 seems to have fixed the issue, we don't have memory issue since we upgraded the backend!