Beats: Memory leak when activating docker metrics

[tobiaskraft]

When activating the docker metrics in metricbeat the Memory and also the CPU usage is continuously increasing. After two days the metricbeat process has consumed 20 % of the available memory (12 GB).
After deactivating the docker metrics the problem disappeared.

For more information and a diagram on the memory usage see also:
https://discuss.elastic.co/t/cpu-memory-consumption-after-upgrading-to-version-5-1-2/72735

• Version: Metricbeat 5.1.2
• Operating System: Ubuntu 16.04.1 LTS
• Steps to Reproduce: Activate docker metrics in the file metricbeat.yml

[aanc]

Same issue here on a Fedora 24 server (Metricbeat 5.1.2 too). Memory consumption after 3 days is 70% (of 16GB), and CPU usage 10%.
Docker module is activated for 10 containers, every 10 sec.

I'm deactivating docker module.

[ruflin]

@aanc @tobiaskraft Thanks for the report. We will investigate this further.

[ruflin]

I potentially found the problem here: https://github.com/elastic/beats/pull/3492

[ruflin]

@aanc @tobiaskraft Could one of you test a nightly build to see if this fixes the bug? https://beats-nightlies.s3.amazonaws.com/index.html?prefix=metricbeat/ You probably need to wait 1-2 more hours until a build is available with it as the PR just got merged.

[tobiaskraft]

Hi Nicolas,

the build is now installed on our system. I will provide feedback in 2 or 3 days on the memory and CPU consumption.

[ruflin]

@tobiaskraft Thanks for testing. I'm hoping this gets it fixed.

[aanc]

It's better now on my side.

Maybe we should wait a few more days to be sure, but seems promising.
Thanks !

[ruflin]

Great to hear. That is already quite a big difference. I will backport the fix to 5.2 https://github.com/elastic/beats/pull/3514

[tobiaskraft]

On my server it is now also fine.
Thanks.

[ruflin]

@tobiaskraft Good to hear. Please ping us in case you something going "wrong" again.

[strootman]

I'm experiencing this leak as well, but it's only affecting a subset of our machines.
Most of our machines are CentOS 7, all supporting Docker, and all have metricbeat running with the docker module.

Only a subset of these machines have the suggested production direct-lvm settings.

The machines with the production settings are not experiencing the memory leak, while the hosts with default settings are.

It may be worthwhile to know whether or not bugs like this are the result of running docker without the suggested production settings.

However, in addition to the previously mentioned production settings, these machines also do not have as many long-running docker containers as the other hosts.

[ruflin]

@strootman Do you still get the memory leak with the latest builds? The memory leak should not be related to the above config. Perhaps you found a new issue we were not aware of yet?

[strootman]

@ruflin Haven't updated yet, I'll report back when I do.

[strootman]

@ruflin I'll keep an eye on it, but everything is looking good :+1: