Proxysql: Memory leak issues - Proxysql 2.0.13 and other versions.

[rmarshe]

Available memory being track by Zabbix 4.0 server is showing us our Proxysql server nodes decrease at of a rate between 10MB-40MB/24 hours. This seems consistent with Percona and Non-Percona versions of Proxysql and version we have tested range from 2.0.3 up to 2.0.13, but for this test we used the newest 2.0.13 Proxysql build.

Proxysql Version: ProxySQL version 2.0.13-107-g91737e0, codename Truls
OS: Ubuntu 18.04.4 LTS
Steps to Re-produce issue: Anytime Proxysql is running its slowly leaking available memory, this seems to be with a very basic configuration. It doesn't seem to matter if the node is even taking traffic behind our Load-Balancers or not, same issue happens with way and does seem worse either way.

proxysql.log

Install package for Proxysql: proxysql_2.0.13-ubuntu16_amd64.deb ( https://github.com/sysown/proxysql/releases/tag/v2.0.13)

I have some memory statastics from Zabbix monitoring if required around Total, Available, Avg-Min-Max specific to the Proxysql pid if need be.

[renecannao]

I think I reproduced the issue.
Will work on a fix

[rmarshe]

Thanks @renecannao, appreciate all your help on this one :)

[FStelzer]

Hi,
we are seeing similar issues with our proxysql's (2.0.13 as well). Traffic or no traffic it slowly leaks memory it never frees up again.
I think i managed to at least significantly slow it down in our case by disabling some monitoring. We are regularly querying proxysql via the mysql admin api for some values and since i stopped doing that the rate of increasing memory usage has gone down significantly.
I could reproduce it easily by running mysqldump against the admin interface in a loop. (which will only get the main & runtime tables since it's not able to use the fully qualified table names with "disk." afaik).
I was actually trying the mysqldump approach to building a "is everything in sync between disk, main & memory" check.

[rmarshe]

Any luck with progress or finding on this one @renecannao ?

[BurningNetel]

@renecannao I've got some extensive monitoring data for this issue which I've collected for the past few days. Let me known if you need it to solve this issue.

It seems that a certain type of query or use of the proxy triggers this leak. We run around 40 websites, each on their own percona xtradb cluster 5.7 with their own proxysql instance running on each webserver.

The bug only triggers on 2 of the 40 sites, and has been causing memory issues since the release of those sites at least since december 2019. The proxysql configuration is exactly the same on all those webservers. The only thing that is different is the backing sql server hostname.

Things we collect for each site every minute from each webserver:

• select command, total_cnt from stats_mysql_commands_counters;
• select * from stats.stats_memory_metrics;
• show mysql status;

The workaround I've got now is to run 'SELECT 1 FROM stats_mysql_query_digest_reset LIMIT 1' every 5 minutes. This clears the leak and solves the memory issues we've been having on those webservers.

I hope you can find the underlying issue! :)

[vesper1978]

@renecannao Is there any way to get this bumped?

[IzzySmith]

Hi @renecannao,

We are also looking into a memory leak problem that we see in 2.0.14. We created this heap profile, maybe it helps in debugging:

output.pdf

We compiled proxysql 2.0.14 with debug flags and libunwind, and excluded the following from the heap profile:

'je_prof_backtrace',
'prof_alloc_prep',
'calloc',
'imalloc',
'je_malloc_default',
'imalloc_body',
'malloc',
'sqlite3MemMalloc',
'ma_alloc_root',
'mallocWithAlarm',
'sqlite3Malloc',
'unpack_fields',
'pcache1Alloc',
 'sqlite3PageMalloc',

Please let me know if you need more information

[bdelvecchio]

We are seeing some serious memory leaks with 2.0.13 in our production setup, and the rate increased after some recent changes to our infrastructure.

@renecannao do you have any clues as to what type of activity might be related to this leak? Is it related to certain client requests? timeouts? client disconnects? orchestrator?

[m4ce]

any updates on this? We are also seeing leaks on 2.0.15.

[akamac]

Seeing serious leaks on 2.1.0. What data should I provide for troubleshooting?

[denishpatel]

@renecannao We are seeing memory leak issues on 2.1.0 as well. @akamac @BurningNetel Have you guys managed to resolve the issue?

[renecannao]

Spoke with @denishpatel .
It was not a memory leak.
Any knows memory leak is fixed: closing this issue as it is generic.
Any specific leak (present of future) can be followed up on distinct issues.
Thanks

[denishpatel]

For the record, our proxysql was logging millions of distinct queries because mysql-query_digests was enabled. We disabled mysql-query_digests='false' and memory and cpu usage came back to normal. We will investigating our application to figure out why all queries has distinct digest.