Timescaledb: Potential memory leak with long lived connections

[linba708]

While testing timescale i ran in to a potential memory leak. The problem seem to appear when a connection is long lived (the reason for the long lived connection is performance since there is a certain overhead associated with creating new connections).

I wrote a small test project where i simulate the conditions (including the config files used) where i test both postgres and timescale using there respective docker containers.
https://github.com/linba708/test-timescale-memory.

Running that and looking at the memory usage of each connection (or using ctop to look for each container), indicates that timescale consumes alot more memory (and continuous to demand more the more inserts one does).

Whats more is that it doesn't necessarily seem to be related to the use of hyper tables. If you remove the raw in the test code that converts the table to a hyper table the problem still persists.

My current solution is the force kill the connections using a cron job during down hours (the application doing the inserts is written by an external data provider, so i cannot easily change it). So any help or suggestions regarding this would be highly appreciated.

[otbutz]

Are you really sure it's a memory leak and not working as intended? https://stackoverflow.com/a/1216841/9914203

the reason for the long lived connection is performance since there is a certain overhead associated with creating new connections

And that's why you use connection pooling. Either as part of your go program or use something like Pgpool or PgBouncer

Edit: fixed Stackoverflow link

[linba708]

I'm not sure, but i don't think it is related to the shared_buffers, since if i set shared_buffers to something small, like 16MB, the timescale containers open connection still consumes way beyond that even after a few seconds (the resident size (RES) of the open connection grows to about ~150MB after 2.5 million inserts).

The fact that the vanilla postgres container does not at all consume more memory over time is what seems strange to me.

It doesn't seem to matter if the inserts are done to a hyper table or not for timescale, the behavior is the same. But if you don't create the extension inside the timescale container, both containers behave in the same way. Suggesting that the "problem" is neither one with postgres or the containers, but rather the timescale extension itself.

I'm not sure whether or not this is a bug but it definitely seems strange.

I'm aware that i should use connection pooling (in fact that is how the real application is going to be deployed in production). But I have limited experience with Pgpool and Pgbounce so the links are greatly appreciated, i will read up on them.

[erimatnor]

@linba708 Thanks for the bug report and the useful tool.

We just submitted PR https://github.com/timescale/timescaledb/pull/575 to fix this.

Would be great if you could rerun your tests and let us know if it fixes the issue.

[linba708]

@erimatnor Thanks for such a quick fix, the initial test seems very promising indeed. I'm gonna use the PR version in a more realistic envirment and see if it solves the issue there. But so far very promising. Thanks again.

[mfreed]

@linba708 I was wondering if you had the opportunity to put this more through its paces?

[mfreed]

0.10.1 has been released, which includes this fix. Our own internal tests with pg_prometheus also shows this problem having been fixed (https://github.com/timescale/prometheus-postgresql-adapter/issues/19), so we'll close out this bug report in a few days unless hear something.

Thanks, @linba708 !