gunicorn does not free memory on a complex requests

[semeyon]

I have a long-running request that consumes a few Gb of RAM(which is expected). But after it finishes successfully gunicorn does not free memory. The app uses a lot of C++ python bindings. So my question is this behaviour is expected? Do gunicorn workers keeping allocated memory until it will be reloaded? Thanks!

[tilgovi]

Gunicorn should not keep allocated memory, but when memory gets freed is implementation dependent and up to the runtime. Python may keep its own heap of values for re-allocation and not release them to the OS.

You could try calling gc.collect() to see if there is garbage that can be freed.

The thing to look for is whether memory grows with _every_ request or only with the first request.

If memory grows with every request, there could be a memory leak either with Gunicorn or your application. A last resort is to use the max_requests configuration to auto-restart workers.

If you can reproduce a leak in Gunicorn please provide any information you can for us to investigate.

[semeyon]

@tilgovi Thanks for the help!
It grows on the first request and keeps at maximum. And I'm using the max_requests already. Just wanted to know who is responsible for such behaviour.

Should I call gc.collect() in my python code(at the end of logic) or at the end request gunicorn callback?

[benoitc]

gunicorn itself doesn’t use much ram and doesn’t buffer.

It may be your application leaking too much ram (c++ code or anything keeping memory in global objects) or the python vm that doesn’t release the ram for another reason and in that case the gc.collect as suggested may help. You should try to instrument and trace your application to find where the leak happen :)

[semeyon]

@benoitc Thanks for the answer.
What instruments you'd recommend to trace memory leaks in this case?

[benoitc]

be aware that gc.collect is synchronous so it can only be a temporary solution. At the end or beginning of a request is not different. I would do it at an interval though.

[benoitc]

@semeyon i generally use opencensus these days. and any python vm stats reporter (like the one for prometheus)

[benoitc]

i forgot but for for c++ code leak detection you should probably use valgrind

[semeyon]

@benoitc Thanks for the advice. I'll try to make a tracing between other tasks.

[tilgovi]

You may not want to force Python to free the memory. Often Python will keep memory allocated from the OS heap so that it can allocate new Python objects without allocating additional memory from the OS. This improves performance when many small objects are created and destroyed.

[benoitc]

closing the issue as answered :)

[semeyon]

@benoitc Yeah. And thanks to everyone for the help!

[cinjon]

@semeyon what was most helpful here? facing the same issue.

[semeyon]

@cinjon Well, kind of. I found that on of a dev's on our team added heavy computation/memory logic right on the flask requests - I've moved it to an async thing.

[cinjon]

How did you find that? Which tools did you use?

On Tue, Aug 20, 2019 at 23:14 Semeyon [e-mail] wrote:

@cinjon https://github.com/cinjon Well, kind of. I found that on of a
dev's on our team added heavy computation/memory logic right on the flask
requests - I've moved it to an async thing.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/benoitc/gunicorn/issues/1981?email_source=notifications&email_token=AAEWHN5QCOFFO7SNVDEO5PTQFSXJJA5CNFSM4GUV7VOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4YJKYY#issuecomment-523277667,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEWHN3AZ6U4CFZYIVXULPDQFSXJJANCNFSM4GUV7VOA
.

[semeyon]

@cinjon Nothing fancy: trace, timeit profiler etc. Looked through the code carefully.

[mcauto]

You may not want to force Python to free the memory. Often Python will keep memory allocated from the OS heap so that it can allocate new Python objects without allocating additional memory from the OS. This improves performance when many small objects are created and destroyed.

@tilgovi Could you add the reference?

[tilgovi]

Here's a place to read about Python memory management: https://realpython.com/python-memory-management/#cpythons-memory-management

Python, like many interpreted languages, allocates memory from the OS in chunks and re-uses it as a pool for allocating and freeing interpreter objects.

[mcauto]

Here's a place to read about Python memory management: https://realpython.com/python-memory-management/#cpythons-memory-management

Python, like many interpreted languages, allocates memory from the OS in chunks and re-uses it as a pool for allocating and freeing interpreter objects.

Thank you ! 👍

[alexeyegorov]

For me, I find this solution pretty nice. Setting the max requests per worker you can force a restart and hence freeing up memory after N requests.
UPD: I see somebody already mentioned this, but for those who didn't see it... :P