Gunicorn: Memory increase after service run a long time

[haolujun]

HI :
I use flask + gunicorn to make an api service. Because I want to share a large data between worker processors, so I init the data in gunicorn.conf and use "gunicorn -c gunicorn.conf" to finish it. Any data in gunicorn.conf would be done only once, so every worker processor share a common large data. The big data in my program is ready only, no writing operation. But when I start my service and run a long time, sometime the memory increase 1G suddenly, I guess there may be some writing operation in gunicorn cause the system copy-on-write mechanism. And How do I share the big data between worker processors? I still confused by this problem, expect your reply!!! Thank you!

[tilgovi]

This is related to #1640.

The only other way to handle this would be to do something that avoids the Python memory management. Perhaps the mmap module could help.

[haolujun]

Thank you!!! It's very helpful!

[haolujun]

HI:
With the information you give me, I upgrade my python from 2.7 to 3.7 and use gc.freeze() to avoid some objects collected by garbage-collector. This time, it dosen't appear that "memory increase 1G suddenly", but it casue a memory growthing slowly util each sub-processor memory increased 500M+, then stop growthing. I think gc.freeze() is work and make a different memory-appearance. I hope the work what I have done is helpful for other people. Thanke you！