Channels: memory leak for simple http requests

[djangojack]

I started testing my django app with channels and I'm running into memory issues for normal requests.

To demonstrate the issue, I deployed this app to heroku:
https://github.com/jacobian/channels-example

And then wrote a simple script to constantly request the home page:

import urllib2

def get():
    url = "https://channels-demo.herokuapp.com"
    response = urllib2.urlopen(url)
    html = response.read()

for i in range(0,30000):
    get()
    if (i%10 == 0):
        print i 

From heroku's memory metrics, you can see that the memory is slowly creeping up. For this simple example, the memory went from 35 to 90 MB in 90 minutes.

I am experiencing this on my own app using channels version 0.14.0.

[andrewgodwin]

Given that I just fixed https://github.com/andrewgodwin/channels/issues/182, which would cause a resource leak in theory, could you try against latest master and see if it still persists?

I'll see if I can try to reproduce it locally as well, but testing memory leaks is tough in Python, as sometimes is grows over time to a natural limit then stops too (and I'm a PyCon this week so can't do it immediately)

[djangojack]

Just tested with the fix for #182, and unfortunately there is still a memory leak.

For my app it increases to > 512 MB, crashes the server, and then repeats, so it never reaches a natural limit.

[unloder]

I am experiencing the same problem. The memory usage slowly grows to 700+ mb, on a not so active server. The memory usage was a lot smaller before i switched to channels and was a lot more dynamic., going up with load in comparison to slowly buiding up to its limit.

[gipi]

I'm not an expert but probably is not totally a fault of channels: every process that wants to allocate memory in the heap cannot return back to the underlying OS also in the case a free() call is done because of the way the memory allocation works.

This is why (I think) several WSGI implementation have options used to respawn the process in order
to decrease the memory usage (uwsgi has max-requests, mod-wsgi has maximum-requests).

I think runworker should implement an option like that, I could try to write something if I find some time at work since otherwise I'm stuck with runworker killed with an OOM after three requests that upload large images; the same happened in the same project with mod-wsgi without maximum-requests.

I don't know if this option should be implemented for runworker or daphne or both.

[andrewgodwin]

That is one problem, but it's also possible there's a slow leak in there in the first place which aggravates the problem. Python worker process will slowly grow up to the size required to handle the largest request, but the Python VM (from my understanding) will re-use free memory it has, so it should not go beyond there.

[d2rk]

Any update one this one? Experiencing the very same problem.

[andrewgodwin]

Not yet, I haven't managed to get a setup where I can isolate daphne's memory and reproduce it. Looking into it today after some IPC improvements though.

[andrewgodwin]

Initial investigation reveals that Daphne memory usage grows by about a megabyte every 2000 requests on my local machine, and continues to go up. runworker does not leak. It can be reproduced using ab against the default Django testproject.

[andrewgodwin]

Further investigation reveals the issue happens with a completely plain HTTPRequest, protocol and factory, pointing to a problem either in logging or Twisted upstream.

[andrewgodwin]

Verified as a leak in twisted.web only under Python 3. Will forward upstream.

[andrewgodwin]

Further further investigation revealed https://twistedmatrix.com/trac/ticket/8164, which shows that this occurs if you do not initialise Twisted's global log; it builds up in memory up to around 400MB. https://github.com/andrewgodwin/daphne/commit/da40761b95118da82306974e12843566a90ae433 fixes this issue.

[djangojack]

@andrewgodwin
I am running Python 2.7 and updated channels (0.17.1) and daphne (0.14.3) , and I am still experiencing memory leaks.

[andrewgodwin]

@djangojack Given that I can no longer reproduce it, I'm going to need a lot more to help debug it :( Can you reproduce it locally?

[djangojack]

@andrewgodwin No, I haven't attempted any memory profiling locally. Just been deploying to Heroku and using their memory metrics.

[andrewgodwin]

Unfortunately, unless I can get a live ipdb directly into a running worker on Heroku, I have to have steps to replicate before I can do anything. The memory leak I found on both 2 and 3 has been fixed, and on my machine it now holds steady even under 10,000 requests from ab, so I'll have to say there's nothing more I can do without steps to replicate or some considerable spare time.

[andrewgodwin]

I would just like to verify though - the memory usage is increasing on the Daphne node, not on the worker node, in heroku, right?

[djangojack]

yep, it is the daphne node.

[jon-torodash]

Has there been any further work done on this issue? I've noticed that Daphne even has problems serving static files when put under strain, if the requests are randomly interrupted (that isn't to say I'm looking to serve static files, but it does help to isolate the problem away from Django request handling). The problem only seems to occur if the requests are interrupted in some way. I realize this may be due to an underlying problem in Twisted, but the real volume I've seen in objgraph are core python primitives: dict, tuple, etc... Not very illuminating.

[andrewgodwin]

I know of no further work. Have you tried an alternative ASGI server other than Daphne? There are some here: https://asgi.readthedocs.io/en/latest/implementations.html