Channels: Performance drop when using Channels 2 with WhitenoiseMiddleware

[OndrejSkalicka]

I've just upgraded my Django project's Channels from 1.x to 2.x (there seems to be an issue with 1.x that it creates new session for every connection), replacing my wsgi application with an asgi one + all other changes required by docs.

After that, my response time for pure-html pages dropped from ~20ms to about 400ms. (I've tested on a real simple one, next to no db involved, no javascript evaluated)

I've fiddled with settings and found out that when I disable whitenoise.middleware.WhiteNoiseMiddleware , response times are back to normal. (both whitenoise==4.0 and whitenoise==3.3.1)

In Channels 1.x, inclusion / exclusion did not seem to make an impact (not measurable anyways).

I run the project using simple python manage.py runserver, but using daphne produces same results.

When I tried to add some logging to the middleware itself, it seems like the __call__ itself is not performing bad (around 10ms), but the performance drop is roughly dependent on number of static files.

I'm still in process of stripping my project to some bare version that I'm able to send (it's rather big now).

[andrewgodwin]

I'm afraid that just saying there's a performance drop isn't something we can really act on - we're going to need concrete "this should happen and doesn't" steps and/or your help to identify what's actually wrong.

I'm going to close this right now as it's not a directly solveable issue (http://channels.readthedocs.io/en/latest/support.html) - feel free to comment though, and re-open if you can find something we need to fix.

[OndrejSkalicka]

@andrewgodwin yeah, I understood, it was my last shot to try to get it working somehow. Will try my luck on 1.x to create a workaround around @channel_session_user_from_http creating a new db session for every WS connect ( = click on webpage).

[majgaard]

@andrewgodwin I've experienced the exact same issue @OndrejSkalicka is referring to. I forked whitenoise and added some debugging output to see what was going on. From what I can see, the WhitenoiseMiddleware.__init__() function is being called on each request. Thus, all staticfiles are being re-added to the {url:staticfile} dict.

I'm fairly sure we shouldn't be initializing middleware on each request, as this conflicts with the Django documentation which states that __init__ is only called once (on server start). @OndrejSkalicka, if you've got the time, could you do something similar and confirm that your middleware is also being initialized on each request?

[andrewgodwin]

Hmm, ASGI middleware is loaded every request (#1099) so it's quite possible I made the same mistake with the Django middleware as the handler is loaded every request too. If so, someone should be able to patch that quite easily.

[Satloff]

@andrewgodwin This issue should probably be reopened

[OndrejSkalicka]

@majgaard Hi, unfortunately I do not have the branch at the ready now to test, but it did behave according to what you say -- the the response rate was slowed down proportionately to number of static resources. Eg. if whitenoise check those on __init__ it is most likely the cause.

[majgaard]

I have posted about this on the django-developers Google group, if anybody would like to pitch in about fixing this. https://groups.google.com/forum/#!topic/django-developers/AgZFv5y8nq4

[andrewgodwin]

I'm going to reopen this now we have enough concrete information about the problem. I won't be able to work on a fix for at least a few weeks, however, so if someone else wants to tackle it feel free.

[majgaard]

I did the following to mitigate the issue:
https://github.com/majgaard/channels/commit/608fe5ff7931aacf2e83ca7dc89d139e8d2a16d0

However, this causes the AsgiHandler to not conform with the tests in asgiref... https://github.com/django/asgiref/blob/master/asgiref/testing.py#L17

It also feels like a bit of a hack... If anyone has thoughts, please let me know.

[andrewgodwin]

Yeah, that's not the right approach - instead, the BaseHandler's load_middleware method needs overloading so that it caches the loaded middleware on the class rather than the instance.