Channels: Channels should handle abnormal database server termination gracefully.

[kltdwrds]

Runtime laptop - Mac 10.13.4 (17E199)
Versions

channels==2.0.2
channels-redis==2.1.1
daphne==2.1.0
Django==2.0.4
Twisted==17.9.0

Expectation
Channels would handle an abnormal database server termination gracefully.
Actual

• I use an authentication middleware class that parses a bearer token from the websocket url and adds the user to the request scope:

def get_user(scope):
    token = get_token(scope)
    if token is not None:
        User = get_user_model()
        try:
            payload = jwt_decode_handler(token)
            user_id = payload['user_id']
            return User.objects.get(id=user_id)
        except (User.DoesNotExist, ExpiredSignatureError, KeyError):
            return None

class AuthMiddleware:
    def __init__(self, inner):
        self.inner = inner

    def __call__(self, scope):
        # Add it to the scope if it's not there already
        if "user" not in scope:
            scope["user"] = get_user(scope)
        # Pass control to inner application
        return self.inner(scope)

JWTAuthMiddleware = lambda inner: AuthMiddleware(inner)

Although conn_max_age=0 is set (as per https://github.com/django/channels/issues/871), I've noticed that this specific connection is kept alive and reused on subsequent websocket connections. This works fine until the database server is shutdown unexpectedly (In my case it's a postgres docker container running locally). Future websocket connections result in django.db.utils.InterfaceError: connection already closed

Server command
daphne safechain.asgi:application -t 60 --access-log - --port $PORT --bind 0.0.0.0 -v2
or
runserver command
Logs

2018-04-23 14:59:50,686 - ERROR - ws_protocol - Traceback (most recent call last):
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/daphne/ws_protocol.py", line 76, in onConnect
    "subprotocols": subprotocols,
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/daphne/server.py", line 184, in create_application
    application_instance = self.application(scope=scope)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/channels/staticfiles.py", line 42, in __call__
    return self.application(scope)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/channels/routing.py", line 51, in __call__
    return self.application_mapping[scope["type"]](scope)
  File "/Users/kyle/code/safechain/safechain-api/rest/web_sockets/auth.py", line 43, in __call__
    scope["user"] = get_user(scope)
  File "/Users/kyle/code/safechain/safechain-api/rest/web_sockets/auth.py", line 13, in get_user
    return User.objects.get(id=user_id)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/models/query.py", line 397, in get
    num = len(clone)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/models/query.py", line 254, in __len__
    self._fetch_all()
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/models/query.py", line 1179, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1067, in execute_sql
    cursor.execute(sql, params)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/Users/kyle/.virtualenvs/safechain-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.

potentially related:

• https://stackoverflow.com/questions/31504591/interfaceerror-connection-already-closed-using-django-celery-scrapy
• https://github.com/django/channels/issues/462

[andrewgodwin]

You need to call any database operations inside of database_sync_to_async so the connection is managed properly. See the docs here: http://channels.readthedocs.io/en/latest/topics/databases.html

[kltdwrds]

@andrewgodwin There was no use of database_sync_to_async in the channels custom auth docs. Am I missing something? or could this be an oversight in the docs? Thanks again!

[andrewgodwin]

Ah yes, there is a problem where the database call in __call__ is not correctly handled (it doesn't need database_sync_to_async as it's a sync context, but it needs... something.)

[kltdwrds]

it needs to have it's connection closed 😀 i've spent some time getting familiar with the channels internals, but am empty handed on a solution... curious to hear your thoughts!

[andrewgodwin]

I updated the docs to run the close connection handler and have a note about it!