grpc.aio: Unexpected asyncio.CancelledError with KeyboardInterrupt

[rschmied]

What version of gRPC and what language are you using?

grpcio 1.33.2
grpcio-tools 1.33.2

Python

What operating system (Linux, Windows,...) and version?

macOS 10.15.7

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.8.3

What did you do?

Running the async example code, basically async_greeter_server.py which I have modified like here:

if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    try:
        asyncio.run(serve())
    except KeyboardInterrupt:
        logging.info("stopping...")

What did you expect to see?

Clean shutdown of the server.

What did you see instead?

This raises an asyncio.CancelledError exception in AioServer._serving_task_crash_handler which is unexpected:

$ python3 async_greeter_server.py 
INFO:root:Starting server on [::]:50051
^CERROR:asyncio:Exception in callback AioServer._serving_task_crash_handler(<Task cancell... __name__>()>>)
handle: <Handle AioServer._serving_task_crash_handler(<Task cancell... __name__>()>>)>
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/[email protected]/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
    self.run_forever()
  File "/usr/local/Cellar/[email protected]/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
    self._run_once()
  File "/usr/local/Cellar/[email protected]/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/local/Cellar/[email protected]/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/selectors.py", line 558, in select
    kev_list = self._selector.control(None, max_ev, timeout)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi", line 887, in grpc._cython.cygrpc.AioServer._serving_task_crash_handler
asyncio.exceptions.CancelledError
INFO:root:stopping...
$

I think the task was already cancelled, so the task.Exception() raises again... https://docs.python.org/3/library/asyncio-future.html#asyncio.Future.exception

Anything else we should know about your project / environment?

don't think so

[lidizheng]

Thanks for the report. I created https://github.com/grpc/grpc/pull/24718 to stop the spammy log :)

[rschmied]

Just a comment: This should generally help when shutting down / cancelling tasks. Ctrl-C / KeyboardInterrupt is just a specific example where this happens. In my case, I needed a way to safely shut down the server and the unexpected exception was preventing this from working.

[lidizheng]

AsyncIO server has a stop(grace) method, which allows applications to shutdown programmatically. It also provides graceful shutdown option, that it will enter a lameduck mode, rejects new connections but keep existing ones for a bit longer.