Aiohttp: Cannot write to closing transport

[ajay1mg]

🐞 Describe the bug
Every few minutes we get Cannot write to closing transport exception on our gateway service which is a part of our microservices infrastructure.

Gateway service calls all other services with a timeout of 10 seconds. The other service that gateway calls are also based on aiohttp framework. One such service which is giving most of the exceptions doesn't raise any on its own server.

We are on ubuntu 16.04 LTS

💡 Expected behavior
Our gateway service should be able to handle the timeouts.

We are getting the following exception on our server every few minutes:
📋 Logs/tracebacks

```python-traceback (paste your traceback in the next line)
Traceback (most recent call last):
File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/client.py", line 502, in _request
resp = await req.send(conn)
File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 629, in send
await writer.write_headers(status_line, self.headers)
File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/http_writer.py", line 112, in write_headers
self._write(buf)
File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/http_writer.py", line 67, in _write
raise ConnectionResetError('Cannot write to closing transport')
ConnectionResetError: Cannot write to closing transport

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/hydra/base_api_request.py", line 75, in request
timeout=cls.request_timeout(timeout))
File "", line 5, in wrapper
File "/home/ubuntu/.pyenv/versions/3.7.2/envs/gateway/lib/python3.7/site-packages/newrelic/hooks/framework_aiohttp.py", line 260, in _coro
response = yield from wrapped(args, *kwargs)
File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/aiohttp/client.py", line 514, in _request
raise ClientOSError(*exc.args) from exc
aiohttp.client_exceptions.ClientOSError: Cannot write to closing transport

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/hydra/decorators/http.py", line 41, in f
result = await wait_for(shield(wrapped_func(self, args, *kwargs)), api_timeout)
File "/home/ubuntu/.pyenv/versions/3.7.2/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
return fut.result()
File "/home/ubuntu/1mg/API-Gateway/gateway/utils.py", line 189, in wrapper
result = await func(args, *kwargs)
File "/home/ubuntu/1mg/API-Gateway/gateway/managers/app_context_manager.py", line 98, in _build_shared_context
result = await func(args, *kwargs)
File "/home/ubuntu/1mg/API-Gateway/gateway/handlers/mingler/service_handler.py", line 14, in services
response = await Services.get_services(params)
File "/home/ubuntu/1mg/API-Gateway/gateway/service_clients/mingler/services.py", line 13, in get_services
result = await cls.post(path, data=params, headers=headers)
File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/hydra/base_api_request.py", line 144, in post
response_headers_list=response_headers_list)
File "/home/ubuntu/.pyenv/versions/gateway/lib/python3.7/site-packages/hydra/base_api_request.py", line 93, in request
raise HTTPRequestException(error={'message': exception_message})
hydra.exceptions.HTTPRequestException

📋 **Your version of the Python**
<!-- Attach your version of the Python. -->
```console
3.7.2

📋 Your version of the ubuntu/aiohttp/yarl/multidict distributions

aiohttp==3.6.2

```console
multidict==4.5.2

```console
yarl==1.3.0

📋 Additional context

This issue is already tracked in a number of threads on issue tracker on GitHub like:
https://github.com/ITISFoundation/osparc-simcore/issues/1100
https://github.com/aio-libs/aiohttp/issues/3648

[arnoldfranz]

Thank you Dahuage for your fix, I will be happy to see it applied on a next release.

[Arnaud-Francois-Fausse]

Hi, Is it possible to by-pass this issue by a slight change in my application code ? Is it linked to the way I manage the session ?
Can anybody explain the root cause ?
Thanks
AF

[WoodyFleurant]

@Arnaud-Francois-Fausse From my understanding some HTTP requests that are closing (because eventually request failed ?) are reused for other requests, but as the Transport is 'Closing' it raise that Exception. @arnoldfranz made a fix to "reuse" only requests that are not using 'Closing' transport. On my end I tested using HTTP/1.0 with a global session and I did not faced that exception, but HTTP Frame were not reused.

[Dahuage]

Hi, Is it possible to by-pass this issue by a slight change in my application code ? Is it linked to the way I manage the session ?
Can anybody explain the root cause ?
Thanks
AF

Hope the explaination helps!

[asvetlov]

Fixed by mentioned PRs

[danielfaust]

I just wanted to leave a note saying that when I have a WebSocket connection open from a browser to an aiohttp server, and then kill that browser via the Task Manager (Windows), then the same error (ConnectionResetError: Cannot write to closing transport) will get raised.

Every following attempt to write to that WebSocketResponse object will raise the same error. When will it effectively close?

[asvetlov]

If a peer (a browser in your case) closes WebSocket -- the server cannot send data through it.
That's why any writing to the closed socket raises an exception.
The exception is caught by aiohttp server internals, a message about the unhandled exception is logged.
You can just ignore such records in logs but if you want tidy-up the code you can catch the exception explicitly and immediately exit from a web-handler.