Aiohttp: compression + encoding should probably happen in executor

[thehesiod]

I just implemented a watchdog utility for our aiohttp servers which tracks when the main thread hangs for more than 5 seconds, and it caught the below callstack. This response returned a large json object with compression enabled. Based on the callstack I think probably compression should be done in a background thread otherwise the web-server can get blocked for several seconds and cause the health checks to fail (note in AWS the default is 5s so if any operation takes more than 5s your server may get recycled).

  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/fbn/fbn.com/api/weather/services/weatherapi/service.py", line 1430 in <module>
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/fbn/fbn.com/api/weather/services/weatherapi/service.py", line 1426 in start_app
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/fbn_async_server/entrypoint.py", line 237 in run_server
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/asyncio/base_events.py", line 422 in run_forever
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/asyncio/base_events.py", line 1434 in _run_once
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/asyncio/events.py", line 145 in _run
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/fbn_async_server/entrypoint.py", line 44 in start
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/aiohttp/web_protocol.py", line 398 in start
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/aiohttp/web_response.py", line 300 in prepare
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/aiohttp/web_response.py", line 605 in _start
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/aiohttp/web_response.py", line 329 in _start
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/aiohttp/web_response.py", line 290 in _start_compression
  | Aug 18 00:08:25.467 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/aiohttp/web_response.py", line 616 in _do_start_compression

probably use run_in_executor?

[thehesiod]

here's another problem:

  | Aug 18 00:08:13.717 |   |   | WeatherAPIInternalService | File "/usr/local/fbn/fbn.com/api/weather/services/weatherapi/service.py", line 734 in get_weather_for_locations
  | Aug 18 00:08:13.717 |   |   | WeatherAPIInternalService | File "/usr/local/fbn/fbn.com/api/weather/python/fbn/asyncio/aiohttp_utils.py", line 25 in json_response
  | Aug 18 00:08:13.717 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/aiohttp/web_response.py", line 632 in json_response
  | Aug 18 00:08:13.717 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/simplejson/__init__.py", line 399 in dumps
  | Aug 18 00:08:13.717 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/simplejson/encoder.py", line 296 in encode
  | Aug 18 00:08:13.717 |   |   | WeatherAPIInternalService | File "/usr/local/lib/python3.6/site-packages/simplejson/encoder.py", line 378 in iterencode

another place with the dumps method can be run in an executor, or perhaps support an async method so caller can run in executor.

[asvetlov]

Thanks for the report.
I expected something like this from the very beginning.

Switching all compression/encoding to thread pool can slowdown simple cases.
As an option, we can use an executor for large data and switch back to direct call for decompressing, say, 1kb buffer.

[thehesiod]

I'm going to code something up, will submit pr since this is high priority for us

[thehesiod]

ok, have a first stab at this, let me know what you guys think