Fastapi: [QUESTION] aiohttp throws RuntimeError

[abdusco]

Description

I'm trying to build an async web scraper using aiohttp library. I have an aiohttp.ClientSession instance in a module that I import and share among various scrapers. A typical scraper is like this:

# app/http.py
import aiohttp
session = aiohttp.ClientSession()


# app/services.py
from app.http import session

async def translate(phrase: str) -> typing.List[str]:
    # ...
    res = await session.get(...)
    html = await res.text()
    # ...
    return translations

My path functions look like this:

# app/__init__.py

@app.get('/api/translation', response_model=typing.List[str])
async def get_translations(
        phrase=Query(...),
):
    return await translate(phrase)

When I send a GET request to /api/translation?phrase=bewegen, I get the following error:

# ...
INFO: ('127.0.0.1', 1549) - "GET /api/translation HTTP/1.1" 500
ERROR: Exception in ASGI application

  # ...

  File "D:\Development\python\langu\venv\lib\site-packages\aiohttp\helpers.py", line 568, in __enter__
    raise RuntimeError('Timeout context manager should be used '
RuntimeError: Timeout context manager should be used inside a task

On the other hand, if I manually run the scraper, it runs just fine:

if __name__ == '__main__':
    async def main():
        return await translate('komm schon')

    results = asyncio.run(main())
    print(results)

Creating a new session for each request also works:

@app.get('/test')
async def test():
    session = aiohttp.ClientSession()
    async with session:
        res = await session.get('https://httpbin.org/get')
    return await res.json()

According to this page:

• https://stackoverflow.com/questions/52232177/runtimeerror-timeout-context-manager-should-be-used-inside-a-task

I need to share the same loop between aiohttp.ClientSession and FastAPI/uvicorn instances, but I'm not sure if that's the best solution or if it's possible at all.

Additional context
Python version: Python 3.7.3

Installed packages

aiodns==2.0.0
aiofiles==0.4.0
aiohttp==3.5.4
async-timeout==3.0.1
attrs==19.1.0
beautifulsoup4==4.7.1
cffi==1.12.3
chardet==3.0.4
Click==7.0
fastapi==0.29.0
h11==0.8.1
idna==2.8
multidict==4.5.2
pycares==3.0.0
pycparser==2.19
pydantic==0.26
soupsieve==1.9.1
starlette==0.12.0
uvicorn==0.7.2
websockets==7.0
yarl==1.3.0

[romeo1m]

I think the issue is that you instantiate your session outside of an async def function. And when you instantiate it no loop is running.

If you want to keep the session at module level you can have something like this (I didn't test it):

```

app/http.py

import aiohttp

_session = None

async def get_session():
global _session
if _session is None:
_session = aiohttp.ClientSession()
return _session

app/services.py

from app.http import get_session

async def translate(phrase: str) -> typing.List[str]:
# ...
session = await get_session()
res = await session.get(...)
html = await res.text()
# ...
return translations
```

[abdusco]

@romeo1m that's exactly how I solved the issue, thanks a lot :)

[tiangolo]

Great! Thanks @romeo1m for the help here!

You might also want to initialize the session at startup: https://fastapi.tiangolo.com/tutorial/events/

As I see you solved your problem, I'll close this issue now. But feel free to add more comments or open new issues.

[zzztimbo]

@tiangolo @abdusco

I'm trying to implement the code suggested by @romeo1m.

 loop = asyncio.get_event_loop()
 loop.run_until_complete(foo())

gives me the following error: RuntimeError: There is no current event loop in thread 'ThreadPoolExecutor-0_0'.

asyncio.run(foo())

succeeds on the first call, but on subsequent calls gives me the follow error: RuntimeError: Event loop is closed

can someone point me to some code that shows how to implement a shared aiohttp.ClientSession() in fastapi?

[tiangolo]

@zzztimbo it's not a direct answer to your question, but it might be worth checking httpx: https://github.com/encode/httpx - https://www.encode.io/httpx/.

As httpx is made by the same Tom Christie (creator of Starlette) and team, it might be easier to debug compatibility issues. It also has a very simple and familiar interface.

[raphaelauv]

An example of fastAPI with aiohttp client :

https://github.com/tiangolo/fastapi/issues/236#issuecomment-616509245