Fastapi: [QUESTION] cannot increase throughput via async def

[ysde]

Description

How can I increase the throughput of fastapi?

I have started a fastapi server.
And I have a api that will call google's api and return the token from google.

I found that if I use "async def", the throughput is slower than normal "def"
The log will output like this, more like sequential to me:

DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None

If I use normal "def" function then the throughput will better.

DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:    https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG:    https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG:    https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None

And I want to increase the throughput that but don't know how, I tried to configure worker numbers, but the throughput seems remain the same.

Thank you very much.

[steinitzu]

How are you calling the google API?
Are you using a blocking http library (e.g. "requests")?

[sm-Fifteen]

The FastAPI documentation mentions that an async def route or dependency will be called on the server's event loop directly, with the implicit requirement that it must not block and should spend very little time running if possible. A def route/dependency, on the other hand, will be run in a separate thread, which is intended to let you call sync libraries like requests or SQLAlchemy without having to do any extra work.

This means that whether a route is marked as async def or def has a number of subtle and sometimes counterintuitive effects:

def common_parameters(q: str = None, skip: int = 0, limit: int = 100):
    return {"q": q, "skip": skip, "limit": limit}

@app.get("/items/")
def read_items(commons: dict = Depends(common_parameters)):
    return commons

Neither the route or the dependency contain await, so shouldn't they be def? No, because FastAPI uses def to determine which functions are supposed to be spawned inside a thread to avoid blocking the server's main thread. It will still work, but it will spawn lots and lots of threads to perform trivial operations, and that could potentially hurt performance.

Likewise (and this is the issue you're probably running into):

@app.get("/")
async def root():
    requests.get('http://slowsite.example/')
    return {'result': 'done'}

Even though requests.get is a network operation, requests only does blocking I/O, so it will pause until it has received a response. That's a problem because async def tells FastAPI that this function doesn't have to run in its own thread and can instead be run on the server's event loop directly. If the request takes 10 seconds to respond, that's 10 seconds during which your server won't be responsive because its main thread/event loop is still busy waiting for that operation (which was supposed to be as short as possible and non-blocking) to complete.

Assuming I guessed correctly, you should either keep the route that calls google's API as a def function (so your request calls happen in separate threads) or swap your HTTP library for an async alternative, like httpx.

[ysde]

How are you calling the google API?
Are you using a blocking http library (e.g. "requests")?

Hi @steinitzu

Thanks for the reply.

I used https://github.com/googleapis/google-auth-library-python to get the google token.

I think it's a blocking library.
And I try to wrap the call with a async def function, and use await to wait the network io.
I thought I could avoid blocking the thread just by using the await and async declaration.

async def get_token():
    google_credentials.refresh(Request())
    return {"token": google_credentials.token}

@router.get("/google/token")
async def get_google_token():
    task2 = asyncio.create_task(get_token())
    return await task2

I also tried to use def, and increase the fastapi's worker number, but couldn't get good result.

def get_token():
    google_credentials.refresh(Request())
    return {"token": google_credentials.token}

@router.get("/google/token")
def get_google_token():
    return get_token()

Is there anyways that I can improve the throughput of my api server ?

Thank you very much

[ysde]

The FastAPI documentation mentions that an async def route or dependency will be called on the server's event loop directly, with the implicit requirement that it must not block and should spend very little time running if possible. A def route/dependency, on the other hand, will be run in a separate thread, which is intended to let you call sync libraries like requests or SQLAlchemy without having to do any extra work.

This means that whether a route is marked as async def or def has a number of subtle and sometimes counterintuitive effects:

def common_parameters(q: str = None, skip: int = 0, limit: int = 100):
    return {"q": q, "skip": skip, "limit": limit}

@app.get("/items/")
def read_items(commons: dict = Depends(common_parameters)):
    return commons

Neither the route or the dependency contain await, so shouldn't they be def? No, because FastAPI uses def to determine which functions are supposed to be spawned inside a thread to avoid blocking the server's main thread. It will still work, but it will spawn lots and lots of threads to perform trivial operations, and that could potentially hurt performance.

Likewise (and this is the issue you're probably running into):

@app.get("/")
async def root():
    requests.get('http://slowsite.example/')
    return {'result': 'done'}

Even though requests.get is a network operation, requests only does blocking I/O, so it will pause until it has received a response. That's a problem because async def tells FastAPI that this function doesn't have to run in its own thread and can instead be run on the server's event loop directly. If the request takes 10 seconds to respond, that's 10 seconds during which your server won't be responsive because its main thread/event loop is still busy waiting for that operation (which was supposed to be as short as possible and non-blocking) to complete.

Assuming I guessed correctly, you should either keep the route that calls google's API as a def function (so your request calls happen in separate threads) or swap your HTTP library for an async alternative, like httpx.

@sm-Fifteen Thank you for the detailed explanation.

If I use google's library https://github.com/googleapis/google-auth-library-python (which seems like a blocking library, I cannot find async one)

How can I increase the number of spawned threads?
Is there any ways that I can improve the throughput?

Thank you

[sm-Fifteen]

If I use google's library https://github.com/googleapis/google-auth-library-python (which seems like a blocking library, I cannot find async one)

How can I increase the number of spawned threads?
Is there any ways that I can improve the throughput?

There wouldn't really be much point. If you have one thread by blocking connection, adding more threads won't really improve things, unless your question is "How can I create more threads to make sure I never have a connection waiting for a thread?". FastAPI and Starlette use the event loop's default executor, which should mean ThreadPoolExecutor. According to the official Python doc, it uses os.cpu_count() * 4 in python 3.5 to 3.7, and min(32, os.cpu_count() + 4) starting with python 3.8.

If you wanted to change that, you could probably just use set_default_executor before you initialize your application, though I would argue that this shouldn't be needed.

EDIT: The library you linked doesn't use Requests, but it uses urllib3 directly instead, so the effect is the same: Calling the auth functions will block the thread until the remote server replies.

[ysde]

Hi @sm-Fifteen

I see.

Thank you very much

[tiangolo]

Thanks for the help here everyone! :clap: :bow:

Thanks for reporting back and closing the issue @ysde :+1: