Fastapi: The memory usage piles up over the time and leads to OOM

[prav2019]

First check

• [x] I added a very descriptive title to this issue.
• [x] I used the GitHub search to find a similar issue and didn't find it.
• [x] I searched the FastAPI documentation, with the integrated search.
• [x] I already searched in Google "How to X in FastAPI" and didn't find any information.
• [x] I already read and followed all the tutorial in the docs and didn't find an answer.
• [ ] I already checked if it is not related to FastAPI but to Pydantic.
• [ ] I already checked if it is not related to FastAPI but to Swagger UI.
• [ ] I already checked if it is not related to FastAPI but to ReDoc.
• [x] After submitting this, I commit to one of:
  
  
  
  • Read open issues with questions until I find 2 issues where I can help someone and add a comment to help there.
  
  
  • I already hit the "watch" button in this repository to receive notifications and I commit to help at least 2 people that ask questions in the future.
  
  
  • Implement a Pull Request for a confirmed bug.
  
  

Example

Here's a self-contained, minimal, reproducible, example with my use case:

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
def read_root():
    return {"Hello": "World"}

Description

• Open the browser and call the endpoint /.
• It returns a JSON with {"Hello": "World"}.
• But I expected it to return {"Hello": "Sara"}.

Environment

• OS: [e.g. Linux / Windows / macOS]:
• FastAPI Version [e.g. 0.3.0]:

To know the FastAPI version use:

python -c "import fastapi; print(fastapi.__version__)"

• Python version:

To know the Python version use:

python --version

Additional context

Tracemalloc gave insight on the lines , that are top consumers of memory: (top one seems to be below line in uvicorn)
/usr/local/lib/python3.6/site-packages/uvicorn/main.py:305:
Line:
loop.run_until_complete(self.serve(sockets=sockets))

[prav2019]

same issue i faced @prav2019

Any solution to overcome that @Riki-1mg

[Riki-1mg]

No @prav2019,
are you using aiohttp as http clients in your service ?
https://github.com/tiangolo/fastapi/issues/1623

[prav2019]

No @Riki-1mg

[teymour-aldridge]

@app.get("/")
def read_root():
  return {"Hello": "World"}

Surely the expected behaviour here is to return {"Hello": "World"}?

If you want this function to return {"Hello": "Sara"} you'd probably need to do something like:

@app.get("/")
def read_root():
  return {"Hello": "Sara"}

Further, I can't reproduce your error on my "machine" (it's sitting on a cloud somewhere). You can see the full details here but everything looks to be working fine.

I suspect that this is specific to your operating system setup, etc. Would you please provide some more info needed/useful to know in context of how to reproduce the error?

• How much RAM does your machine have?
• How much memory is each function using (ideally include all the debugging output).
  This wil look something like the below (example from the Python documentation)

[ Top 10 ]
<frozen importlib._bootstrap>:716: size=4855 KiB, count=39328, average=126 B
<frozen importlib._bootstrap>:284: size=521 KiB, count=3199, average=167 B
/usr/lib/python3.4/collections/__init__.py:368: size=244 KiB, count=2315, average=108 B
/usr/lib/python3.4/unittest/case.py:381: size=185 KiB, count=779, average=243 B
/usr/lib/python3.4/unittest/case.py:402: size=154 KiB, count=378, average=416 B
/usr/lib/python3.4/abc.py:133: size=88.7 KiB, count=347, average=262 B
<frozen importlib._bootstrap>:1446: size=70.4 KiB, count=911, average=79 B
<frozen importlib._bootstrap>:1454: size=52.0 KiB, count=25, average=2131 B
<string>:5: size=49.7 KiB, count=148, average=344 B
/usr/lib/python3.4/sysconfig.py:411: size=48.0 KiB, count=1, average=48.0 KiB

• Does your out-of-memory error include a traceback? Please include that if that's the case.

[prav2019]

@teymour-aldridge : here is the debug statistics:
Memory Statistics
Top 10 Files
/usr/local/lib/python3.6/site-packages/uvicorn/main.py:305: size=1652 KiB (+1499 KiB), count=4597 (+4172), average=368 B KB

/usr/local/lib/python3.6/site-packages/starlette/applications.py:136: size=1288 KiB (+1173 KiB), count=2290 (+2086), average=576 B KB

/usr/local/lib/python3.6/threading.py:347: size=943 KiB (+854 KiB), count=1836 (+1657), average=526 B KB

/usr/local/lib/python3.6/queue.py:145: size=919 KiB (+835 KiB), count=1783 (+1619), average=528 B KB

/usr/local/lib/python3.6/asyncio/locks.py:233: size=885 KiB (+807 KiB), count=9633 (+8771), average=94 B KB

/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:82: size=788 KiB (+717 KiB), count=6876 (+6264), average=117 B KB

/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:77: size=751 KiB (+684 KiB), count=2289 (+2086), average=336 B KB

/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:146: size=725 KiB (+662 KiB), count=15984 (+14611), average=46 B KB

/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/result.py:376: size=657 KiB (+590 KiB), count=10490 (+9426), average=64 B KB

/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:285: size=609 KiB (+555 KiB), count=4589 (+4183), average=136 B KB

Scanned Lines that consumes more memory
uvicorn/main.py

loop.run_until_complete(self.serve(sockets=sockets))
starlette/applications.py

scope["app"] = self
python3.6/threading.py

waiters_to_notify = _deque(_islice(all_waiters, n))
python3.6/queue.py

self.not_empty.notify()
asyncio/locks.py

self._waiters = collections.deque()
http/httptools_impl.py

self.parser = httptools.HttpRequestParser(self)
http/httptools_impl.py

self.config = config
http/httptools_impl.py

self.parser.feed_data(data)
engine/result.py

for obj_elem in elem[4]
http/httptools_impl.py

self.timeout_keep_alive, self.timeout_keep_alive_handler

[prav2019]

@teymour-aldridge the above grows and cause OOM after a while

[teymour-aldridge]

@prav2019 I can't reproduce the bug on either my machine or on a cloud-hosted linux container; this leads me to believe that the problem is in the way that your machine/environment setup.

In the issue template, it asks for the following fields – would you mind filling them in?

OS: [e.g. Linux / Windows / macOS]:
FastAPI Version [e.g. 0.3.0]:

Also, there's a "checklist" at the top of the issue which you should fill out!

[prav2019]

@teymour-aldridge , this usually happens when there are some traffic over a period of time. Usually this is happening in our prod environment.
OS: using docker image: python 3.6 [so debian ] **Linux
FastAPI version: fastapi[all]==0.20.0

[prav2019]

@teymour-aldridge , current status:
Top 10 Files
/usr/local/lib/python3.6/site-packages/uvicorn/main.py:305: size=3650 KiB (+1922 KiB), count=10158 (+5349), average=368 B KB

/usr/local/lib/python3.6/site-packages/starlette/applications.py:136: size=2851 KiB (+1504 KiB), count=5069 (+2673), average=576 B KB

/usr/local/lib/python3.6/threading.py:347: size=2104 KiB (+1110 KiB), count=4083 (+2155), average=528 B KB

/usr/local/lib/python3.6/queue.py:145: size=2049 KiB (+1087 KiB), count=3974 (+2109), average=528 B KB

/usr/local/lib/python3.6/asyncio/locks.py:233: size=1948 KiB (+1017 KiB), count=21295 (+11208), average=94 B KB

/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:82: size=1744 KiB (+920 KiB), count=15226 (+8037), average=117 B KB

/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:77: size=1663 KiB (+877 KiB), count=5068 (+2673), average=336 B KB

/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:146: size=1598 KiB (+839 KiB), count=35181 (+18488), average=47 B KB

/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:285: size=1349 KiB (+712 KiB), count=10163 (+5364), average=136 B KB

/usr/local/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py:216: size=1288 KiB (+676 KiB), count=30112 (+15818), average=44 B KB

[teymour-aldridge]

0.20.0 is a somewhat old version of FastAPI, what happens if you use the latest release instead?

[prav2019]

@teymour-aldridge haven't tried with latest version ? but i like to give it a shot, if there were any memory fix done with the versions after 0.20.0

[prav2019]

@teymour-aldridge updated to latest version, will report back, once I see the result, Thanks

[prav2019]

@teymour-aldridge I checked on our staging, after updating to latest versions [fast api and uvicorn], the memory issue still exists

[prav2019]

@teymour-aldridge can this one work, i saw this in uvicorn documentation:
--limit-max-requests - Maximum number of requests to service before terminating the process. Useful when running together with a process manager, for preventing memory leaks from impacting long-running processes.

[teymour-aldridge]

@prav2019 I don't know, it might do. How many requests are you handling ~and how many machines do you have~?

[prav2019]

@teymour-aldridge or trying to add gunicorn!!!

[curtiscook]

im running into the same issue - memory usage slowly builds over time, runnign on gunicorn with 4 uvicorn workers

[erikreppel]

+1, seeing the same issue

fastapi==0.55.1
uvicorn==0.11.5
gunicorn==19.10.0

Gunicorn + uvicorn worker class

[curtiscook]

Reading through the uvicorn code.. adding the max-requests effectively just restarts the server as soon as you hit some arbitrary number of requests?

https://github.com/encode/uvicorn/blob/e77e59612ecae4ac10f9be18f18c47432be7909a/uvicorn/main.py#L537-L539

I can't find any good documentation on what this number should be either 500? 1000? 10k? 100k?

If anyone has any experience/advice here, I'm all ears

[prav2019]

@curtiscook The max-requests restarts the service completely, we need to configure workers to keep one running always, when we restart another, was able to solve memory issue, but got into one more, now sometimes I get multiple requests to workers with same data and each worker creates new entry into database.

[drisspg]

@prav2019 So what exactly solved your OOM issue, was it setting the max-requests?

[curtiscook]

Hi,

I actually have not solved my memory leak issue but it's small enough to not be a huge concern. I'm also seeing the memory leak in other async processes so it might be an issue with long running event loops in async python?

@curtiscook The max-requests restarts the service completely, we need to configure workers to keep one running always, when we restart another, was able to solve memory issue, but got into one more, now sometimes I get multiple requests to workers with same data and each worker creates new entry into database.

That's what I thought it might do. Not really a great solution then :(

[teymour-aldridge]

I would have thought it would be pretty tricky to have a memory leak in Python. Perhaps an underlying issue in the interpreter (C is very good for memory leaks :D) or a C extension module?

Anyway this has peaked my interest, so I'll try and investigate a little to see what's causing the problem.

Is this only happening in Docker containers, or can it be reproduced across a number of devices? I'm not experiencing this issue on my machine, having left a FastAPI process running for a few days (nothing unusual happened).

It's generally a good move to use ephemeral (short-lived) processes to run applications and regularly recycle them in order to reduce the risk/impact a memory leak (which tend to build up over time) can have.

[curtiscook]

I haven't tested outside of docker containers/ heroku docker containers

[teymour-aldridge]

Ah. I'll try and do some profiling. Unfortunately my time is pretty scarce these days with the number of different projects I'm working on but fingers crossed.

[binbinah]

+1

python 3.6
fastapi==0.60.1
uvicorn==0.11.3

uvicorn main:app --host 0.0.0.0 --port 8101 --workers 4
docker：2 core 2GB memory，CentOS Linux release 7.8.2003 (Core)

client call the function below per minute, and server memory usage slowly builds over time.

...
from fastapi import BackgroundTasks
...

@router.get('/tsp/crontab')
def tsp_crontab_schedule(topic: schemas.AgentPushParamsEnum,
                         background_tasks: BackgroundTasks,
                         api_key: str = Header(...)):
    crontab = CrontabMain()

    if topic == topic.schedule_per_minute:
        background_tasks.add_task(crontab.schedule_per_minute)

[binbinah]

update:

I add async before def and it worked
refer： https://github.com/tiangolo/fastapi/issues/596#issuecomment-647704509

@router.get('/tsp/crontab')
def tsp_crontab_schedule(topic: schemas.AgentPushParamsEnum..)
    pass

to

@router.get('/tsp/crontab')
async def tsp_crontab_schedule(topic: schemas.AgentPushParamsEnum..)
    pass

[prav2019]

https://github.com/tiangolo/fastapi/issues/1624#issuecomment-676517603 . The max request fixed OOM, as it restarts server, but it opened to lot of concurrency issues.

[prav2019]

did replacing def with async def helped, because I tried long back it didn't!!!!!

[curtiscook]

update:

I add async before def and it worked
refer： #596 (comment)

@router.get('/tsp/crontab')
def tsp_crontab_schedule(topic: schemas.AgentPushParamsEnum..)
    pass

to

@router.get('/tsp/crontab')
async def tsp_crontab_schedule(topic: schemas.AgentPushParamsEnum..)
    pass

either way, you shouldn't have expanding memory. running the route in def is supported and meant to process requests in a new threadpool?

did replacing def with async def helped, because I tried long back it didn't!!!!!

I've always been running async

[AdolfVonKleist]

I am running this in docker with:

• python==3.8.6
• fastapi==0.61.1
• uvicorn==0.11.8

and I am seeing the same issue for some time. If I understand this thread, it seems that the recommendation is to restart the uvicorn worker(s) periodically, and never run uvicorn alone without a process manager. I guess it also better to transfer attention to uvicorn? Or just not use uvicorn perhaps.

[ycd]

Related to #596, that issue already contains a lot of workarounds and information, please follow the updates over there.

[curtiscook]

Related to #596, that issue already contains a lot of workarounds and information, please follow the updates over there.

It looks like #596 is due to using def instead of async def ? Ive seen this with async def

[ycd]

@curtiscook no my point was there are already a lot of workarounds, keeping them in one place would be better for future references, and the developers who face the same problem in the future, can search for a solution in one place that fits their case.

Even if most answers are about using async def instead of def. The title and the context are very clear actually Because the issue wasn't about using def instead of async def in our case. My colleague opened that issue more than one year ago and we're no longer facing memory problems, we're not using async def either.

[delijati]

run into the same error def -> memory leak async def -> no memory leak thanks @curtiscook