Uvicorn: High CPU usage when using --reload

[nymous]

Using the example code

async def app(scope, receive, send):
    assert scope['type'] == 'http'

    await send({
        'type': 'http.response.start',
        'status': 200,
        'headers': [
            [b'content-type', b'text/plain'],
        ],
    })
    await send({
        'type': 'http.response.body',
        'body': b'Hello, world!',
    })

and running uvicorn main:app, CPU usage is unnoticeable (less than 1% of 1 core).
But when running with uvicorn main:app --reload, CPU usage jumps to 54% of 1 core.

Environment:
python -V: Python 3.7.2
uvicorn: 0.6.1
uname -a: Linux architect 5.0.4-arch1-1-ARCH #1 SMP PREEMPT Sat Mar 23 21:00:33 UTC 2019 x86_64 GNU/Linux

[tomchristie]

You should only use --reload in development, so I don't see this as a pressing issue.
I'd be very happy to see an alternative optional reload implementation based on watchdog, since that'd be far more efficient than our stat-based reloader. But the impetus for that would need to come from someone taking on a pull request.

[nymous]

Yes I use it only in dev, I just wanted to know if it was normal to hear my fans ramp up.

[tomchristie]

We could bump down the frequency at which we check the file timestamps?
Happy to take any other suggestions?

Reloader implementation: https://github.com/encode/uvicorn/blob/master/uvicorn/supervisors/statreload.py

[kevchentw]

Should we make reload check time configurable?
My fans is always running in full speed when I use --reload

[tomchristie]

Should we make reload check time configurable?

No to configuring reload check time - that’s too many dials. Tho there’s more efficient file watching systems (eg gunciorn has an optional dependency, and only falls back to stat checks if needed) we should probably do the same. It could also be reasonable for us to allow controls into which files are being watched (eg perhaps it’s having to scan a very large, unchanging, virtualenv, as well as the actual source code?)

[kevchentw]

This make sense. I think that we can choose which folder to watch will be great.

[kevchentw]

I found that there are --reload-dir settings already. By applying app folder, CPU usage significant decrease.

[wshayes]

@tomchristie How do you feel about using watchdog (https://github.com/gorakhargosh/watchdog) for file watching? It would add a dev dependency to the project, but it looks like it handles file watching very efficiently for most OS's with a fallback to the collect all files and then stat them every X milliseconds. I'm happy to attempt a PR with it.

[tomchristie]

How do you feel about using watchdog

Yes, but probably (?) as an optional dependency, rather than mandatory.

[skorokithakis]

This should not be closed, it's very much still an issue, and it's a bug that uvicorn takes up 50% of CPU just for polling.

[euri10]

Have you tried the new watchGod reloader?
What os are you on?
Mac os coupled with docker seems a common problem.
Without more information it's impossible to help you.

Le jeu. 7 mai 2020 à 6:45 PM, Stavros Korokithakis [e-mail]
a écrit :

This should not be closed, it's very much an issue and a bug that uvicorn
takes up 50% of CPU just for polling.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/encode/uvicorn/issues/338#issuecomment-625369140, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAINSPUQWG4NVOKR7VBWJ5LRQLQTZANCNFSM4HCAKVHQ
.

[skorokithakis]

I have not tried watchdog, is it available? My info is:

Running uvicorn 0.11.5 with CPython 3.8.2 on Linux

[euri10]

on 0.11.5 it should be imported if watchdog is found so either you pip install watchdog or pip install uvicorn[watchgodreloader]

tell us if this helps, what is the number of files you're watching ? Last time I tried on @rafalp misago project which watches quite a large number of files, I was at a nice 4% cpu all along in both statsreloader and watchgod mode, and on the same @rafalp was at 20% on a macos, so it's rather hard to tell what's going on, I'm on debian

[wshayes]

https://www.uvicorn.org/settings/

$ pip install uvicorn[watchgodreload]

On my mac, I find this keeps the docker containers running with very low CPU when inactive - 1-3% using ctop with a few hundred files being watched in multiple directories. However, Mac Docker Desktop doesn't handle mounted files well at all and docker uses a LOT of CPU even when idle.

[skorokithakis]

I'm watching around 20-30 files and it was taking up 50% CPU, I'll try watchdog and reply, thank you.

[wshayes]

Are you sure you are watching only 20-30 files? Are you using reload-dir to specify the directory and not adding in the virtualenv directory? That's an enormous idle load for FastAPI/uvicorn for just watching 20-30 files unless you are on a Raspberry Pi Linux server.

[skorokithakis]

I can check, but the virtualenv is in a different directory completely. That's why I was so aghast at the cpu usage.

[skorokithakis]

Yep:

22:04:49 $ fd | wc -l
43

uvicorn is taking up 45% of one core just watching those 43 files.

With watchgod it falls to 15%, which is still way higher than I would expect. Running without --reload has a CPU usage of 0.0, which is what I expect. There's something weird going on with the watcher, running watchgod alone takes up 7% CPU.

I have half a mind to write an abstraction library over inotify/kqueue/etc, as this is a very common thing that should exist.

Could this issue be reopened until --reload CPU usage has dropped below 1%?

[euri10]

I made 2 tests for 5min on a small project ie 93 files watched, the other from @rafalp 889 files watched for 10 minutes.

both with watchgod reloader, on my laptop, which is my slowest machine and not a beast (overall rank is 1018 according to https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i5-6300U+%40+2.40GHz&id=2609).

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       39 bits physical, 48 bits virtual
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  2
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               78
Model name:          Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz
Stepping:            3
CPU MHz:             866.920
CPU max MHz:         3000.0000
CPU min MHz:         400.0000

for misago, aka the big project:

for the small project:

so it's fair to say that on 800+ files watched it's sweating, but on a rather small one we're at 1-2%

I dont know what to add, would gladly help but as you can see I really can't reproduce.

Maybe others will have ideas as to things to look at, I have to admit I'm at a loss right now.

[thomhickey]

pipenv install "uvicorn[watchgodreload]"

then use reload_dirs:

    uvicorn.run("main:app", host="0.0.0.0", port=int(config.APP_PORT), reload=config.is_local_env(),
                reload_dirs=["app", "views", "conf"])

now fan is quiet and my MBP isn't too hot to put on my lap anymore ... wow!

[rafalp]

Here's another observation for Misago in Docker: When I disable --reload in docker container and restart it, docker's CPU usage stays at comfortable ~10%. So even if watchgod is not using CPU on its own, its doing something that causes docker to rev up.

Here's CPU without reloader:

Here's with reloader:

This CPU usage makes sense for statreload because its the one that keeps poking filesystem a lot. Watchgod is supposed to listeen for filesystem events and react to those instead.

So I've checked what Watchgod is actually doing, and then realized it _doesn't use filesystem events_ for changes detection. Instead it uses same approach that statreload does: 4 times a second it walks filesystem and compares mtime of every file against dict with previous mtimes. This has no chance to be performant in docker containers used for dev.

So we actually lie to people in the docs when we say that watchgod is faster because it uses filesystem events for change detection.

I think we should implement something like Watchman to actually have performant in-dev code reload strategy. I know Django does this.

There's also an issue of frequency at which filesystem is checked for changes. Django checks filesystem every second. Uvicorn every 250ms. Filesystem ops are relatively cheap in Docker, as long as they are infrequent. Here we are doing IO operation potentially thousands times a second.

For comparison, here's my CPU graph after I've edited Uvicorn to pool every second:

CPU usage is still noticeable, but it's not enough to get my laptop's fans to spin.

So we still have performance issue. Its not for small projects or people running on watcher natively, but it's still a deal-breaker for people using dev setups based on docker.

[uSpike]

I had this issue with my project.

With only --reload, it was eating up 25% of a 4 core CPU watching 8135 files in my venv/ folder.

By adding --reload-dir src/ where src/ contains my application, the CPU issue went away completely.

Maybe a decent solution is to have uvicorn ignore common virtualenv directory names, like venv/ .venv/, etc?

[shawnwall]

I've learned I have to be very specific in what folders get mounted via docker, and what folders are watched to not end up using things like .mypy_cache, .coverage, .nox, etc

[rafalp]

Here's my CPU usage in 0.12 running with --reload --reload-delay 1 --reload-dir misago --reload-dir plugins combined with few accurate docker volumes instead of one volume for project's repo:

I'll say this is much better.

[florimondmanca]

@rafalp Looking great! Nice little knob added there in #774. Shall we consider this issue as resolved, then?

[rafalp]

Let's close this. If what we've got in is not enough for somebody, we'll open new issue to track work for event-based reloader.