Httpx: HTTPX AsyncClient slower than aiohttp?

[imbolc]

I just found async-client to be significantly slower comparing to aiohttp, escecially for single request:

| |aiohttp | httpx | aiohttp/httpx |
|---|---|---|---|
|single, rps |1062 | 141 | 7.5 |
| session, rps |1862|1197 | 1.5 |
| session/single | 1.8 | 8.5 | |

I tried both release and master, results are pretty the same. The code of the benchmark is here: https://gist.github.com/imbolc/15cab07811c32e7d50cc12f380f7f62f

[florimondmanca]

Thanks, this is very interesting quantified insights!

I went ahead and added the aiohttp/httpx ratio to your table.

I used your script to run the benchmark on my own machine, and observe similar results:

| | aiohttp | httpx | aiohttp/httpx |
|--|--|--|--|
| single, rps | 220 | 48 | 4.6 |
| client, rps | 495 | 331 | 1.5 |
| client/single | 2.3 | 6.9 | |

[florimondmanca]

The client/single ratio for HTTPX is not surprising to me — we know that using a client significantly increases performance.

The aiohttp/httpx ratio in the client case isn't surprising either — I had already noted that we were slower than aiohttp in the past (don't think I posted the results on GitHub though).

What's more surprising to me is, as you said, the 3x higher aiohttp/httpx ratio in the single case. I interpret it as "setting up a client (or opening a single connection, or whatever) is not as efficient as it could be".

I'll run httpxprof over your scripts and see what other insights I can come up with. :-)

[florimondmanca]

Okay, so apparently one _massively useless_ thing we do is eager TLS setup on Client instantiation, _even though none of the eventually requested URLs uses HTTPS_.

If we run:

pip install -e git+https://github.com/florimondmanca/httpxprof.git#egg=httpxprof
httpxprof run async_single
httpxprof view async_single

We get this view:

=> 92% (!) of the time spent instantiating the client (which itself is 62% of the total time making a single request with a client) is spent setting up SSL (.load_ssl_context()).

Compare this to aiohttp -- client instantiation is not even visible on this graph, and the bulk of the time is only spent, well, making the actual request to the non-TLS URL:

I used the following to get the profile above:

# aiohttp_single.py
import asyncio
import aiohttp
import httpxprof

async def main() -> None:
    for _ in httpxprof.requests():
        async with aiohttp.ClientSession() as session:
            async with session.get(httpxprof.url):
                pass

asyncio.run(main())

httpxprof run aiohttp_single.py
httpxprof view aiohttp_single.py

So one action point already might be to lazily load the SSL configuration on the first request that uses HTTPS.

[victoraugustolls]

I might be missing something, but what if loading the SSL context is needed in both aiohttp and httpx, is the time doing so somewhat similar? And the total time? I can also try to take this numbers later

I see that in the gist the host is http and not https, that’s why I’m questioning this

[florimondmanca]

what if loading the SSL context is needed in both aiohttp and httpx

Yes, I actually just realized that optimizing for requests against an insecure host is probably marginally useful for real-world situations anyway. We should be comparing aiohttp and HTTPX requesting a host over HTTPS.

I'm going to setup a local server with HTTPS turned on (see instructions here and run the profiling again, this time requesting https://localhost:8000. :+1:

[victoraugustolls]

Don't know how much this would affect the final result, but aiohttp uses cchardet and aiodns for optimization.

[imbolc]

optimizing for requests against an insecure host is probably marginally useful for real-world situations anyway

Yep, an important one is a bunch of microservices spinning on localhost

[florimondmanca]

@imbolc I forked your gist and created a version that runs wrk on a server running on HTTPS, in which both aiohttp and HTTPX request the server using the same CA bundle: https://gist.github.com/florimondmanca/fbc85b58e9ce61e74b73df1e42829838

Running it on my machine, I get the updated results below:

| | aiohttp | httpx | aiohttp/httpx |
|----------------|---------|-------|---------------|
| single (req/s) | 121 | 48 | 2.5 |
| client (req/s) | 284 | 221 | 1.2 |
| client/single | 2.3 | 4.6 | 0.5 |

The difference for the single request case went from 8x to 2-3x, which is more reasonable, and not entirely surprising to me (we haven't been very focused in optimization so far).

---

Besides, running httpxprof again on an HTTPS server, in the single-request case:

• Client instantiation now only represents 33% of the time spent making a single request (instead of 60+%).
• TLS setup is now only about 50% of client instantiation time -- so about 15% of the total.

(The improvement with the previous setup might come from the fact that I now explicitly pass verify="client.pem", whereas previously HTTPX had to lookup certs via certifi.)

The aiohttp equivalent setup (using SSL Control for TCP Sockets) spends about 16% of the time in ssl.create_default_context().

So actually, there's no real burden on our side due to TLS/SSL.

Still, aiohttp looks to set up TLS more efficiently than we do. We get our certs from certifi (I've already seen some people argue this may not actually be the best choice?), but I'm not sure how aiohttp handles defaults certs… Anyone's got a clue? I haven't seen anything in their docs.

[florimondmanca]

but I'm not sure how aiohttp handles defaults certs…

Ah, so from ClientSession.request() I see that they use ssl.create_default_context():

ssl: SSL validation mode. None for default SSL check (ssl.create_default_context() is used) [...]

I tried using httpx.get("https://google.com", verify=ssl.create_default_context()), and it works like a charm. So do we _need_ certifi? Could this be related to #302?

[victoraugustolls]

Don't know if it 100% performance related, but on #832 I added a repository with sample code where timeouts happen with httpx, but not with aiohttp with the same request volume.

[florimondmanca]

Hi, going to close this off as "yes, we're maybe 2x-3x slower than aiohttp, but getting up to speed there isn't a priority for 1.0", though PRs on anything that might be a performance burden are still very much welcome! Thanks all.

[tomchristie]

I'd be pretty skeptical that we're comparing like-for-like (eg. there's various SSL, .netrc behaviour etc. stuff that may differ and mean that we have a heavier client instantiation than aiohttp), or that single requests to a local server are a meaningful metric.

If we do look at any benchmarking at some point, I'd want to look at measurements after a client instance is created, since you really want to be instantiating a single client instance which is then used for the lifetime of the app.

[imbolc]

I'd want to look at measurements after a client instance is created

It's included into the benchmark, the second row in the table

[JustJia]

Okay, so apparently one _massively useless_ thing we do is eager TLS setup on Client instantiation, _even though none of the eventually requested URLs uses HTTPS_.

If we run:

pip install -e git+https://github.com/florimondmanca/httpxprof.git#egg=httpxprof
httpxprof run async_single
httpxprof view async_single

We get this view:

=> 92% (!) of the time spent instantiating the client (which itself is 62% of the total time making a single request with a client) is spent setting up SSL (.load_ssl_context()).

Compare this to aiohttp -- client instantiation is not even visible on this graph, and the bulk of the time is only spent, well, making the actual request to the non-TLS URL:

I used the following to get the profile above:

# aiohttp_single.py
import asyncio
import aiohttp
import httpxprof

async def main() -> None:
    for _ in httpxprof.requests():
        async with aiohttp.ClientSession() as session:
            async with session.get(httpxprof.url):
                pass

asyncio.run(main())

httpxprof run aiohttp_single.py
httpxprof view aiohttp_single.py

So one action point already might be to lazily load the SSL configuration on the first request that uses HTTPS.

Okay, so apparently one _massively useless_ thing we do is eager TLS setup on Client instantiation, _even though none of the eventually requested URLs uses HTTPS_.

If we run:

pip install -e git+https://github.com/florimondmanca/httpxprof.git#egg=httpxprof
httpxprof run async_single
httpxprof view async_single

We get this view:

=> 92% (!) of the time spent instantiating the client (which itself is 62% of the total time making a single request with a client) is spent setting up SSL (.load_ssl_context()).

Compare this to aiohttp -- client instantiation is not even visible on this graph, and the bulk of the time is only spent, well, making the actual request to the non-TLS URL:

I used the following to get the profile above:

# aiohttp_single.py
import asyncio
import aiohttp
import httpxprof

async def main() -> None:
    for _ in httpxprof.requests():
        async with aiohttp.ClientSession() as session:
            async with session.get(httpxprof.url):
                pass

asyncio.run(main())

httpxprof run aiohttp_single.py
httpxprof view aiohttp_single.py

So one action point already might be to lazily load the SSL configuration on the first request that uses HTTPS.

How do you got the code Flame Graph ? It look like usefull

[florimondmanca]

@JustJia It's from httpxprof which is a small wrapper around SnakeViz, code is here :) https://github.com/florimondmanca/httpxprof