Tide: benchmark comparison between tide and actix-web

[kingluo]

I make a simple benchmark for actix-web and tide.
The client and server are placed on separate machine, which has 2 CPU.

The client runs wrk:

wrk -c100 -d60s http://test_server:8080

tide:

source code:

use async_std::task;

fn main() -> Result<(), std::io::Error> {
    task::block_on(async {
        let mut app = tide::new();
        app.at("/").get(|_| async move { "hello" });
        app.listen("0.0.0.0:8080").await?;
        Ok(())
    })
}

result:

160% CPU

  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.98ms  232.59us   7.82ms   84.98%
    Req/Sec    51.18k     0.94k   53.76k    73.75%
  6111056 requests in 1.00m, 815.91MB read
Requests/sec: 101840.52
Transfer/sec:     13.60MB
````

**actix-web**:

source code:

```rust
use actix_web::{get, middleware, web, App, HttpRequest, HttpResponse, HttpServer};

#[get("/")]
async fn no_params() -> &'static str {
    "hello"
}

#[actix_rt::main]
async fn main() -> std::io::Result<()> {
    std::env::set_var("RUST_LOG", "actix_server=info,actix_web=info");
    env_logger::init();

    HttpServer::new(|| {
        App::new()
            .service(no_params)
    })
    .bind("0.0.0.0:8080")?
    .workers(2)
    .run()
    .await
}

result:

150% cpu

  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   535.53us  386.02us   9.49ms   87.82%
    Req/Sec    89.62k    27.92k  128.04k    38.00%
  10698218 requests in 1.00m, 1.21GB read
Requests/sec: 178283.33
Transfer/sec:     20.57MB

It looks like tide runs most slower than actix-web and uses more CPU.
From high level vision, could anybody explain the possible reason?

[chrisabruce]

First of all, I think Tide is doing quite well for how young the project is, I suspect there will be lots of room for improvement. Additionally, with the push to remove unsafe from Actix, I wonder if some performance will take a hit.

How are you getting CPU usage? Didn't realize wrk outputs that?
Also, do you need to explicitly set workers(2) in actix?

Looking at the results seems to indicate Tide has less latency. So I wonder if Tide is making full use of both cores?

I will look to do similar benchmarking. For me I would like to see the results run over 30 minutes or so. In my testing, I have seen actix go down in rps over longer times (not sure why).

[kingluo]

How are you getting CPU usage? Didn't realize wrk outputs that?
Also, do you need to explicitly set workers(2) in actix?

I check the cpu in top manually, that's not the output of wrk.
Yes, I need to set workers number to 2 because I have 2 cpu. On the other hand, nothing to setup in tide, because the tide uses all cpu by default, I don't know why, maybe async-std does so?

Looking at the results seems to indicate Tide has less latency.

The actix has less latency.

[chrisabruce]

I misread the latency. You are correct Actix is less. Actix still has more deviation and higher max latency. However, this could be different if run longer.

I thought, maybe incorrectly, that actix defaulted to the number of CPUs as well. Guess not.

[yoshuawuyts]

@kingluo Thanks for sharing your benchmark numbers. Generally in benchmarks like these it's useful to have more details on the system they've been run on: Which OS was used? Which version of Rust? Which build flags? CPU and architecture? Ram? What kind of network were the machines connected through? All these variables matter.

Generally we're not too interested in benchmarks right now because it's not been an area of focus. It's expected we'll perform less well than a highly-optimized framework such as Actix. But I think from your benchmark it shows that even if we score lower than Actix, we still have reasonable throughput.

I'm going to go ahead and close this issue; our focus right now is for Tide to become a framework that empowers people to become _productive_, while sidestepping any obvious performance footguns. Optimization is currently not a priority, and even if benchmarks like these are pretty cool and fun to play with, they're not really useful for the development of Tide at this stage.

I hope this makes sense. Thanks a lot for the work you've done!

[pickfire]

@yoshuawuyts But as for productivity, isn't that what rocket.rs have been targeting? Similarly, their main target is not performance too IIRC.