Tokio: Major performance regression in 0.2 release

[Ralith]

Quinn's 1GiB loopback benchmark has lost nearly 50% of its speed across the changes made just prior to release:

tokio 0.2.0-alpha.6, 2x single-threaded runtime: 1.17Gbps, 166% CPU usage, flamegraph, code
tokio 0.2.1, 2x single-threaded runtime: 0.66Gbps, 107% CPU usage, flamegraph, code
tokio 0.2.1, 1x multi-threaded runtime: 0.86Gbps, 270%(!!) CPU usage, flamegraph

To reproduce, clone, select branch master or tokio-0.2-bench, cd bench && cargo run --release.

The benchmark normally uses an independent single-threaded executor for each of the two endpoints, because there is little meaningful parallelism opportunity and the objective is to benchmark Quinn's mechanics rather than stress-test tokio's wakeups, so it's particularly remarkable that in tokio 0.2.1 the multithreaded runtime is faster

[Ralith]

Here's a flamegraph for transferring 10GiB on 0.2.1 2x single-threaded, to help weed out the noise.

[hawkw]

@Ralith do you have a similar flamegraph for the 2.0-alpha.6 benchmark?

[Ralith]

Sure. I think flamegraphs are probably not going to be the key to this, though, they're way too similar.

[Ralith]

Looks like this benchmark was probably a casualty of the switch from lazy to eager registration with reactors; that should be straightforward to fix on quinn's side. Will have updated figures within a day or two, allowing for the holiday.

[carllerche]

After updating the quinn bench, i see identical perf:

https://gist.github.com/carllerche/9de58fe76dd9fac1be6502b930dcc812

[Ralith]

For a little more context for posterity: quinn's benchmark was structured in a way that lead to two completely independent threads in the alpha, but in stable resulted in a single thread being responsible for issuing wakeups to both endpoints. This was working as intended on tokio's side, and fixable by rearranging how setup happened in quinn's benchmark. Code that doesn't do weird things like create multiple runtimes in the same place is unlikely to run into a similar issue.