Lit-html: Benchmarks

[justinfagnani]

We need some basic rendering benchmarks to make sure we don't regress, and measure any improvements.

[IagoLast]

I always wanted to try speedracer, maybe I could set up a simple demo with some basic metrics. what do you think?

[kyuwoo-choi]

I love @IagoLast's idea. I have a hunch lit gotta a be fast. But demos and examples are lacking.

[ruphin]

I have dbmon implementation with lit-html if you are interested in that. There's no automated way to benchmark with it, but it gives an idea of what performance is like.

[merlinnot]

Is it published? I can’t find it on the website.

[ruphin]

No, it's a private repo of mine that I use for testing. I can publish it somewhere if there's interest.

[CaptainCodeman]

lit-html is included in this JS framework benchmark

See latest snapshot results

[ruphin]

lit-html is testing a little slower than I expect it to in that benchmark. I'm working on optimised implementations for lit-html, lit-element, lite-html, and Gluon.

The benchmarking suite overall is very advanced and robust (a framework benchmark framework?) but it's not very convenient to get quick results. I'll also add some wrapper script to make running a single benchmark a little more convenient so it is easier to use it as a tracking mechanism for performance goals.

[ruphin]

I ran all the benchmarks, and I think the results are overall quite positive. Results are available here: https://files.ruph.in/benchmarks.html

Justin was correct in his assumption that the repeat() benchmark was broken, the fixed version is somewhat less well off. Notably, HyperHTML is doing quite good in the keyed department, so perhaps it's worth investigating how they get this level of performance.

I'm working on a trimmed down version of this benchmark that only runs lit-html, with a single command.

[Westbrook]

@ruphin I wonder if resurrecting any of the work found https://github.com/Polymer/lit-html/pull/260 would help with the keyed results.

[ruphin]

Once I have a more compact benchmarking tool it will be quite easy to test small changes compared to previous implementation. I'll post when I have updates.

[justinfagnani]

@ruphin I've been working on a benchmark runner to include in this repo. It automatically launches browsers, records the benchmark result and machine stats, and writes them to a file. I've been trying to get that finished to test all PRs against.

[logicalphase]

@ruphin @justin All, Really encouraging performance results on 0.11.x compared to say hyperHTML Stoked about this project.

[ruphin]

All right, I'll just upstream my changes to the benchmarking framework and abandon work on a custom benchmark for lit-html.

[justinfagnani]

@ruphin I can push my work asap... do you have your benchmarks anywhere?

[ruphin]

I have a fork of that benchmark framework that includes benchmarks for lit-html: https://github.com/ruphin/js-framework-benchmark/tree/update_lit-html

You can find the lit-html implementations that conform to the benchmark here: https://github.com/ruphin/js-framework-benchmark/tree/update_lit-html/frameworks/non-keyed/lit-html
https://github.com/ruphin/js-framework-benchmark/tree/update_lit-html/frameworks/keyed/lit-html

The benchmarking framework itself does a lot of nice things, but it is incredibly over-engineered and uncomfortable to use. I don't expect to be able to extract their benchmark runners with little effort. I was thinking about removing all of its components we don't care about and wrapping it with a single script/runner that does all the things we do want with a single command, but it becomes sort of a 'black box' testing machine.

The nice thing about using this benchmark in particular is that it gives us the ability to test against other frameworks in addition to testing against our own changes.

[ruphin]

I think my benchmarking tool is useful enough to warrant putting some more effort in, so I cleaned it up:

https://github.com/ruphin/bench-lit

That lets you easily run and process the benchmarks with a single command. It produces JSON and nice HTML table-formatted output.

[justinfagnani]

Closing this as we have tachometer integration now, though we continue to need more and better benchmarks. We should file individual issues for benchmark needs around specific cases.