Envoy: proposal: Add tracking latencies and flamegraphs in CI

[oschaaf]

Filing this issue to get a feel for interest in this

Goal:

Add a means to track and persist latency numbers and perf visualizations like flamegraphs over time in CI. This would allow us to track how we're doing over time as well as have perf information
at hand when a latency regression is observed.

Description:

Nighthawk uses a lightweight python-based framework for integration testing.
This framework serves as a basis for writing NH's own benchmarks.

With a small bit of modification this could be modified to:

• make consumption very low friction in foreign code bases (like Envoy)
• allow it to inject proxies in the between the client and test server. For example, Envoy at a certain sha
• scavenge tests from external locations

More details, and some concrete scripts for getting an idea of what this would look like can be found here.

/cc @danzh2010 @htuch

[htuch]

I think even redline QPS would be an amazing contribution here, everything else proposed seems like gravy. +1000.

[mattklein123]

See https://github.com/envoyproxy/envoy/issues/961. I desperately want this. This will require a lot of thought in terms of how to structure repeatable tests, but yes, we really need to do this.

[antoniovicente]

I don't know if this exists yet, but it would also be good to have a few relatively small benchmark scenarios that can be used for A/B comparison of performance after changes to data plane components, specially in cases where we expect some performance impact. Tracking performance data for the small benchmarks over time on a calibrated environment would be great.

[oschaaf]

I have started exploring this. Tracking progress here.
@antoniovicente It might be good to take a look at test_benchmarks.py, to see if that allows enough flexibility. The idea is that consumers can specify their own locations where the suite should scavenge tests, which in turn can supply custom fixtures with custom Envoy configurations.

[mattklein123]

cc @marcomagdy who is also interested in helping with this effort.

[snowp]

We'd also be interested in this, so let me know how I can help to move this forward

[oschaaf]

[antoniovicente]

[oschaaf]

[abaptiste]

[htuch]

[mattklein123]

[abaptiste]

[mattklein123]

[abaptiste]

[htuch]

[mattklein123]

[oschaaf]