Kibana: Performance benchmarking via functional test times on ci

[stacey-gammon]

I looked for an existing issue but couldn't find one.

Has anyone on the Ops team looked into tracking functional test completion times, over time, on ci, as a crude way to see if any particular changes may have caused a performance regression?

I had/have a system that does this but it's outside ci so it's difficult to maintain and hasn't been running in months. I would love to see something like this incorporated into ci, and while it's far from perfect, it is better than nothing. Happy to give a walk through of my system if someone would be able to pick this up.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[tylersmalley]

@brianseeders, I believe you are doing this as part of splitting the suites?

[brianseeders]

@tylersmalley Yeah, that's the plan. It will probably happen in a couple of phases, but I want to get the metrics pushed into ES somewhere, and also have warnings on PRs if a test suite gets much slower

[LeeDr]

cc/ @wayneseymour

[wayneseymour]

@stacey-gammon great idea! @brianseeders My boss and I have a cluster that we push metrics into (as you know), perhaps we can push it there :)

[tylersmalley]

@wayneseymour, I think we should keep with pushing this into build-stats or we should create a cluster on cloud for all things Kibana CI related.

[LeeDr]

@tylersmalley The Infra team pushed back a bit on adding Code Coverage to that existing 6.8.3 build-stats cluster. They have some concerns that upgrading it will break some Jenkins interactions and the builds that depend on them.
We're writing code coverage data to a Cloud cluster which I just upgraded to 7.6.1.
I have an open Infra ticket to get it named kibana-stats

[stacey-gammon]

@wayneseymour will updates for this project be posted to this issue?