Amphtml: Measure the performance of the AMP runtime as part of continuous integration

[rsimha]

For a long time, we've wanted to be able to write end-to-end performance tests for AMP page rendering and interaction. This issue is meant to track the investigation of what technology to use, and following that, the implementation and deployment of these tests.

[rsimha]

Ideally, these tests ought to be a part of continuous integration testing for PRs and push builds.

Based on https://github.com/travis-ci/travis-ci/issues/352 and https://github.com/travis-ci/travis-ci/issues/3313, there is no native support from Travis for performance tests, due to the lack of a guarantee that all tests are run on equally powerful VMs.

[rsimha]

One option for a third party tool that integrates with Travis CI is here: https://loadimpact.com/integrations/performance-testing-travisci

Edit: Seems like this tool does a lot more than we really need. Sample test result here.

[rsimha]

/cc @erwinmombay @cramforce @dvoytenko @choumx for comments on past efforts in this direction.

[rsimha]

More options to consider:

browser-perf: https://github.com/axemclion/browser-perf
perfjankie: https://github.com/axemclion/perfjankie
phantomas: https://github.com/macbre/phantomas

[rsimha]

Read this!

https://github.com/ampproject/amphtml/blob/master/extensions/amp-viewer-integration/TICKEVENTS.md

[cramforce]

Also important to note that we write these events to the performance timeline. That can be read using DevTools protocol or window.performance.getEntriesByType('mark') (From memory)

[ampprojectbot]

This issue hasn't been updated in awhile. @rsimha Do you have any updates?

[ampprojectbot]

This issue hasn't been updated in awhile. @rsimha Do you have any updates?

[ampprojectbot]

This issue hasn't been updated in awhile. @rsimha Do you have any updates?

[rsimha]

This has been on the infra backlog for a while, but unfortunately, we don't yet have a solution.

@kristoferbaxter I recall you're having someone on your team tackle this during the summer. Moving to Performance. Let us know if there are any tooling needs -- happy to help.

[estherkim]

Here's the plan to add gulp performance to our CI -

• Add test fixtures to config.json
• First, run a subset of ads, analytics, and mock real world pages on master builds
  
  
  
  • This allows wg-ads to easily find performance regressions, meeting needs of https://github.com/ampproject/amphtml/issues/27402
  
  
  • We can use master builds to estimate job duration and assess CI resource constraints
  
  
• Then, run full suite on nightly builds when nightly becomes available
  
  
  
  • Depending on the frequency of performance regressions, running only on nightly builds could be a happy medium for wg-ads
  
  
• Optimize gulp performance as we go :)

[rsimha]

👋 @ampproject/wg-performance folks! Bumping this old issue because I'm build cop this week and noticed that the performance tests currently take ~30 minutes on Travis with ~20% of the tests failing.

Couple questions:

• Are these known failures that are expected to be fixed by upcoming code changes? If not, does it make sense to repair / skip these tests (possibly lowering overall CI running time)?
• Anything else that can be done in order to make the performance tests a blocking check, as we initially set out to do?

[erwinmombay]

@rsimha i only started looking at this, how are the variances like on these numbers?

cc @kevinkimball

[rsimha]

i only started looking at this, how are the variances like on these numbers?

The variance is low (performance tests regularly take ~30 mins), while the failure rate is high (20% failures).

[erwinmombay]

if they are consistently failing, i'm all for skipping them.

@kevinkimball would it make sense to adjust or check against these performance tests per release?
i honestly have not looked at these at all during releases but with our drive on performance i wonder if we should give this more weight than we currently do.

[kevinkimball]

I am working on figuring out what can be done to improve the stability of these tests. Will update by EOW

[kevinkimball]

In addition to the issue with the images test failing consistently, it seems like the browser fails to launch occasionally (see example). Not sure if we need a separate issue to track that