Amphtml: Frequent Percy failures

[samouri]

summary
In the past ~week or so I've noticed what feels like a significant uptick in spurious Percy failures. In some cases, I've been unable to get them to pass even with multiple rounds of rebasing/rerunning.

It also seemed to me like there were just a couple of test cases failing most of the time (e.g. "Grid layer (fill)" failed for most of my PRs).

We should harden our tests to be less flaky, or at least skip the worst offenders

/to @rsimha

[rsimha]

@samouri Thanks for the report. Can you post some example links to failures?

• @calebcordry is this week's build cop and can help alleviate this by skipping any test that regularly flakes.
• @gmajoulet should know what those STAMP visual tests are trying to do, and how to make them more reliable.
• @danielrozenberg is the overall area expert for visual tests, and can advise on whether the tests are using any known anti-patterns (like not waiting long enough for a UI element to appear)

[calebcordry]

the mega-menu visual diff tests also seem to pop up often, eg https://percy.io/ampproject/amphtml/builds/4467899

[caroqliu]

I've also had trouble with amp-story: https://percy.io/ampproject/amphtml/builds/4471771
Occasionally I will see amp-sidebar, amp-lightbox-gallery

[gmajoulet]

@caroqliu the first one looks like a bug, thanks for posting about it. The second one is an image loading issue though, which caused many of the failures I've seen.

[danielrozenberg]

Thanks everyone for adding more "evidence" here, I'll look into it tomorrow. In the meantime please everyone keep adding more broken/flaky builds that you see!

// special callout to @ampproject/wg-ui-and-a11y @ampproject/wg-runtime @ampproject/wg-ads @ampproject/wg-stories

[samouri]

Two more examples:

1. "amp-mega-menu: template render (click menu item one)"
2. "amp-story: info dialog"

[samouri]

Two more examples, and one repeat:

1. "amp-story: unsupported browser (desktop)" -- I've definitely seen this one many times
2. "AMP Sticky Ad"
3. "amp-story: info dialog" (repeat from above)

[caroqliu]

fade-in & fade-in-scroll: https://percy.io/ampproject/amphtml/builds/4506500

This also appears to be an image loading issue

[danielrozenberg]

Contacted Percy support, I'll keep y'all posted here. Thanks for the various examples!

[ChrisAntaki]

I'm also seeing issues with images loading
https://percy.io/ampproject/amphtml/builds/4560241

[danielrozenberg]

I'm working on applying some of the suggestions that Percy support gave me, will keep this thread updated!

[danielrozenberg]

I merged #27292 two days ago, which should, hopefully, have fix the missing images flakes. Let me know if you see any more flaky visual diff tests and I will debug them

[danielrozenberg]

After making the change in #27292 and disabling the flaky tests last week it looks like this issue is mostly resolved. Please reopen this if you see more flaky visual diff tests!